VDM+ Dataset | Dataset details

Data Sources used in v1.000 of the Enhanced Vortex Data Message (VDM+) Dataset

This page provides information about the source datasets used to build v1.000 of the VDM+ dataset. From here, the reader can also peruse the raw VDMs that are used in this version and download stand-alone versions of the NHC Storm Table and the Extended Best Track and SHIPS Development datasets. Each of these source datasets has been converted into the convenient Network Common Data (NetCDF) format.


NOTE that this page provides the data source information for an OLD version of the VDM+ dataset. An equivalent page provides the data source information for the latest version.

The enhanced Vortex Data Message Dataset (VDM+) is a compilation of several source datasets as well as a synthesis of data contained in the raw Vortex Data Messages. Please use the following jump links to access more information about each of these source datasets:

 

Master Storm Table

The National Hurricane Center (NHC's) Master Storm Table is utilized to provide a consistent mapping between storm numbers, storm names, and the unique storm identification tags. This is necessary because storm numbers are sometimes relabeled after the season.

The storm table is updated in real-time and is available on the NHC web server at: ftp://ftp.nhc.noaa.gov/atcf/index/storm_list.txt

To aid in the processing of these data to the Tier 1 data files, this ASCII-formatted file has been converted to a separate, self-contained Network Common Data format (NetCDF) file:

The VDM+ v1.000 dataset uses the storm table that was downloaded from NHC's website on 01 November 2013.

 

Best Track (BT) Database

The VDM+ dataset includes the full parameter set of the NHC's Hurricane Database (HURDAT2), otherwise known as the "Best Track" database. Since HURDAT2 is actually built from the underlying Automated Tropical Cyclone Format (ATCF) b-deck files (Franklin 2013, personal communication), the VDM+ dataset uses these b-deck files as the definitive source of Best Track information, rather than the compiled HURDAT2 dataset. It should be noted that not all parameters in the b-deck files undergo the rigorous post-season subjective analysis by NHC hurricane specialists. The parameters which are best-tracked include:

  • Storm status (TY)
  • Position of the storm center (LAT/LON)
  • Maximum sustained surface wind speed (VMAX, otherwise known as 'Intensity')
  • Minimum central pressure (MSLP)
  • Wind radii in each quadrant for the 34-, 50-, and 64-kt thresholds (beginning in 2004, given by RAD, WINDCODE, and RAD1, RAD2, RAD3, and RAD4)

It should be noted that not all parameters in the b-deck files undergo the rigorous post-season subjective analysis by NHC hurricane specialists. The parameters in the b-decks that are not best-tracked include:

  • Radius of maximum winds (MRD)
  • Maximum gusts (GUST)
  • Eye diameter (EYE)
  • Radius of the last closed isobar (RADP)
  • Pressure of the last closed isobar (RRP)
  • Maximum seas (MAXSEAS)
  • Radial extent of seas of the given threshold (SEAS, SEASCODE, SEAS1, SEAS2, SEAS3, SEAS4)

The b-deck files are freely available for download at ftp://ftp.nhc.noaa.gov/atcf/archive/. The HURDAT2 database itself may be downloaded from: http://www.nhc.noaa.gov/data/#hurdat. Documentation of HURDAT2 is available at: http://www.nhc.noaa.gov/data/hurdat/hurdat2-format-atlantic.pdf. The abr-deck file format is described by: http://www.nrlmry.navy.mil/atcf_web/docs/database/new/abrdeck.html.

The VDM+ v1.000 dataset uses the b-deck files that were downloaded from NHC's website on 01 November 2013.

 

Extended Best Track (EBT) Dataset

Since the Best Track database does not include wind radii values prior to 2004 when NHC began best-tracking these, the Extended Best Track (EBT) dataset provides these from the values that were operationally analyzed in real-time.The EBT dataset also includes many of the non-best-tracked parameters for storms that are currently missing in the b-deck data files, and adds an additional parameter which provides the distance to the nearest land. The EBT dataset extends back to 1988 and is maintained by Dr. Mark DeMaria at NOAA's Satellite and Information Service (NOAA/NESDIS).

The VDM+ dataset merges the BT and EBT data into one set of combined Best Track data by using the BT parameter values whenver they exist; when the BT values are missing for a given parameter, the EBT values are used. When data are present in both datasets, the BT values are used. This merging is done for the following b-deck parameters: LAT, LON, VMAX, MSLP, MSLP, MRD, EYE, RADP, RRP, and the 34-kt, 50-kt, and 64-kt wind radii.

The EBT dataset is freely available for download from: http://rammb.cira.colostate.edu/research/tropical_cyclones/tc_extended_best_track_dataset/. Additional documentation on the EBT dataset are available from: ftp://rammftp.cira.colostate.edu/demaria/ebtrk/ebtrk_readme.txt and in the following reference: 

  Demuth, J., M. DeMaria, and J.A. Knaff, 2006: Improvement of advanced
           microwave sounder unit tropical cyclone intensity and size estimation
           algorithms. J. Appl. Meteor., 45, 1573-1581.

Prior to compiling the VDM+ dataset, the EBT dataset hs been converted from its native ASCII text format into a separate, self-contained Common Network Data Format (NetCDF). Dr. Mark DeMaria has granted permission for TCDP to provide the NetCDF version of the EBT dataset as a convenience to researchers wishing to use the EBT dataset directly. We ask that users remember that the ASCII text version constitutes the official version of the dataset and that it is possible that errors may have been introduced into the NetCDF version during the conversion process. The file may be downloaded using the following link:

The VDM+ v1.000 dataset uses the v2.01 of the EBT dataset files that were posted on 22 February 2013.

 

SHIPS Development Dataset

VDM+ includes another dataset to provide information about the tropical cyclone's environment and structure. These parameters are derived from model analyses of the Global Forecasting System (GFS) model (prior to 2000, the NCEP/NCAR reanalysis has been used) and various satellite data.

The SHIPS developmental datasetand additional documentation is freely available for download at: http://rammb.cira.colostate.edu/research/tropical_cyclones/ships/. Additional documentation is available at: http://rammb.cira.colostate.edu/research/tropical_cyclones/ships/docs/SHIPS_predictor_file_2013.doc

Because the ASCII-formatted SHIPS development datasetfile can be very computationally-intensive to read by interpreted languages such as NCL and Matlab, it has first been converted to a separate, self-contained netCDF file. Data values in the netCDF file have been converted from the original scaled units of the ASCII file into the respective physical units for each parameter, according to the conventions described in the SHIPS development datasetdocumentation. Dr. Mark DeMaria has granted permission for TCDP to provide the NetCDF version of the SHIPS Development Dataset as a convenience to researchers wishing to use the SHIPS dataset directly. We ask that users remember that the ASCII text version constitutes the official version of the dataset and that it is possible that errors may have been introduced into the NetCDF version during the conversion process. The file may be downloaded using the following link:

The VDM+ v1.000 dataset uses the SHIPS Development Dataset files that were posted on 10 May 2013.

 

Raw Vortex Data Messages

A VDM is a short text-formatted message that summarizes the vital structure and intensity parameters found by an aircraft reconnaissance plane each time it makes a full pass through the center of the tropical cyclone. An example VDM is given below:

	URNT12 KNHC 220739
	VORTEX DATA MESSAGE
	A. 22/07:14:30Z
	B. 24 deg 48 min N
	  087 deg 46 min W
	C. 700 mb 2208 m
	D. NA  kt
	E. NA  deg     nm
	F. 225 deg 148 kt
	G. 134 deg 013 nm
	H.         899 mb
	I.   9 C/ 3047 m
	J.  31 C/ 3043 m
	K.  -3 C/ NA
	L. CLOSED WALL
	M. C16
	N. 12345/ 7
	O. 0.02 / 1  nm
	P. AF307 1618A RITA         OB 11
	MAX FL WIND 165 KT NE QUAD 05:34:00 Z
	STADIUM EFFECT VERY VISIBLE IN MOONLIGHT
	FREQUENT LIGHTNING WITHIN EYEWALL

The basic anatomy and structure of a VDM is described in Vigh et al (2012).

VDMs for the Atlantic, Eastern Pacific, and Central Pacific basins for 1989-2012 have been gleaned from several sources including NHC’s ftp and http recon archives, the Florida State University archive, and the personal archives of Mark Zimmer and Steve Feuer. Scripts have been used to cleanup some 5616 unique VDMs by removing extraneous unprintable characters and then organize them into text files by storm and year.

Since this data format originates back from the 1950's when reports were transmitted by voice over shortwave radio, a designator representing a letter from the phonetic alphabet (ALPHA, BRAVO, CHARLIE, etc.) is used for each data line of the VDM. A detailed guide to decoding the VDMs is available in Table 5-2 of the National Hurricane Operations Plan (2012, available online at: http://www.ofcm.gov/nhop/12/nhop12.htm). Numerous hand edits were necessary to correct errors so that the data could be read by an automated reader. Most of these errors are typographical errors or variants in formatting or abbreviation. Often, the on-board operator might have put a '0' in place of a 'O' or vice versa, used the wrong units (e.g., 'M' instead of 'MB'), or otherwise made a formatting error in which it was easy to make a correction based on the context. Some errors involved putting the wrong data on the wrong lines (such as transposing the information on lines L. and M.). In other cases, it was not possible to determine a plausible correction so no correction has been attempted. In 1998, the VDM format was changed so that the repeated fix information (old item N.) was eliminated. This means that the new item N. corresponds to the old item O., new O. corresponds to the old P., and the new P. corresponds to the old Q. The new VDMs do not have item Q.

Some VDMs may be missing, especially from some of the NOAA missions which were being conducted for research purposes instead of operational reconnaissance. VDMs may have been generated from some of these research missions, but it seems that they may now exist only on hardcopy at Aircraft Operations Center (AOC, personal communication, Barry Damiano 2009). Whether apocryphal or not, those NOAA VDMs have not been included here. Also due to a change in data archiving procedures on NHC's servers, VDMs from the INVEST and CYCLONE stages of a storm may not be included from 2005 onward. This is because after 2004, VDMs were collected into files for each storm based on the storm name. Some effort has been undertaken to include the tropical depression stages, but some INVEST and CYCLONE VDMs have undoubtedly been left out. Starting in 2007, the unique STORMID has been included in the VDM format, so VDMs from systems should always be included now once the system is operationally designated as a tropical depression.

The automated reader has been designed with certain data consistency checks (such as looking for standard units). In some rare cases, uncorrectable data will not clear these checks; when this occurs, such data items will subsequently be left out of the dataset.

The resulting cleaned and hand-edited VDMs organized by storm and year are made available by TCDP in the following forms:

Documentation of the hand-edits that have been made to the VDMs is provided here.

 

This page was last updated 19 November 2015 by Jonathan Vigh./p>