VDM+ Dataset | Dataset details

Data Sources Used in v1.100 of the Enhanced Vortex Data Message (VDM+) Dataset

This page provides information about the source datasets used to build v1.100 of the VDM+ dataset. From here, the reader can also peruse the raw VDMs that are used in this version and download stand-alone versions of the NHC Storm Table and the Extended Best Track and SHIPS Development datasets. Each of these source datasets has been converted into the convenient Network Common Data (NetCDF) format.

The enhanced Vortex Data Message Dataset (VDM+) is a compilation of several source datasets as well as a synthesis of data contained in the raw Vortex Data Messages. Please use the following jump links to access more information about each of these source datasets:

 

Master Storm Table

The National Hurricane Center (NHC's) Master Storm Table is utilized to provide a consistent mapping between storm numbers, storm names, and the unique storm identification tags. This is necessary because storm numbers are sometimes relabeled after the season.

There are two versions of the NHC storm table, one which is is updated in real-time and another that corresponds to the post-season best-track (but may also include provisional data for the current year cyclones). VDM+ uses the post-season archive version, available on the NHC server at: ftp://ftp.nhc.noaa.gov/atcf/archive/storm.table

To aid in the processing of these data to the Tier 1 data files, this ASCII-formatted file has been converted to a separate, self-contained Network Common Data format (NetCDF) file:

v1.100 of the VDM+ Dataset uses the above storm table file that was converted to NetCDF from a storm table downloaded from NHC's website on 24 November 2015.

 

Best Track (BT) Database

The VDM+ dataset includes the full parameter set of the NHC's Hurricane Database (HURDAT2), otherwise known as the "Best Track" database. Since HURDAT2 is actually built from the underlying Automated Tropical Cyclone Format (ATCF) b-deck files (Franklin 2013, personal communication), the VDM+ dataset uses these post-season archive b-deck files as the definitive source of Best Track information, rather than the compiled HURDAT2 dataset. The parameters in the b-deck files that undergo rigorous post-season subjective analysis by NHC include:

  • Storm status (TY)
  • Position of the storm center (LAT/LON)
  • Maximum sustained surface wind speed (VMAX, otherwise known as 'Intensity')
  • Minimum central pressure (MSLP)
  • Wind radii in each quadrant for the 34-, 50-, and 64-kt thresholds (beginning in 2004, given by RAD, WINDCODE, and RAD1, RAD2, RAD3, and RAD4)

It is important to note that not all of the parameters in the b-deck files undergo the rigorous post-season subjective analysis by NHC hurricane specialists. The parameters in the b-decks that are not best-tracked include:

  • Radius of maximum winds (MRD)
  • Maximum gusts (GUST)
  • Eye diameter (EYE)
  • Radius of the last closed isobar (RADP)
  • Pressure of the last closed isobar (RRP)
  • Maximum seas (MAXSEAS)
  • Radial extent of seas of the given threshold (SEAS, SEASCODE, SEAS1, SEAS2, SEAS3, SEAS4)

The b-deck files are freely available for download at ftp://ftp.nhc.noaa.gov/atcf/archive/.

The HURDAT2 database itself may be downloaded from: http://www.nhc.noaa.gov/data/#hurdat.

Documentation of HURDAT2 is available at: http://www.nhc.noaa.gov/data/hurdat/hurdat2-format-atlantic.pdf.

The abr-deck file format is described by: http://www.nrlmry.navy.mil/atcf_web/docs/database/new/abrdeck.html.

Changes that have been made to the NHC Best Track b-decks since the last version of VDM+ are described here.

NOTE that best track b-decks were not available for 2015 tropical cyclones at the time that v1.1000 of the dataset was built. Thus, the VDM+ Dataset uses operational working b-decks for these tropical cyclones. Additionally, best track data may be missing for the Western Pacific portion of any tropical cyclones that formed in the Eastern or Central Pacific basins and subsequently traveled into the Western Pacific basin.

v1.100 of the VDM+ Dataset uses the b-deck files that were downloaded from NHC's website on 04 November 2015.

To learn more about the Best Track Database, please see the Best Track Database References.

 

Extended Best Track (EBT) Dataset

Since the Best Track database does not include wind radii values prior to 2004 when NHC began best-tracking these, the Extended Best Track (EBT) dataset provides these from the values that were operationally analyzed in real-time.The EBT dataset also includes many of the non-best-tracked parameters for storms that are currently missing in the b-deck data files, and adds an additional parameter which provides the distance to the nearest land. The EBT dataset extends back to 1988 and is maintained by Dr. Mark DeMaria at NOAA's Satellite and Information Service (NOAA/NESDIS).

The VDM+ dataset merges the BT and EBT data into one set of combined Best Track data by using the BT parameter values whenver they exist; when the BT values are missing for a given parameter, the EBT value has been used. When data are present in both datasets, the BT values are used. This merging is done for the following b-deck parameters: LAT, LON, VMAX, MSLP, MSLP, MRD, EYE, RADP, RRP, and the 34-kt, 50-kt, and 64-kt wind radii.

The EBT dataset is freely available for download from: http://rammb.cira.colostate.edu/research/tropical_cyclones/tc_extended_best_track_dataset/.

Additional documentation on the EBT dataset are available from: ftp://rammftp.cira.colostate.edu/demaria/ebtrk/ebtrk_readme.txt.

Prior to compiling the VDM+ dataset, the EBT dataset has been converted from its native ASCII text format into a separate, self-contained Common Network Data Format (NetCDF) file. Dr. Mark DeMaria has granted permission for TCDP to provide the NetCDF version of the EBT dataset as a convenience to researchers wishing to use the EBT dataset directly. We ask that users remember that the ASCII text version constitutes the official version of the dataset and that it is possible that errors may have been introduced into the NetCDF version during the conversion process. The file may be downloaded using the following link:

NOTE that this version of EBT only provides data up to 2014 for the Atlantic cyclones. EBT data end in 2013 for Eastern Pacific and Central Pacific cyclones.

The VDM+ v1.100 Dataset uses the above v2.03 of the EBT dataset file converted to NetCDF from the dataset file that was downloaded from the Extended Best Track Dataset web site on 04 November 2015. The v2.03 file was posted on taht web site on 06 August 2015.

To learn more about the Extended Best Track Database, please see the Extended Best Track Dataset References.

 

SHIPS Development Dataset

VDM+ includes another dataset to provide information about the tropical cyclone's environment and structure. These parameters are derived from model analyses of the Global Forecasting System (GFS) model (prior to 2000, the NCEP/NCAR reanalysis has been used) and various satellite data.

The SHIPS Developmental Dataset and additional documentation is freely available for download at: http://rammb.cira.colostate.edu/research/tropical_cyclones/ships/.

Additional documentation about the SHIPS parameters is available at: http://rammb.cira.colostate.edu/research/tropical_cyclones/ships/docs/SHIPS_predictor_file_2013.doc

Because the ASCII-formatted SHIPS Development Dataset file can be very computationally-intensive to read by interpreted languages such as NCL and Matlab, it has first been converted to a separate, self-contained netCDF file. Data values in the netCDF file have been converted from the original scaled units of the ASCII file into the respective physical units for each parameter, according to the conventions described in the SHIPS Development Dataset documentation. Dr. Mark DeMaria has granted permission for TCDP to provide the NetCDF version of the SHIPS Development Dataset as a convenience to researchers wishing to use the SHIPS dataset directly. We ask that users remember that the ASCII text version constitutes the official version of the dataset and that it is possible that errors may have been introduced into the NetCDF version during the conversion process. The file may be downloaded using the following link:

The VDM+ v1.100 Dataset uses the above SHIPS Development Dataset file converted to NetCDF from the dataset files that were downloaded from the SHIPS web site on 04 November 2015. The 2013 SHIPS Development Dataset files were originally posted on that web site on 25 September 2014.

To learn more about the SHIPS Development Database, please see the SHIPS Development Dataset References.

 

Raw Vortex Data Messages

A VDM is a short text-formatted message that summarizes the vital structure and intensity parameters found by an aircraft reconnaissance plane each time it makes a full pass through the center of the tropical cyclone. An example VDM is given below:

        URNT12 KNHC 220739
	VORTEX DATA MESSAGE
	A. 22/07:14:30Z
	B. 24 deg 48 min N
	  087 deg 46 min W
	C. 700 mb 2208 m
	D. NA  kt
	E. NA  deg     nm
	F. 225 deg 148 kt
	G. 134 deg 013 nm
	H.         899 mb
	I.   9 C/ 3047 m
	J.  31 C/ 3043 m
	K.  -3 C/ NA
	L. CLOSED WALL
	M. C16
	N. 12345/ 7
	O. 0.02 / 1  nm
	P. AF307 1618A RITA         OB 11
	MAX FL WIND 165 KT NE QUAD 05:34:00 Z
	STADIUM EFFECT VERY VISIBLE IN MOONLIGHT
	FREQUENT LIGHTNING WITHIN EYEWALL

The basic anatomy and structure of a VDM is described in Vigh et al 2012 and the accompanying supplement.

VDMs for the Atlantic, Eastern Pacific, and Central Pacific basins for 1989-2015 have been gleaned from several sources including NHC's ftp and http recon archives, the Florida State University archive, and the personal archives of Mark Zimmer and Steve Feuer. Scripts have been used to cleanup some 6448 unique VDMs by removing extraneous unprintable characters and then organize them into text files by storm and year.

Since this data format originates back from the 1950's when reports were transmitted by voice over shortwave radio, a designator representing a letter from the phonetic alphabet (ALPHA, BRAVO, CHARLIE, etc.) is used for each data line of the VDM. A detailed guide to decoding the VDMs is available in Table 5-2 of the National Hurricane Operations Plan (2015, available online at: http://www.ofcm.gov/nhop/15/nhop15.htm). Numerous hand edits were necessary to correct errors so that the data could be read by an automated reader. Most of these errors are typographical errors or variants in formatting or abbreviation. Often, the on-board operator might have put a '0' in place of a 'O' or vice versa, used the wrong units (e.g., 'M' instead of 'MB'), or otherwise made a formatting error in which it was easy to make a correction based on the context. Some errors involved putting the wrong data on the wrong lines (such as transposing the information on lines L. and M.). In other cases, it was not possible to determine a plausible correction so no correction has been attempted. In 1998, the VDM format was changed so that the repeated fix information (old item N.) was eliminated. This means that the new item N. corresponds to the old item O., new O. corresponds to the old P., and the new P. corresponds to the old Q. The new VDMs do not have item Q.

Some VDMs may be missing, especially from some of the NOAA missions which were being conducted for research purposes instead of operational reconnaissance. VDMs may have been generated from some of these research missions, but it seems that they may now exist only on hardcopy at Aircraft Operations Center (AOC, personal communication, Barry Damiano 2009). Whether apocryphal or not, those NOAA VDMs have not been included here. Also due to a change in data archiving procedures on NHC's servers, VDMs from the INVEST and CYCLONE stages of a storm may not be included from 2005 onward. This is because after 2004, VDMs were collected into files for each storm based on the storm name. Some effort has been undertaken to include the tropical depression stages, but some INVEST and CYCLONE VDMs have undoubtedly been left out. Starting in 2007, the unique STORMID has been included in the VDMs, so VDMs from systems from 2007 onward should always be included for all stages after the system is operationally designated as a tropical depression.

The automated reader has been designed with certain data consistency checks (such as looking for standard units). In some rare cases, uncorrectable data will not clear these checks; when this occurs, such data items will subsequently be left out of the dataset.

The raw VDMS have been cleaned (to remove unprintable characters) and hand-edited in order to organize the VDMs by storm and year. The raw, hand-edited collection of VDMs that have been used to build v1.100 of the VDM+ dataset are made available by TCDP in the following forms:

Documentation of the hand-edits that were made to the VDMs is provided here.

In 2013, the VDM format experienced some minor changes to how certain mandatory remarks are coded in the REMARKS section. The code used to decode and build the VDM+ Dataset from the raw VDMs has now been updated to read the new combined outbound and maximum wind remark type, resulting in a substantial increase in wind maxima data for 2013-2015. The logic change has resulted in a small handful of wind maxima not being read for a few storms in previous years. The missing wind maxima in v1.100 does not result in any substantive changes to the upper bound of intensity for all of the cases that were reviewed. Another change made to the VDM format in 2013 was to start including a remark providing the final wind speed report of dropsondes dropped in the center of the storm. This information can be used to infer the minimum central pressure of the tropical cyclone in instances when the dropsonde did not fall into the precise center. The VDM+ Dataset does not yet decode this new remark type. For more on the changes to the VDM format, please see the red-highlighted markup in the pages 5-6 to 5-9 of Chapter 5 of the 2013 OFCM National Hurricane Operations Plan.

 

This page was last updated 25 November 2015 by Jonathan Vigh.