FLIGHT+ Dataset | Versions & data sources
Versions and Data Sources
The current version of the FLIGHT+ Dataset is v1.3. This page provides documentation of what has changed in each version.
Version 1.0
Version 1.0 of the FLIGHT+ Dataset was released to the RPI Member Companies on 14 August 2014. This version included data from 1999 through 2013.
Version 1.1
Version 1.1 of the FLIGHT+ Dataset was released to the public on 20 April 2016. This version extends the dataset through 2015 and contains several new parameters, a bug fix for storm-relative winds, and other minor changes. The differences between v1.0 and v1.1 are summarized below.
1. FLIGHT+ v1.1 fixes a critical error in the calculation of the storm relative winds. In v1.0, this bug resulted in incorrect storm relative winds. The potential magnitude of the errors for the storm-relative quanities are up to twice the translation speed of the storm.
2. A major change been made to the truncation of the binned (RLB_*) parameters. In v1.0, the remainder of a leg (going outward from the center of the cyclone) was truncated if the flight level pressure deviated by more than a set threshold (typically 10 mb) from the value of the average flight level pressure within 25 km of the center of the cyclone. In v1.1, this truncation is no longer conducted, meaning that data are still included for the leg segments when the plane's altitude was changing significantly. This change ensures that the 'good_radial_leg' metadata now returns information about the 'true' maximum wind speed for the full radial leg, regardless of changes in the plane's flight level pressure. Because some users may still wish to mask out these leg portions when the pressure was fluctuating (to avoid distracting artifacts from appearing in their plots or analysis), a radial leg mask variable is now provided (value = 1 when the flight level pressure is within tolerance of the near-center average flight level pressure; value = 0 when it is out of tolderance, indicating that the plane's flight level pressure has deviated significantly). This variable is: FL_RLB_flight_level_pressure_is_close_to_average_flight_level_pressure_near_storm_center
3. Several additional metadata parameters have been added related to the leg-maximum wind speed. New metadata include:
- the average flight level pressure within 25 km of the cyclone center (used to establish the baseline for determining if the plane's flight level pressure was changing significantly from the nominal flight level pressure for that leg)
- the flight level pressure at the location of the leg-maximum flight level wind speed
- the flight level pressure at the location of the leg-maximum SFMR wind speed
4. The Willoughby-Chelmow wind center information is now included in the L3 data product (this information is read from the .trak files produced by HRD).
5. The FLIGHT+ Dataset has been updated to include data for the 2014 and 2015 seasons. At this point, the files for these years are considered as 'beta' versions. They have not undergone the same visual QC that the other years have. Additionally, the iBTRaCS stormID may not be present for certain 2015 TCs.
6. The names of the following parameters have changed in v1.1. If you have already developed code that uses v1.0 of the dataset, you may need to update the following parameter names.
| Parameter Name in v1.0 | Parameter Name in v1.1 |
|---|---|
| FL_candidate_radial_leg_average_pressure | FL_candidate_radial_leg_average_flight_level_pressure |
| FL_candidate_radial_leg_minimum_pressure | FL_candidate_radial_leg_minimum_flight_level_pressure |
| FL_candidate_radial_leg_maximum_pressure | FL_candidate_radial_leg_maximum_flight_level_pressure |
| FL_candidate_radial_leg_std_pressure | FL_candidate_radial_leg_stdev_flight_level_pressure |
| FL_good_radial_leg_average_pressure | FL_good_radial_leg_average_flight_level_pressure |
| FL_good_radial_leg_minimum_pressure | FL_good_radial_leg_minimum_flight_level_pressure |
| FL_good_radial_leg_maximum_pressure | FL_good_radial_leg_maximum_flight_level_pressure |
| FL_good_radial_leg_std_pressure | FL_good_radial_leg_stdev_flight_level_pressure |
| FL_good_radial_leg_azimuth | FL_good_radial_leg_average_azimuth |
7. A number of new parameters have been added to v1.1.
| New Parameter in v1.1 |
|---|
| FL_RLB_flight_level_pressure_is_close_to_average_flight_level_pressure_near_storm_center |
| FL_good_radial_leg_average_pressure_within_25km_of_center |
| FL_good_radial_leg_flight_level_pressure_of_flight_level_rmax |
| FL_good_radial_leg_flight_level_pressure_of_sfmr_rmax |
| FL_good_radial_leg_time_offset_of_flight_level_rmax |
| FL_good_radial_leg_time_offset_of_sfmr_rmax |
| RLB_flight_level_pressure_is_close_to_average_flight_level_pressure_near_storm_center |
| FL_WC_wind_center_time_offset |
| FL_WC_wind_center_latitude |
| FL_WC_wind_center_longitude |
| FL_WC_wind_center_zonal_velocity_interpolated |
| FL_WC_wind_center_meridional_velocity_interpolated |
Version 1.2
Version 1.2 of the FLIGHT+ Dataset was an interim version which added the 2014-2015 seasons. Apart from some data improvements to a small handful of TCs, this is largely a minor update. The most important change that may affect users is that the filenaming convention has been changed to be more consistent across all 3 data product levels. More details on the differences between v1.1 and v1.2 are summarized below:
- The filenaming convention for the Level 1/2/3 files has been modified. The Level 1 and Level 2 files now use the same naming convention as the Level 3 files. Additionally, all filenames will now include the dataset version number to make it easier for users to see which version of the data they are working with. The new naming convention is: FLIGHT_[stormyear]_[ATCF storm id]_[storm name]_L[dataset product level]_v[dataset version number].nc
- HRD has now provided Willoughby-Chelmow wind center .trak files for the 1997-1998 storms, allowing radial legs to be provided for the TCs in these years. This will add L3 output for 15 additional storms, compared with v1.1.
- Dataset products have been updated to use higher temporal resolution data (1-sec) data that HRD has provided. This includes:
- Erika (1997)
- Danielle (1998)
- Bonnie (1998)
- Mitch (1998)
- Bertha (2014)
- Since HRD sometimes provides multiple .trak files, it is necessary to pick one for the L2 and L3 processing. The following .trak selections have been made:
- Danny (1997; using the NOAA .trak file)
- Guillermo (1997; using the NOAA42 .trak file)
- Cristobal (2014; using the NOAA .trak file)
- TD05 (1997; .trak file now available)
- Minor changes have been made to the file-level metadata
Version 1.3
Version 1.3 is the current version of FLIGHT+. This vesion added some additional automatic quality control checks and provides partial fixes for two major issues:
1. Bad time metadata in NOAA NetCDF data files: A key piece of metadata that FLIGHT+ uses construct the time coordinate for these data files was off by exactly five hours in all NOAA NetCDF Data files from 2005-2012. This issue resulted in very few radial legs being parsed from any of these 160 flights (and any radial legs that were parsed were likely parsed backward). All AFRES flights during this period were unaffected. The FLIGHT+ code has been updated to detect incorrect time stamps and to correct the time, if necessary. As a result of this fix, approximately 1000+ radial legs have been added to the v1.3 dataset. It is possible that the processing of NOAA data files from 2012 onward still have issues. If so, then radial legs may be missing from these flights for the more recent years. This issue will be investigated further and hopefully fixed in future Version 1.4.
2. Improvements in the radial binning to account for low precision lat/lon data: Some of the early source data files only provide the lat/lon of the plane location in 0.1 deg increments even though the data frequency had been increased to 1-second. As a result, the calculation of the distance from plane to the center of the cyclone can appear to be constant for 10-15 seconds, then jumps to a new value. When the motion of the cyclone itself is removed, the distance to the center may slowly increase or decrease by slight amounts, then undergo a jump (depending on whether the motion of the storm is along with the motion of the plane or against it). This affects the radial binning of the flight level data into the radial legs. In previous versions of FLIGHT+, all values in a particular radial leg were sorted by distance to center. Although an effort was likely made to filter and interpolate the distance to center to fix the jumps, it has been determined that the method of doing so was inadequate. As a result, data from +/- 0.6 km of a given point may have been interspersed, resulting in a decrease in the effective grid spacing of the radial data from 100 m to approximately 1 km. This likely would have mattered most for any users attempting to compute radial derivatives of these data. In v1.3, the algorithm for detecting the effects of this low precision on the distance to center has been improved, and any points in which the distance to the center is changing against the direction of the jumps are now filtered out and linear interpolation is used to fill these in (which is reasonable since the plane is flying at a fairly constant speed). The issue still remains however for jumps that occur in the direction of the storm motion. This issue will be investigated further and hopfully fixed in future Version 1.4.
In May 2020, work was completed to extend v1.3 to include the 2014 - 2018 seasons. Data for the 2014 - 2015 seasons were previously available as "beta" releases because the visual quality control checks had not been completed to the same degree as the 1997 to 2013 seasons. These checks have now been completed and the 2014-2018 data should be of the same quality as the earlier parts of the dataset. We note, however, that 43 out of the 714 flights for the 2014-2018 seasons have issues which, if included, would have impacted the assessment of either the intensity, radius of maximum winds, or the outermost extent of wind radii. Due to disruptions related to the COVID-19 pandemic, access to computing resources to address these issues is not possible, so for now, these problem flights have been "quarantined" from the final dataset version. The nature of the issues for each exluded flight is noted in the data provenance Google Doc listed above. If users would like to access the full set of flight level data including problem flights, they are welcome to download the v1.3-beta version of the dataset.
In January 2021, work was completed to extend v1.3 to include the 2019 season. This update adds 11 storms, 119 flights, and 496 good radial legs to the dataset. A minor update was made in March 2021 for Dorian (2019) and Karen (2019). The initial Jan 2021 release for these storms included several flights with artifacts that affected the analysis of the storm intensity, radius of maximum winds, or wind radii. For example, one of the earlier Dorian flights had a 99.3 m/s surface maximum wind speed, while the flight level wind speed maximum was just 54 m/s. These problem flights have been removed in the updated files provided in the March 2021 release. For reference, these problem flights are included in the v1.3-beta data products. Details about the specific issues and flights which have been excluded from the official v1.3 data product can be found in the data provenance notes available at: https://docs.google.com/spreadsheets/d/11Z_XnN0bYxIUhEUajHjMQxIX4dPoJnlvj2KDEnBYFak/edit?usp=sharing.
Potential Near-term Updates
Over the next few months, we plan to add azimuthal mean profiles to the dataset as a Level 4 data product.
Funding has not yet been identified to extend the dataset for the 2020 season. Approximately $10-15k will be needed to accomplish this update.
Future Version 2.0
Version 2.0 of the FLIGHT+ Dataset will include one major change that involves computing the flight level pressure from other flight navigation altitude/pressure variables. Because most AFRES flights prior to 2004 do not include the flight level pressure, adding in a calculated version of this variable will enable much more research to be done with these data. This version will also includes fixes for the issues that were partially adressed in v1.3. This version will also add the capability to use multiple .trak files (for instance, when planes are in the storm at the same time, but at different levels. This change is expected to improve the quality of the navigation to storm relative coordinates for storms such as Cristobal (2014). No release date has been scheduled for v2.0 and this work is entirely subject to finding future funding support. If this funding does materialize, this new version likely will not be available until later in 2022.
Real-time FLIGHT+ Data Stream
Initial scoping work is underway to prepare a Real-time FLIGHT+ Data Stream based on real-time high resolution NOAA flight level data and/or High Density Observations (HDOBS). A key challenge is to come up with a robust center-finding technique that will work on partial (incomplete) flights.
Get Email Updates
When changes are made to the dataset, an update will be sent out to the tcdp-flight-news e-mail list. If you would like to subscribe to that list, please click here.
Data Sources
The Flight Level Dataset has been compiled from flight level data collected by the NOAA and Air Force Reserve (AFRES) Hurricane Hunter aircraft. The data have been post-processed and quality-controlled by the NOAA Aircraft Operations Center (NOAA/AOC) and the NOAA Atlantic Oceanography and Meteorological Laborary's Hurricane Research Division (NOAA/AOML/HRD).
The intially RPI-funded effort undertook extensive efforts to further quality control and standardize the data. The resulting "cleaned" data files have been sent back to HRD for inclusion in their data catalog. From the perspective of this project, these cleaned and QC'd data files are termed the "Level 0" data. Users who wish to download Level 0 data can do so directly from HRD:
The NOAA and AFRES flight level data come in a wide variety of data formats. Information about the formats supported in the FLIGHT+ dataset can be accessed here.
Information about the correspondence between parameter names in the Level 0 data files and the parameter names in the FLIGHT+ dataset can be accessed here.
This page was last updated 07 March 2021 by Jonathan Vigh.