=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.4 README for NCEP Real-time (NRT) Marine Observations 27 December 2007 ======================================================================= Document Revision Information (previous version: 24 September 2007): Added technical note indicating that all ship ids are masked from December 2007 onward. Removed references to weekly file which is no longer needed and thus not produced. ------------------------------------------------------------------------------- {1. Introduction} Monthly and weekly files of surface marine data gathered from the Global Telecommunications System (GTS) by NOAA's National Centers for Environmental Prediction (NCEP) become available at NCEP two days after the data month or week. These are basic observational data (individual marine reports from ships, buoys, etc.) in a simple ASCII format containing selected weather elements. The data are largely unedited, although one major change in processing became effective with March 1997 data (see section 4). Data for the immediately preceding month and week are available at NCEP; monthly files back to January 1991 are available from the NOAA Earth System Research Laboratory (ESRL) Physical Science Division (PSD). The data are also used at PSD to construct monthly summaries for 2-degree latitude x 2-degree longitude boxes. For additional information about these summary products, please refer to: http://www.cdc.noaa.gov/cdc/data.nmc.marine.html Copies of the monthly data, and the 2-degree summaries, become available at PSD about 2-5 days following each data month. Section 2 gives detailed information about ftp access to the data at NCEP or PSD, and Section 3 provides the format description. Delayed data not available in these files, and marine reports containing the full suite of observed weather elements, eventually become available in GTS archives stored at the NOAA National Climatic Data Center (NCDC) in Asheville, and at the National Center for Atmospheric Research (NCAR) in Boulder. The full marine reports, plus a variety of delayed data, are merged into the International Comprehensive Ocean-Atmosphere Data Set (ICOADS). For more information about ICOADS including current data availability, please visit the ICOADS Website (http://icoads.noaa.gov). {2. Availability via anonymous ftp} Although the basic data are in a simple ASCII format (section 3), the files are stored and transferred via ftp in compressed form. Thus, depending on the computer type to which you are transferring the data, binary mode should generally be used for ftp transfers. The data at NCEP are compressed using the "gzip" utility; to ensure more widespread availability, the data at PSD have been recompressed using "Z" compression. (To uncompress "filename.gz" on a Unix system, type "gunzip filename.gz"; or to uncompress "filename.Z", type "uncompress filename.Z".) a) To obtain data for the immediately preceding month or week: ftp ftp.emc.noaa.gov (user: anonymous; password: your e-mail address) cd /cmb/obs/gts (i.e., web URL: ftp://ftp.emc.ncep.noaa.gov/cmb/obs/gts). The monthly gzip'd file is "gts.sfcmar.gz" is overwritten on the second day of each month. A file "gts.sfcmar.partial.gz" is also available for users that need data for the current month before that month is complete (updated daily, but with a 2-day delay). Contact: Diane Stokes W/NP23, RM 202, WWBG NOAA 5200 Auth Rd Camp Springs, MD 20746-4304 USA phone: +1 301-763-8000 x7268 e-mail: Diane.Stokes@noaa.gov b) To obtain monthly data back to January 1991 (file "nqYYMM.Z", where YY and MM are the last two digits of the year and the month, e.g., "nq9601.Z"): ftp ftp.cdc.noaa.gov (user: anonymous; password: your e-mail address) cd Public/icoads/ncep_obs (i.e., web URL: ftp://ftp.cdc.noaa.gov/Public/icoads/ncep_obs/). The data can also be downloaded via links from this webpage: http://icoads.noaa.gov/nrt.html which also includes monthly plots to help monitor the flow of data. Contact: Scott Woodruff NOAA Earth System Research Laboratory (R/PSD3) 325 Broadway Boulder, CO 80305-3328 USA phone: +1 303-497-6747 e-mail: Scott.D.Woodruff@noaa.gov {3. NCEP GTS surface marine data ASCII format} This is a simple ASCII format with each record 49 characters in length (Table 1). The records are generally sorted by date, but not necessarily by hour (some variations may exist due to the way the data are gathered). Note that present weather, complete cloud elements, and various other minor fields have been omitted from the format. Table 1. The NCEP GTS surface marine data ASCII format. Report type, BUFR file type, and the wind speed indicator are defined in Tables 2-4. NOTE: Distinguishing platform type according to report type or BUFR file type is not always reliable. The most frequent discrepancies involve moored and drifting buoys, for which it is best to examine the third digit in the 5-digit (numeric) buoy ID: generally, a value less than 5 indicates a moored buoy, and a value greater than or equal to 5 indicates a drifting buoy (some Western Pacific C-MAN stations also have 5-digit numeric IDs beginning with "91"; see sec. 4). ------------------------------------------------------------------------------- Column Format Field from to =============================================================================== 1 2 I2 Year (last two digits: 1999 = '99'; 2000 = '00') (UTC) 3 4 I2 Month (UTC) 5 6 I2 Day (UTC) 7 10 F4.2 Hour (UTC) 11 15 F5.2 Latitude (degrees) -- negative = southern hemisphere 16 20 F5.2 Longitude (degrees) -- 0 to 360 West 21 22 I2 or 2I1 Report type; or BUFR file type and wind speed indicator* 23 28 A6 Station ID/ship call sign 29 32 F4.1 Sea level press minus 900.0 (mb) missing=9999 33 35 I3 Wind direction (code=0 or 1-360 degs**) missing=999 36 38 I3 Wind speed (knots) missing=999 39 42 F4.1 Air temp (deg c) missing=9999 43 45 F3.1 Dew point depression (deg c) missing=999 46 46 A1 Cloud cover*** missing=9 47 49 F3.1 Sea surface temp (deg c) missing=999 ------------------------------------------------------------------------------- * Prior to March 1997, positions 21-22 contain Office Note 124 report type (Table 2); starting during March 1997 (see section 4), position 21 contains BUFR file type (Table 3), and position 22 contains the wind speed indicator (Table 4). ** In the original SHIP code, 0 is used for calm and 99 for variable. This format does not have a representation for variable direction, but some code 99 values are mapped by NCEP into code 0 (see technical note k). *** Prior to 4 August 1991 this contains low cloud amount; starting then it contains total cloud amount. ---------- Table 2. Office Note 124 report types. Until March 1997 (see section 4), positions 21-22 contain report type minus 500 (this field may be unreliable as an indicator of platform type; see Table 1). ------------------------------------------------------------------------------- Code Type of Report =============================================================================== ---------Ocean Station------- 521 Fixed (OWS) 522 Moving ship with name 523 Moving ship without name ---------Marine reporting station (MARS)--------- 531 Fixed (Stationary) 532 Moving ---------Quality control data--------- 551 By latitude-longitude* ---------Buoy--------- 561 Moored 562 Drifting ------------------------------------------------------------------------------- * This is listed as a valid "report type" in NCEP documentation, but no reports of this type have ever been encountered. ---------- Table 3. BUFR file type. Starting in March 1997 (see section 4), position 21 contains the type of BUFR file from which the report was obtained (this field may be unreliable as an indicator of platform type; see Table 1). ------------------------------------------------------------------------------- Code BUFR file type =============================================================================== 1 ship 2 drifting buoy 3 moored buoy 4 Coastal-Marine Automated Network (C-MAN) ------------------------------------------------------------------------------- Table 4. The wind speed indicator (starting in March 1997), showing the units in which and the method by which wind speed was originally recorded (WMO code 1855). All winds are stored in this format in knots. The indicator is always missing until 21 October 1997 (the BUFR format at NCEP was expanded to include the wind speed indicator information, if available in the original GTS report, for data decoded into BUFR starting approximately 21 October 1997). ------------------------------------------------------------------------------- Code =============================================================================== 0 meter per second, estimated 1 meter per second, measured (obtained from anemometer) 3 knot, estimated 4 knot, measured (obtained from anemometer) 9 missing ------------------------------------------------------------------------------- {4. Technical notes} a) Original weekly files that existed in some of the earlier years have been consolidated into single monthly files for retention at PSD. b) The original files for January-12 February 1991 were supplemented with some ATLAS buoy data that may be unavailable in the regular NCEP archives for that time period. c) During ftp, a small amount of data was truncated from the original January 1991 file "nq9101.Z," which was also used for construction of the existing 2-degree summary files. Missing data, except possibly the supplementary ATLAS data noted above, have been recovered in the file "nq9101_complete.Z." Similarly, days 1-8 were inadvertently omitted, and days 9-16 duplicated, from the original June 1991 file "nq9106.Z," also used for construction of the existing 2-degree summary files. Missing data have been recovered in the file "nq9106_complete.Z." d) Starting 1 March 1997, data are available processed by NCEP into the BUFR format (files "nq9703.Z," etc.). In addition, overlapping data were processed into NCEP's previous Office Note (ON) 124 format until 19 April ("nq9703_ON.Z" and "nq9704_ON.Z"). Due to concerns about temperature biases in initially available BUFR data, the ON file was used to construct the 2-degree summaries for March 1997. e) In March 1997 we became aware that ON data from moored buoy 52312 reported close to 180 longitude were sporadically impacted by a change in the latitude sign, such that latitudes ~2S were changed to ~2N. Data for March and April 1997 were manually edited (only in "nq9703_ON.Z" and "nq9704_ON.Z"; not in the original "gtsobs.gz" files) by changing positive latitudes to negative with the same magnitude. No previous data were corrected, and the scope of the problem is not fully known. Files "nq9703_ON_52312" and "nq9704_ON_52312" contain all reports from 52312 (before editing, plus reports not subject to editing). f) "Dup/merge" software is used at NCEP in processing the BUFR data to combine partially complete duplicate reports. Duplicate reports were not merged in the ON data; only the most recent report was kept. g) For the period 1545 UTC 30 May to 1530 UTC 2 June 1998, data from drifting and moored buoys (including the TOGA/TAO and PIRATA arrays) were inadvertently not transmitted over the GTS, and thus not received by NCEP. The TAO array provides important coverage in the tropical Pacific. h) US National Data Buoy Center (NDBC) Western Pacific C-MAN (Westpac) stations identified by 5-digit numeric IDs beginning with "91" were equipped with both a primary and secondary data system. Data from the primary system were received through NDBC's normal satellite communication channels identified by the third digit less than 5 (91nxx), and data from the secondary system from Service Argos in the drifting buoy code identified by the third digit greater than or equal to 5 (91mxx). Data from both systems were quality controlled at NDBC and the best data released under 91nxx. 91mxx data appearing in the NCEP data came directly from Service Argos and were not released by NDBC. Also, it is known that secondary wind directions identified under 91852 are erroneous. NDBC suggests that data from all Westpac stations identified as 91mxx be disregarded. Thus starting December 1997 and applicable to the Westpac area (0-16N; 130-170E), drifter reports of the form 91mxx are deleted. i) Storage of temperature values in BUFR was extended to hundredths Kelvin at NCEP applicable to the NRT data starting 17 February 1999. Usage at NCEP of the factor 273.15 for conversion of Celsius temperatures, and rounding to tenths Kelvin precision (until that date the maximum precision available in BUFR), previously lead to some temperature errors of 0.1 degree Celsius. j) During conversion from BUFR into the abbreviated NRT format, reports are not output if the selected NRT meteorological elements are entirely missing. Thus some platforms that are present in the original BUFR format but report only wave data, are not output at all into NRT; ships/buoys occasionally reporting fewer data elements than usual might have a few reports omitted, etc. Thus the BUFR data should be consulted to obtain more complete platform reporting statistics. k) In conjunction with other BUFR changes that became operational approximately 21 October 1997, wind direction in BUFR started being set to code 0 whenever GTS code 99 (variable direction) was reported with a wind speed greater than 0. Apparently BUFR discards code 99 when reported with 0 wind speed, and discarded code 99 under all circumstances prior to that date (since BUFR currently has no way to store the variable code in its direction field). l) Effective with the files for December 2007 and onward, all ship obs have ids masked to the string "MASKST".