[an error occurred while processing this directive] [an error occurred while processing this directive] [an error occurred while processing this directive] [an error occurred while processing this directive] ICOADS Merged NRT


ICOADS Web information page (Wednesday, 11-May-2016 19:04:19 UTC):

Characteristics of a Merged (NCEP-NCEI) Global Telecommunication System (GTS) Near-Real-Time (NRT) Product


Note: This DRAFT webpage contains some incomplete information, which the 2012 termination of ICOADS development provided insufficient time to finalize.

Previous limited comparisons (available here) of the effects of different decoding methods, and between NOAA National Centers for Environmental Information (NCEI; formerly NCDC) and NOAA National Centers for Environmental Prediction (NCEP) GTS data, showed differences between the two sources, which need to be resolved through decoding software changes and further validation. However, those comparisons also strongly suggested benefits from blending GTS sources, including:
  1. Capturing ship callsign (ID) information largely missing since Dec. 2007 in the NCEP GTS data (with most IDs set to "MASKSTID" by NCEP), but more frequently available in the NCEI GTS data. (Note: As part of any improved dual-source NRT processing, the feasibility of retrospective reprocessing of GTS data for ICOADS back to at least 2007 should be considered, to help resolve the embedded ship callsign masking problem.)
  2. NCEI GTS appears to have more Coastal-Marine Automated Network (C-MAN) data (possibly owing to higher temporal resolution data, or different stations).
  3. The representation of Tide Gauge (TG) stations in both GTS sources differs, with only NCEP offering both C-MAN-format and CREX-format stations (whereas NCEI currently has only the C-MAN-format type).
  4. Other studies have indicated that, if blended together, both sources would provide some additional unique (e.g. ship) GTS observations.
  5. However, the role of NCEP's GTS "dup-merge" program needs to be kept in mind when interpreting comparisons, which essentially adds value to the NCEP GTS data by compositing and rescuing some records corrupted in transmission. This can result in fewer total records, but with a higher quality overall.
The NCEP and NCEI data have been assigned to the decks listed in Table 1:

Table 1.
NCEP NCEI  
792 992 GTS Ship Data
793 993 GTS Buoy Data (transmitted in FM 13 "SHIP" code)
794 994 GTS Buoy Data (transmitted in FM 18 "BUOY" code)
795 995 GTS Coastal-Marine Automated Network (C-MAN code) Data [note: may include Tide Gauge stations]
796 996 GTS Miscellaneous (OSV, plat, and rig) Data [note: currently no data are assigned to these decks]
797 997 GTS CREX code [note: includes Tide Gauge stations; deck 797 data begin 3 Oct. 2000, and no deck 997 data are available presently]


  1. Duplicate elimination (dupelim)

    Generally during dupelim processing, all reports are compared with all other reports within a circular buffer, such that a duplicate status (DUPS) flag within each report may be reset (to a higher, but not lower, value) as a result of subsequent matches. Matches therefore refers to all "transient" matches that were made and DUPS results that may be altered later, during this processing. The initial output from dupelim is the "intermediate" file, containing all duplicates (classified either as "certain" or "uncertain"), as flagged by DUPS. From that intermediate output, the final "merged" output is created, generally by removing all but DUPS 0 (unique) or 1 (best duplicate). These are all DUPS entries that appeared (in non-trivial numbers in bold) where "CERTAIN" and "UNCERTAIN" pertain to the "WEATHER ELEMENT" certainty (number of weather elements in common and number different among W, VV, WW, W1, SLP, AT, and SST):
         0 UNIQUE
         1 BEST DUPLICATE
         4 UNCERTAIN WEATHER ELEMENT
         8 CERTAIN WEATHER ELEMENT
         9 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE
        10 CERTAIN WEATHER ELEMENT WITH TIME/SPACE
        11 TIME/SPACE/ID
        12 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID
        13 CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID

    We experimented with several different dupelim algorithms before selecting the current operational version (see this webpage for details). In this version: (i) IDs are required to match except if one or both are generic, (ii) deck 797 stations are included with a two tenths difference allowance for wind speed and SST, (iii) activestations.xml is used to set (NCEI) or reset (NCEP) locations of C-MAN-format and CREX-format receipts, and (iv) DUPS 6 (time/space) is disallowed. Worse duplicate is determined by the following in decreasing significance: (a) larger quality code, (b) larger priority code, (c) generic ID, (d) smaller deck number, (e) smaller source id, or (f) first in sort order. Figure 1-1 shows January 2013 merged product dupelim results based on this algorithm. In panel (A) the middle column excluding deck 797 (orange) is the currently available NCEP-based ICOADS preliminary product. Taking advantage of NCEP's dup-merge processing, the merged product is approximately this preliminary product plus (1) deck 992 (dark grey) ship reports are given a preferred priority to help resolve the deck 792 ship callsign masking problem, and (2) unique and best duplicate deck 995 (dark red) and to a lesser extent deck 797 (orange) tide gauge reports augment the total number of reports (91% more reports than the current preliminary product). In Panel (B) the columns above DUPS 0 and DUPS 1 show the number of unique reports and the number of best duplicates. Columns above DUPS 4 through DUP 13 show the numbers of increasingly certain worse duplicates with almost all being DUPS 13, the most certain. Panel (C) shows small numbers of unexpected deck 792-792 and deck 992-992 matches (red) and deck 797-995 DUPS 11 matches (orange) (see also Examples 1,1.-2.). Panel (D) shows sometimes zero weather elements in common for deck 794-994, 797-995 and 995-995 matches (red) (see Examples 1,3.-4.).

    Figure 1-2 shows dupelim performance in combining the two GTS sources into the final merged product for ships. Also, it shows improvements (i.e. with respect to the existing NCEP-based product) in recovering REAL ship callsigns in the merge (see also Examples 1,1.)

    4 panel

    Figure 1-1. Dupelim skill, in combining NCEI GTS and NCEP GTS data for January 2013.
    (A) Input (NCEI and NCEP) and output ("merge") from dupelim statified by deck (colors).
    (B) Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis).
    (C) DUPS (colors) for the various deck-to-deck matches (horizontal axis).
    (D) Number of weather elements in common (colors) for the various deck-to-deck matches (horizontal axis).

    bar plot

    Figure 1-2. Dupelim skill, in combining NCEP GTS (left-hand bar) and NCEI GTS (middle bar) data for January 2013 into the final merged product (right-hand bar), is shown for platform type (PT) ships. The bars are subdivided by ID patterns; and above the top of each bar, the number of reports in thousands is printed. Also, the following table summarizes the percentages associated with all the different ID patterns:
    REALMASKMASKSTIDSHIPunknown
    NCEP1.0%0.2%98.7%0.1%0.0%
    NCEI61.9%22.8%0.0%15.1%0.2%
    merge70.1%26.1%1.7%2.0%0.2%

    Examples 1

    These are examples to accompany Figs. 1-1 and 1-2, in each case showing the IMMA-format reports, followed in some instances by the corresponding NCEP supplementary data (SUPD) for those reports. DUPS is at the far right, with noteworthy fields highlighted by red lettering.

    1. NCEP deck 792 and NCEI deck 992 ship duplicates. Effective 8 May 2012 at 0000 UTC callsign masking was implemented for all Meteorological Service of Canada (MSC) Automatic Voluntary Observing Ship (AVOS) reports. MSC sends messages in real-time with callsigns replaced with SHIP and also sends these messages with unmasked callsigns to National Meteorological Centres.
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1 100 5070 23250 13 000   2MASKSTID   01604 15      102961  4   49        51          A               165 129 7792103 5102
    2013 1 1 100 5070 23250 13 000   2MASKSTID   01604 15      102961  4   49        51          A               165 129 7792103 5102
    2013 1 1 100 5070 23250 130000   2SHIP       01604 15      102961  40  49        51          A               165 129 7992114 5102
    2013 1 1 100 5070 23250 130000   1CG2958     01604 15      102961  40  49        51          A               165 129 7992114 5 12
    BBXX01010120130112SNVD70KWBC010100RRB////// XXXX 01014 99507 71275 46/// /1603 10049 20051 40296 51004 7//// 8//// 22200 0//// 2////
    BBXX01010120130112SNVD70KWBC010100RRB////// XXXXXX 01014 99507 71275 46/// /1603 10049 20051 40296 51004 7//// 8//// 22200 0//// 2////
    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 4870 23650 13 000   2MASKSTID   03604 10      102861 11   37        36          A               165 16583792103 5102
    2013 1 1   0 4870 23650 13 000   2MASKSTID   03604 10      102861 11   37        36          A               165 16583792103 5102
    2013 1 1   0 4870 23650 130000   2SHIP       03604 10      102861 110  37        36          A               165 16583992114 5102
    2013 1 1   0 4870 23650 130000   1CGBR       03604 10      102861 110  37        36          A               165 16583992114 5 12
    BBXX01010120130011SMVD70KWBC010000RRD////// XXXX 01004 99487 71235 46/// /3602 10037 20036 40286 51011 7//// 8//// 22200 0//// 2////
    BBXX01010120130012SMVD70KWBC010000RRE////// XXXX 01004 99487 71235 46/// /3602 10037 20036 40286 51011 7//// 8//// 22200 0//// 2////
    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1111200 3100 19160 13 064   2MASKSTID   02004 10 98 21101360 13  1900 170  100 1 2001   9               165 19718792103 5102
    2013 1111200 3100 19160 13 064   2MASKSTID   02004 10 98 21101360 13  1900 170  100 1 2001   9               165 19718792103 5102
    2013 1111200 3100 19160 132064   2SHIP       02004 10 98 21101360 130 1900 170  100 1 2001   9               165 19718992114 5 12
    BBXX01011120131252SMVD70KWBC111200RRM////// XXXX 11124 99310 71684 41998 12002 10190 20100 40136 50013 70211 8//// 91200 22264 00200 2//// 3//// 4//// 80170
    BBXX01011120131259SMVD62RJTD111200B7H////// XXXXX 11124 99310 71684 41998 12002 10190 20100 40136 50013 70211 8//// 91200 22264 00200 2//// 3//// 4//// 80170
    

    1. NCEP deck 797 and NCEI deck 995 tide gauge duplicates, DUPS 11 (time/space/id). DUPS was less certain because only a two tenths difference allowance was permitted for NCEP deck 797 wind speed and SST.
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 1797 29295 1225     5MGIP4      5 37  18      10174      253                                    165 279777971031611
    2013 1 1   0 1797 29295 1325     5MGIP4      0 404 21      101743 100 253           1 272    A               165 2797799511416 1
                                                                                                                                    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1  50 4791 23536 1225     5LAPW1                                                55                    165 165747971031611
    2013 1 1  50 4791 23536 1325     5LAPW1      0 704 10      10274    0  40           1  58    A               165 1657499511416 1
    

    1. NCEP deck 794 and NCEI deck 994 drifting buoys duplicates, zero weather elements in common. Not complement duplicates.
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 8350 30056 1225     347545                                                                      165  2839994114 713
    2013 1 1   0 8350 30056 1225     347545                                                                      165  2839794103 7 1
    ZZYY01010120130011SSVX02CWAO010000B7H////// 47545 01013 0000/ 783498 059435 6113/ 444 20130 01013 0000/ 50001 80136 80001 9////
                                                                                                                                    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 8331 31013 1225     347537                                                                      165  2939994114 713
    2013 1 1   0 8331 31013 1225     347537                                                                      165  2939794103 7 1
    ZZYY01010120130011SSVX02CWAO010000B7H////// 47537 01013 0000/ 783308 049865 6113/ 444 20130 01013 0000/ 50001 80134 80001 9////
    

    1. NCEP deck 797 and NCEI deck 995 tide gauge duplicates, zero weather elements in common. Complement duplicates where in the first set the selection of an NCEP deck 795 report eliminated only the units digit (in deck 797) of wind direction (degraded to tens of degrees in all but the CREX format). In the second and third sets winds were eliminated, and in the fourth set A, PPP and SST were eliminated.
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 4690 23589 1225     5WPTW1      5 81  69      10275       32                                    165 165647971031613
    2013 1 1   0 4690 23589 1325     5WPTW1      0 804 67                                        A               165 165649951141613
    2013 1 1   0 4690 23589 1325     5WPTW1                    102752 100  32           1  84    A               165 165649951141613
    2013 1 1   0 4690 23589 1325     5WPTW1      0 804 67      102752 10   32           1  84                    165 1656479510316 1
    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 4791 23536 1225     5LAPW1      5 57  21      10273       42                                    165 165747971031613
    2013 1 1   0 4791 23536 1325     5LAPW1      0 604 21                                        A               165 165749951141613
    2013 1 1   0 4791 23536 1325     5LAPW1                    102733 190  42           1  62    A               165 1657499511416 1
    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 4813 23656 1225     5PTAW1      5196  21      10284       37                                    165 165837971031613
    2013 1 1   0 4813 23656 1325     5PTAW1      02004 21                                        A               165 165839951141613
    2013 1 1   0 4813 23656 1325     5PTAW1                    10284    0  37           1  86    A               165 165839951141613
    2013 1 1   0 4813 23656 1325     5PTAW1                    102843 170  37           1  86    A               165 1658399511416 1
    
                             A                                               W    D                              A A               D
                    L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
       Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
       R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
    2013 1 1   0 4855 23699 1325     5FRDW1      03204  5               0  28                    A               165 165839951141613
    2013 1 1   0 4855 23699 1325     5FRDW1                    102843 110               1  82    A               165 165839951141613
    2013 1 1   0 4855 23699 1225     5FRDW1      5322   5      10284       28                                    165 1658379710316 1
    

  2. C-MAN/Tide Gauge

    We are expanding ICOADS to include meteorological measurements from Tide Gauge (TG) stations, as well as considering the additional possibility (linked with extensive IMMA format changes planned under IVAD) to eventually extend IMMA to include TG water level data. Some TG data are transmitted over the GTS in WMO's CREX format, others are transmitted in the C-MAN format (see Table 2-1 for a breakdown of C-MAN-format and CREX-format receipts by station categories, and Fig. 2-1 for the geographic distribution of all the deck 795/797 stations listed in Table 2-1). The NOAA National Data Buoy Center (NDBC) manages data from many fixed-location near-coastal stations that are transmitted over GTS in C-MAN format. Possibly originally designed with the view to save GTS transmission costs, this C-MAN format lacks latitude and longitude (thus downstream users need to add location information independently; a longstanding issue currently under discussion with NDBC). Some of the C-MAN-format stations are "official" NDBC-operated Coastal-Marine Automated Network (C-MAN) stations, delayed-mode (quality checked) data from which are archived by the NOAA National Oceanographic Data Center (NODC). However a variety of additional station types, including but not limited to TG stations operated and archived in delayed-mode by the NOAA National Ocean Service (NOS), also utilize the C-MAN format for circulating data on the GTS.

    Table 2-1. Embedded plot: amounts of ICOADS preliminary (NCEP-based) data for Feb. 2012 by deck (792-795), plus CREX data (deck 797; not presently utilized for the ICOADS preliminary product). The left side of the table shows the breakdown of total Feb. 2012 NCEP GTS receipts (numbers of reports) in decks 795 and 797 according to station categories, as listed by NDBC here: http://www.ndbc.noaa.gov/to_station.shtml (webpage accessed on 15 May 2012; webpage stated: "Page last modified: July 16, 2007") and where "(blank)" indicates ID not found. The right side of the table similarly shows the breakdown of total receipts: (i) for the only two individual station IDs OCIM2 and WPTW1 (both NOS TG stations) that overlap between decks 795 and 797, and (ii) for each individual ID within the complete set of unrecognized "(blank)" IDs. These "(blank)" IDs also were not found here: http://www.ndbc.noaa.gov/activestations.xml (created="2012-05-15T20:40:02UTC"). Presently, setting of platform type (PT) within decks 795 and 995 (also decks 797 and 997) is not well resolved (and further coordination may be needed with NDBC), since possibly only the station category "National Data Buoy Center Stations" should be set to PT=13 (13 - Coastal-Marine Automated Network (C-MAN) (NDBC operated)). For January 2013 data forward, however, we have been utilizing operational downloads of the continuously updated activestations file to make some improvements in the assignment of PT. station inventory

    station map

    Figure 2-1. Map showing the geographical coverage of deck 795 and 797 Feb. 2012 NCEP data, as listed in Table 2-1, stratified according to these categories: C-MAN-format receipts (green), CREX-format receipts with IDs listed by NDBC (blue), and CREX-format receipts with IDs not listed by NDBC (red; "(blank)" in Table 2-1).

    Table 2-2. Number of extant CREX code data elements for 15 February 2012 in NCEP BUFR (file sfcshp.ibm.20120215.gz).
    BUFR number and description IMMA abbrev.NCEP extant
    0 01 198 REPORT IDENTIFIER ID 4001
    0 02 001 TYPE OF STATION 4001
    0 04 001 YEAR YR 4001
    0 04 002 MONTH MO 4001
    0 04 003 DAY DY 4001
    0 04 004 HOUR HR 4001
    0 04 005 MINUTE HR 4001
    0 04 015 TIME INCREMENT IN MINUTES 3977
    0 04 025 TIME PERIOD OR DISPLACEMENT 3977
    0 04 200 YEAR - TIME OF RECEIPT 7286
    0 04 201 MONTH - TIME OF RECEIPT 7286
    0 04 202 DAY - TIME OF RECEIPT 7286
    0 04 203 HOUR - TIME OF RECEIPT 7286
    0 04 204 MINUTE - TIME OF RECEIPT 7286
    0 05 002 LATITUDE (COARSE ACCURACY) LAT 4001
    0 06 002 LONGITUDE (COARSE ACCURACY) LON 4001
    0 07 001 HEIGHT OF STATION 4001
    0 08 202 RECEIPT TIME SIGNIFICANCE 7286
    0 10 051 PRESSURE REDUCED TO MSL SLP 3219
    0 11 001 WIND DIRECTION D 2246
    0 11 002 WIND SPEED W 2189
    0 12 101 TEMPERATURE/DRY BULB TEMPERATURE AT 3004
    0 22 038 TIDAL ELEV WITH RESPECT TO CHART 7954
    0 22 043 SEA TEMPERATURE SST 2966
    0 22 120 TIDE STATION AUTOMATED WATER DATA CHECK 3977
    0 22 121 TIDE STATION MANUAL WATER DATA CHECK 3977
    0 22 122 TIDE STATION AUTOMATED METEOROLOGICAL DATA CHECK 3309
    0 22 123 TIDE STATION MANUAL METEOROLOGICAL DATA CHECK 3309
    0 33 195 SDMEDIT/QUIPS QUALITY MARK FOR WIND 1
    0 33 215 CORRECTED REPORT INDICATOR 4001
    0 35 021 BULLETIN BEING MONITORED (TTAAii) 7286
    0 35 022 BULLETIN BEING MONITORED (YYGGgg) 7286
    0 35 023 BULLETIN BEING MONITORED (CCCC) 7286
    0 35 194 BULLETIN BEING MONITORED (BBB) 7286
    0 35 195 CHANNEL SEQUENCE NUMBER 7286
    0 58 008 RAW REPORT STRING SUPD 75532


  3. Coast Guard coastal stations

    Since mid-February 2012, some Coast Guard coastal station reports have been included in the NCEP BUFR files (BUFR subtype NC001007; presently about 200 such reports per day). Following are examples of the raw report strings (see format discription), and Figure 3-1 and Table 3-1 provide additional information. Because of their near-coastal nature and limited geographic coverage, these data are not considered a priority for ICOADS at this time.
    L27 C10 / CALM / FLAT / 58 / 59 / AVALON HARBOR
    L58 CYRW-6 / SW12 / 0508 / 57 / 57 / 1016 / MX6 MISSION BEACH
    L79 C10 / SW10 / 0408 / 56 / 61 / W/MX5 CHANNEL ISLANDS HBR
    9L0 CY10 / S05 / / 59 / 58 / 1016 / DANA POINT
    

    station map

    Figure 3-1. Map showing the geographical coverage of May 2012 NCEP Coast Guard coastal stations data.

    Table 3-1. Number of extant Coast Guard station data elements for 15 May 2012 in NCEP BUFR (file sfcshp.ibm.20120515.gz).
    BUFR number and description IMMA abbrev.NCEP extant
    0 01 198 REPORT IDENTIFIER ID208
    0 02 002 TYPE OF INSTRUMENTATION FOR WIND MEASUREMENT WI208
    0 04 001 YEAR YR208
    0 04 002 MONTH MO208
    0 04 003 DAY DY208
    0 04 004 HOUR HR208
    0 04 005 MINUTE HR208
    0 04 200 YEAR - TIME OF RECEIPT 208
    0 04 201 MONTH - TIME OF RECEIPT 208
    0 04 202 DAY - TIME OF RECEIPT 208
    0 04 203 HOUR - TIME OF RECEIPT 208
    0 04 204 MINUTE - TIME OF RECEIPT 208
    0 05 002 LATITUDE (COARSE ACCURACY) LAT208
    0 06 002 LONGITUDE (COARSE ACCURACY) LON208
    0 07 001 HEIGHT OF STATION 208
    0 08 202 RECEIPT TIME SIGNIFICANCE 208
    0 10 004 PRESSURE 14
    0 10 051 PRESSURE REDUCED TO MSL SLP11
    0 11 001 WIND DIRECTION D208
    0 11 002 WIND SPEED W208
    0 11 041 MAXIMUM WIND SPEED (GUSTS) 2
    0 12 101 TEMPERATURE/DRY BULB TEMPERATURE AT73
    0 20 001 HORIZONTAL VISIBILITY VV100
    0 20 003 PRESENT WEATHER WW33
    0 20 010 CLOUD COVER (TOTAL) 89
    0 20 011 CLOUD AMOUNT N89
    0 22 003 DIRECTION OF SWELL WAVES SD12
    0 22 011 PERIOD OF WAVES WP61
    0 22 021 HEIGHT OF WAVES WH62
    0 22 043 SEA TEMPERATURE SST48
    0 22 073 MAXIMUM WAVE HEIGHT 5
    0 33 215 CORRECTED REPORT INDICATOR 208
    0 35 021 BULLETIN BEING MONITORED (TTAAii) 208
    0 35 022 BULLETIN BEING MONITORED (YYGGgg) 208
    0 35 023 BULLETIN BEING MONITORED (CCCC) 208
    0 35 194 BULLETIN BEING MONITORED (BBB) 208
    0 35 195 CHANNEL SEQUENCE NUMBER 208
    0 58 008 RAW REPORT STRING SUPD1082


  4. GTS header information

    Since 2003 when NCEI began providing marine GTS data for ICOADS, an NCEI-defined 18-character header has been included, which (together with a blank separating character) prefixes the originally reported GTS message stored in SUPD. For the NCEP GTS data, this header has been replicated in SUPD. The header contains selected GTS bulletin header information and data receipt timestamp information (note: derived independently by NCEI and NCEP, and not part of the GTS bulletin header). Effective with January 2013 data, the NCEP header has been extended to 43 characters, to include additional potentially useful "abbreviated heading" (ref.: WMO No. 386) and quality mark information; however, a planned similar extension to 37 characters of the NCEI header has been postponed indefinitely owing to unresolved technical issues at NCEI. Table 4-1 documents the content of the 18- and 37(43)-character header fields.

    Table 4-1. The field content of the 18-character header (e.g. "CMAN01042020122100") in SUPD (i.e. fields 1-7 in this table), extended by the addition of fields 8-11(17) to create the new 37(43)-character header (e.g. "CMAN01042020122100SXUS22KWNB202200RRC//////"). Columns 3-7 are similar receipt timestamp information, but derived independently by NCEI and NCEP, and the information may differ, but generally these differences are restricted to minute, for a given original GTS report. The NCEP 18-character header used observation instead of receipt time.
    No.FieldE.g. segmentNCEINCEP
     1MjMjMjMjCMAN BBXX, ZZYY or CMANBBXX, ZZYY, CMAN or CREX
     2"01" 01 sequence number for split messages1 "01"
     3Month 04 timestamp month2MONTH - TIME OF RECEIPT
     4Day 20 timestamp day2DAY - TIME OF RECEIPT
     5Year 2012 timestamp year2YEAR - TIME OF RECEIPT
     6Hour 21 timestamp hour2HOUR - TIME OF RECEIPT
     7Minute00 timestamp minute2MINUTE - TIME OF RECEIPT
     8TTAAiiSXUS22WMO abbreviated heading: data type/geographic/time/etc. designatorssame
     9CCCC KWNB WMO abbreviated heading: originating/compiling station indic.same
    10YYGGgg202200WMO abbreviated heading: international date-time groupsame
    11BBB RRC WMO abbreviated heading: indic. for corrections/amendments/etc.same
    12QMAT / n/aSDMEDIT/QUIPS QUALITY MARK FOR TEMPERATURE3
    13QMDD / n/aSDMEDIT/QUIPS QUALITY MARK FOR MOISTURE3
    14QMPR / n/aSDMEDIT/QUIPS QUALITY MARK FOR PRESSURE3
    15QMST / n/aSDMEDIT/QUIPS QUALITY MARK FOR SEA SURFACE TEMPERATURE3
    16QMWH / n/aSDMEDIT/QUIPS QUALITY MARK FOR WAVE HEIGHT3
    17QMWN / n/aSDMEDIT/QUIPS QUALITY MARK FOR WIND3
    1 obsolete; unused
    2 GTS file creation date; could possibly change if the GTS file has to be recreated. The NCEP time of receipt information is determined independently, and therefore may differ, generally to a small extent, such as shown in the following examples.
    3 NCEP 43-character header was extended by quality marks, or "/" is assigned in our processing if the quality marks were missing in BUFR (note: more information about NCEP's processing that sets the quality marks is provided here).

    Examples of NCEI and NCEP GTS reports (in that order) with 37(43)-character headers:
    CMAN01042020122100SXUS22KWNB202100RRX 20214 NPDW3 46/// /3420 10057 40149 92110 333 91226 555 11020 22021
    CMAN01042020122149SXUS22KWNB202100RRX////// 20214 NPDW3 46/// /3420 10057 40149 92110 333 91226 555 11020 22021
                    ^^
    

  5. GTS messages — systematic differences

    (using coads_20120205_gts_jan.gz)

    Possible explanations:
    1. NCEI in their earlier ingest process
    2. centers that are providing NCEI with the GTS data
    3. Local User Terminals vs. main GTS receipt stations

    NCEP cancels (Xs) out the ID in the original messages.

    • NCEI BBXX Section 3 and Section 5 unreadable because of "/"s.
      NCEI 99 0 BBXX               44040 01001 99410 70736 46/// /2103 10092 20013 40182 92345 22200 1//00 70001 333/91204 555/11041 22046
      NCEP 99 0 BBXX               44040 01001 99410 70736 46/// /2103 10092 20013 40182 92345 22200 1//00 70001 333 91204 555 11041 22046
                                                                                                                    ^         ^
      
    • BBXX difference with no effect.
      NCEI 99 0 BBXX               WAHV 01063 99376 10105 43599 52930 10160 2013/ 40208 53005 82606 22274 00162 20905 80140
      NCEP 99 0 BBXX               XXXX 01063 99376 10105 43599 52930 10160 20130 40208 53005 82606 22274 00162 20905 80140
                                                                                ^
      
    • NCEI CMAN hour in group YYGGi not rounded up when minutes in group 9GGgg are 30.
      NCEI 99 0 CMAN               01054 ANCF1 46/// /0000 10157 40228 90530 222// 00189 333 91201 555 TIDE1123
      NCEP 99 0 CMAN               01064 ANCF1 46/// /0000 10157 40228 90530 222// 00189 333 91201 555 TIDE1123
                                      ^
      
    • NCEI ZZYY Section 3 deleted.
      NCEI 99 0 ZZYY               23010 18012 1200/ 303987 080511 6112/ 444 20120 18012 1047/ 521//
      NCEP 99 0 ZZYY               23010 18012 1200/ 303987 080511 6112/ 33311 88872 20001 32865 20020 32850 43405 20040 32822 43428 20060 32388 43508 \
      20080 32050 20100 31946 20120 31871 20140 31764 20180 31448 20300 31143 20500 30974 444 20120 18012 1047/ 521//
                                                                         ^^^
      
    • NCEI ZZYY Section 4 deleted.
      NCEI 99 0 ZZYY               25615 10012 2259/ 785271 169671 6113/ 11119 49991 57005 22219 01097
      NCEP 99 0 ZZYY               25615 10012 2259/ 785271 169671 6113/ 11119 49991 57005 22219 01097 444 20130 10012 2054/ 50101 80136 8//// 9/015
                                                                                                       ^^^
      
    • ZZYY latitude longitude differences.
      NCEI 99 0 ZZYY               21210 02012 0700/ 138112 146475 6111/ 11119 10106 29082 49971 22219 00000 444 20110 02012 0749/ 522//
      NCEP 99 0 ZZYY               21210 02012 0700/ 138109 146476 6111/ 11119 10106 29082 49971 22219 00000 444 20110 02012 0749/ 522//
                                                         ^^      ^
      
    • ZZYY differences with no effect.
      NCEI 99 0 ZZYY               46642 12012 0359/ 752500 130970 6110/ 11119 40272 57015 22219 00074 444 201// 12012 0406/ 80080 80005
      NCEP 99 0 ZZYY               46642 12012 0359/ 75250/ 13097/ 6110/ 11119 40272 57015 22219 00074 444 201// 12012 0406/ 80080 80005
                                                          ^      ^
      

  6. Unresolved issues

    Platform type (PT) and ID indicator (II)

    1. One issue for eventual ICOADS delayed-mode processing is that WMO Pub. 47 contains the REAL callsign, not a masked ID. In the original GTS data (FM 13), masked IDs generally take two forms: (i) the generic ID "SHIP," or (ii) a ship-specific alias referred to as "MASK" (see this article for additional information). JCOMMOPS maintains a secured database of REAL vs. masked callsigns, which could be used (resources permitting and assuming access was authorized by WMO) to sort out the problem. As an additional complication, since Dec. 2007, most NCEP BUFR ship reports (deck 792) have been uniformly remasked by NCEP to "MASKSTID," and with "X...X" overwriting any real FM 13 callsign information. (Note: MASKSTID is considered a generic ID, in addition to SHIP, BUOY, and some other non-ship forms.)

    2. MASK is presently classified incorrectly as callsign in the ICOADS preliminary product (see Figure 6-1B), partly because we have no definitive list of what are all legal MASK aliases, and it is unknown whether any might resemble REAL callsigns (see also Table 6-1).

    3. More work is needed to ensure that official NDBC C-MAN stations, tide gauge stations, etc. are properly assigned platform type (PT) and ID indicator (II) (possibly using additional information available in: http://www.ndbc.noaa.gov/activestations.xml). As part of this issue, II configurations currently defined in the IMMA format, including "5 - Coastal-Marine Automated Network (C-MAN) ID (US NDBC operated)" may need adjustment/amplification.

    DCK by II

    Figure 6-1. April input (A) ID indicator (II) (horizontal axis) stratified by deck (colors); (B) deck 992 stratified by estimated improved ID categories (see Table 6-1). (Note: following is the list of IDs currently defined for contemporary data types as generic: SHIP, MASKSTID, BUOY, DRIB, RIGG, PLAT, and NNXX.)

    Table 6-1. April input deck 992 IDs categorized as MASK, i.e. identified simply as a pattern of five letters and two numbers, and their frequencies.
    generic unique IDs

Appendix A: Changes in the Processing of Relative Humidity (RH) and Dew Point Temperature (DPT) in NCEP and NCEI GTS Data

While both the FM 13 SHIP code and the FM 18 BUOY code have the capacity to report RH, the IMMA format (version IMMA0) lacked a regular field space for relative humidity (RH). Recently however, an RH field was added to IMMA1. Presently for FM 13 (apparently applicable also for FM 18), the reporting of DPT (TdTdTd; to tenths Celsius) or RH (UUU; whole percent) is governed by Regulation 12.2.3.3.1 (ref. WMO-No. 306, Manual on Codes):
    "Under unusual conditions, when the dew-point temperature is temporarily unavailable (e.g. because of instrument failure) but relative humidity is available, the group 29UUU shall replace the group 2snTdTdTd. Every attempt shall first be made, however, to convert relative humidity to dew-point temperature, and the relative humidity included only as a last resort."

For NCEP GTS data starting from March 1997, Table A1 provides a timeline of processing changes impacting RH and DPT. Similarly, NCEI data extending back to at least Nov. 2003 that have been already translated into IMMA are impacted by some such changes, but those data have not yet been utilized for ICOADS and may be reconverted to IMMA1 so as to alleviate processing inconsistencies.

Figure A1 (scatter plot) and Figure A2 (distribution plot) show how at mid reported humidity values recomputed humidity values are slightly larger.

Table A1. Timeline of decoding and other processing changes in NCEP GTS data applicable to RH, DPT, and RF (the trimming flag for RH, which is also stored in IMMA but set during a later QC processing step). The algorithm used for computing missing DPT conforms with ICOADS preconditioning rules (e.g. here), and the algorithm used for computing missing RH conforms with that used for ICOADS monthly summary processing, i.e. the codes available here.
Effective monthProcessing modification
Mar. 1997 IMMA fields (including DPT) decoded from original GTS messages, but reported RH values omitted.
Oct. 1999 IMMA fields (including DPT) translated from BUFR instead, but reported RH values still omitted.
Jan. 2005 When available instead of DPT, RH was translated from BUFR and converted to DPT for storage in IMMA0. Then RH was temporarily recomputed from DPT (and reported air temperature) only for setting RF.
TBD Transition to IMMA1 allows direct storage of RH. Now when RH is reported, missing DPT is computed from RH; conversely, when DPT is reported, missing RH is computed from DPT (and associated fields are set to indicate which field was originally reported versus computed). Thus it is no longer necessary to temporarily recompute RH for setting RF. However, the RH values being flagged represent a mixture of originally reported and computed RH data, between which systematic differences may exist.


RH scatter

Figure A1. Scatter plot of September 2012 RH measured (originally in whole %) vs. the same RH recomputed (i.e. in accordance with the procedure effective from Jan. 2005 in Table A1, such that RH was translated from BUFR and converted to DPT, then RH was temporarily recomputed from DPT for setting RF).
RH histogram

Figure A2. Distribution plot of September 2012 RH measured vs. the same RH recomputed (i.e. in accordance with the procedure effective from Jan. 2005 in Table A1, such that RH was translated from BUFR and converted to DPT, then RH was temporarily recomputed from DPT for setting RF).

Appendix B: Great Lakes CREX data LST time zone

For local operational reasons, CREX bulletins for the Great Lakes were going out (through 14Z 24 Oct 2012) in LST time zone. This accounts for some of the large data differences seen on this webpage (in its Examples 3), i.e. the apparent CREX duplicates are displaced by five or six hours, since any reports with LST times were interpreted by our then current translation programs as UTC times.

time gap
Figure B1. 24 October 2012 NCEP Great Lakes data: year, month, day, hour in hundredths (horizontal axis) vs. number of CREX bulletins in Central Standard Time through 8 a.m. (thin red), in Eastern Standard Time through 9 a.m. (thin blue), and in UTC after 14Z (number of non-CREX with same IDs solid blue).

map
Figure B2. Great Lakes tide gauge locations by time zone (colors: red Central, blue Eastern).

Table B1. Great Lakes tide gauge IDs (ref: NOS_SHEF_CREX_05-03-12.xls). CREX ID MI220 appears twice but in the NCEP data has the same location as SHEF ID FWNM4 (42.30N 83.09W).
NOS #CREX IDSHEF IDLST
9087044IL010CMTI2CST
9087057WI010MLQW3CST
9087068WI015KWNW3CST
9087072WI020SBCW3CST
9087079WI040GBWW3CST
9087088 MNMM4CST
9099064MN100DULM5CST
9099090MN110GDMM5CST
NOS #CREX IDSHEF IDLSTNOS #CREX IDSHEF IDLSTNOS #CREX IDSHEF IDLST
8311030NY034OBGN6EST9052058NY041RCRN6EST9075035MI016ESVM4EST
8311062NY036ALXN6EST9052076NY042OCTN6EST9075065 LPNM4EST
9014070MI220AGCM4*EST9063007NY043NGAN6EST9075080MI019MACM4EST
9014080MI250SCRM4EST9063009NY046AMFN6EST9075099MI100DTLM4EST
9014087MI260PHXM4EST9063012 NIAN6EST9076024 RCKM4EST
9014090MI255MBRM4EST9063020NY050BUFN6EST9076027 WNEM4EST
9014096MI256DUPM4EST9063028NY060PSTN6EST9076033 LTRM4EST
9014098MI245FTGM4EST9063038PA012EREP1EST9076060MI110USSM4EST
9034052MI230SCSM4EST9063053OH010FAIO1EST9076070MI120SWPM4EST
9044020MI200GRTM4EST9063063OH020CNDO1EST9087023MI025LDTM4EST
9044030MI210WDTM4EST9063079OH025MRHO1EST9087031MI290HLNM4EST
9044036MI220FWNM4EST9063085OH030THRO1EST9087096MI020PNLM4EST
9044049MI221WMPM4EST9063090MI010FPPM4EST9099004MI130PTIM4EST
9052000 CAVN6*EST9075002MI257LKPM4EST9099018MI140MCGM4EST
9052030NY040OSGN6EST9075014MI013HRBM4EST9099044MI145OGOM4EST
* See Figure B3 for possible corresponding CREX ID.

map
Figure B3. Plot of CREX format IDs 3 Oct. 2000 through 31 Dec. 2012 after removing IDs in Table B1; with only CREX IDs MI270, MI240 and NY038 remaining in the Great Lakes region. MI240 and NY038 may have Table B1 SHEF ID counterparts:
IDNCEP locationNOS_SHEF_CREX_05-03-12.xls location
MI27044.66N 83.29W
MI24042.62N 82.53W
  AGCM442.62N 82.57W42.621000N 82.526900W
NY03844.13N 76.34W
  CAVN644.13N 76.33W 44.130190N 76.331970W


[(none)][ICOADS]

U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads@noaa.gov
Updated: May 11, 2016 19:04:19 UTC
http://icoads.noaa.gov/merge.html