ICOADS Web information page (Wednesday, 11-May-2016 19:04:19 UTC):

Characteristics of a Merged (NCEP-NCEI) Global Telecommunication System (GTS) Near-Real-Time (NRT) Product

Note: This DRAFT webpage contains some incomplete information, which the 2012 termination of ICOADS development provided insufficient time to finalize.

Previous limited comparisons (available here) of the effects of different decoding methods, and between NOAA National Centers for Environmental Information (NCEI; formerly NCDC) and NOAA National Centers for Environmental Prediction (NCEP) GTS data, showed differences between the two sources, which need to be resolved through decoding software changes and further validation. However, those comparisons also strongly suggested benefits from blending GTS sources, including:

Capturing ship callsign (ID) information largely missing since Dec. 2007 in the NCEP GTS data (with most IDs set to "MASKSTID" by NCEP), but more frequently available in the NCEI GTS data. (Note: As part of any improved dual-source NRT processing, the feasibility of retrospective reprocessing of GTS data for ICOADS back to at least 2007 should be considered, to help resolve the embedded ship callsign masking problem.)
NCEI GTS appears to have more Coastal-Marine Automated Network (C-MAN) data (possibly owing to higher temporal resolution data, or different stations).
The representation of Tide Gauge (TG) stations in both GTS sources differs, with only NCEP offering both C-MAN-format and CREX-format stations (whereas NCEI currently has only the C-MAN-format type).
Other studies have indicated that, if blended together, both sources would provide some additional unique (e.g. ship) GTS observations.
However, the role of NCEP's GTS "dup-merge" program needs to be kept in mind when interpreting comparisons, which essentially adds value to the NCEP GTS data by compositing and rescuing some records corrupted in transmission. This can result in fewer total records, but with a higher quality overall.

The NCEP and NCEI data have been assigned to the decks listed in Table 1:

Table 1.

NCEP	NCEI
792	992	GTS Ship Data
793	993	GTS Buoy Data (transmitted in FM 13 "SHIP" code)
794	994	GTS Buoy Data (transmitted in FM 18 "BUOY" code)
795	995	GTS Coastal-Marine Automated Network (C-MAN code) Data [note: may include Tide Gauge stations]
796	996	GTS Miscellaneous (OSV, plat, and rig) Data [note: currently no data are assigned to these decks]
797	997	GTS CREX code [note: includes Tide Gauge stations; deck 797 data begin 3 Oct. 2000, and no deck 997 data are available presently]

Duplicate elimination (dupelim)

Generally during dupelim processing, all reports are compared with all other reports within a circular buffer, such that a duplicate status (DUPS) flag within each report may be reset (to a higher, but not lower, value) as a result of subsequent matches. Matches therefore refers to all "transient" matches that were made and DUPS results that may be altered later, during this processing. The initial output from dupelim is the "intermediate" file, containing all duplicates (classified either as "certain" or "uncertain"), as flagged by DUPS. From that intermediate output, the final "merged" output is created, generally by removing all but DUPS 0 (unique) or 1 (best duplicate). These are all DUPS entries that appeared (in non-trivial numbers in bold) where "CERTAIN" and "UNCERTAIN" pertain to the "WEATHER ELEMENT" certainty (number of weather elements in common and number different among W, VV, WW, W1, SLP, AT, and SST):

     0 UNIQUE
     1 BEST DUPLICATE
     4 UNCERTAIN WEATHER ELEMENT
     8 CERTAIN WEATHER ELEMENT
     9 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE
    10 CERTAIN WEATHER ELEMENT WITH TIME/SPACE
    11 TIME/SPACE/ID
    12 UNCERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID
    13 CERTAIN WEATHER ELEMENT WITH TIME/SPACE/ID

We experimented with several different dupelim algorithms before selecting the current operational version (see this webpage for details). In this version: (i) IDs are required to match except if one or both are generic, (ii) deck 797 stations are included with a two tenths difference allowance for wind speed and SST, (iii) activestations.xml is used to set (NCEI) or reset (NCEP) locations of C-MAN-format and CREX-format receipts, and (iv) DUPS 6 (time/space) is disallowed. Worse duplicate is determined by the following in decreasing significance: (a) larger quality code, (b) larger priority code, (c) generic ID, (d) smaller deck number, (e) smaller source id, or (f) first in sort order. Figure 1-1 shows January 2013 merged product dupelim results based on this algorithm. In panel (A) the middle column excluding deck 797 (orange) is the currently available NCEP-based ICOADS preliminary product. Taking advantage of NCEP's dup-merge processing, the merged product is approximately this preliminary product plus (1) deck 992 (dark grey) ship reports are given a preferred priority to help resolve the deck 792 ship callsign masking problem, and (2) unique and best duplicate deck 995 (dark red) and to a lesser extent deck 797 (orange) tide gauge reports augment the total number of reports (91% more reports than the current preliminary product). In Panel (B) the columns above DUPS 0 and DUPS 1 show the number of unique reports and the number of best duplicates. Columns above DUPS 4 through DUP 13 show the numbers of increasingly certain worse duplicates with almost all being DUPS 13, the most certain. Panel (C) shows small numbers of unexpected deck 792-792 and deck 992-992 matches (red) and deck 797-995 DUPS 11 matches (orange) (see also Examples 1,1.-2.). Panel (D) shows sometimes zero weather elements in common for deck 794-994, 797-995 and 995-995 matches (red) (see Examples 1,3.-4.).

Figure 1-2 shows dupelim performance in combining the two GTS sources into the final merged product for ships. Also, it shows improvements (i.e. with respect to the existing NCEP-based product) in recovering REAL ship callsigns in the merge (see also Examples 1,1.)

4 panel

Figure 1-1. Dupelim skill, in combining NCEI GTS and NCEP GTS data for January 2013.
(A) Input (NCEI and NCEP) and output ("merge") from dupelim statified by deck (colors).
(B) Deck composition (colors) of the intermediate output from dupelim for each DUPS (horizontal axis).
(C) DUPS (colors) for the various deck-to-deck matches (horizontal axis).
(D) Number of weather elements in common (colors) for the various deck-to-deck matches (horizontal axis).

bar plot

Figure 1-2. Dupelim skill, in combining NCEP GTS (left-hand bar) and NCEI GTS (middle bar) data for January 2013 into the final merged product (right-hand bar), is shown for platform type (PT) ships. The bars are subdivided by ID patterns; and above the top of each bar, the number of reports in thousands is printed. Also, the following table summarizes the percentages associated with all the different ID patterns:

	REAL	MASK	MASKSTID	SHIP	unknown
NCEP	1.0%	0.2%	98.7%	0.1%	0.0%
NCEI	61.9%	22.8%	0.0%	15.1%	0.2%
merge	70.1%	26.1%	1.7%	2.0%	0.2%

Examples 1

These are examples to accompany Figs. 1-1 and 1-2, in each case showing the IMMA-format reports, followed in some instances by the corresponding NCEP supplementary data (SUPD) for those reports. DUPS is at the far right, with noteworthy fields highlighted by red lettering.

NCEP deck 792 and NCEI deck 992 ship duplicates. Effective 8 May 2012 at 0000 UTC callsign masking was implemented for all Meteorological Service of Canada (MSC) Automatic Voluntary Observing Ship (AVOS) reports. MSC sends messages in real-time with callsigns replaced with SHIP and also sends these messages with unmasked callsigns to National Meteorological Centres.

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1 100 5070 23250 13 000   2MASKSTID   01604 15      102961  4   49        51          A               165 129 7792103 5102
2013 1 1 100 5070 23250 13 000   2MASKSTID   01604 15      102961  4   49        51          A               165 129 7792103 5102
2013 1 1 100 5070 23250 130000   2SHIP       01604 15      102961  40  49        51          A               165 129 7992114 5102
2013 1 1 100 5070 23250 130000   1CG2958     01604 15      102961  40  49        51          A               165 129 7992114 5 12
BBXX01010120130112SNVD70KWBC010100RRB////// XXXX 01014 99507 71275 46/// /1603 10049 20051 40296 51004 7//// 8//// 22200 0//// 2////
BBXX01010120130112SNVD70KWBC010100RRB////// XXXXXX 01014 99507 71275 46/// /1603 10049 20051 40296 51004 7//// 8//// 22200 0//// 2////

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 4870 23650 13 000   2MASKSTID   03604 10      102861 11   37        36          A               165 16583792103 5102
2013 1 1   0 4870 23650 13 000   2MASKSTID   03604 10      102861 11   37        36          A               165 16583792103 5102
2013 1 1   0 4870 23650 130000   2SHIP       03604 10      102861 110  37        36          A               165 16583992114 5102
2013 1 1   0 4870 23650 130000   1CGBR       03604 10      102861 110  37        36          A               165 16583992114 5 12
BBXX01010120130011SMVD70KWBC010000RRD////// XXXX 01004 99487 71235 46/// /3602 10037 20036 40286 51011 7//// 8//// 22200 0//// 2////
BBXX01010120130012SMVD70KWBC010000RRE////// XXXX 01004 99487 71235 46/// /3602 10037 20036 40286 51011 7//// 8//// 22200 0//// 2////

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1111200 3100 19160 13 064   2MASKSTID   02004 10 98 21101360 13  1900 170  100 1 2001   9               165 19718792103 5102
2013 1111200 3100 19160 13 064   2MASKSTID   02004 10 98 21101360 13  1900 170  100 1 2001   9               165 19718792103 5102
2013 1111200 3100 19160 132064   2SHIP       02004 10 98 21101360 130 1900 170  100 1 2001   9               165 19718992114 5 12
BBXX01011120131252SMVD70KWBC111200RRM////// XXXX 11124 99310 71684 41998 12002 10190 20100 40136 50013 70211 8//// 91200 22264 00200 2//// 3//// 4//// 80170
BBXX01011120131259SMVD62RJTD111200B7H////// XXXXX 11124 99310 71684 41998 12002 10190 20100 40136 50013 70211 8//// 91200 22264 00200 2//// 3//// 4//// 80170

NCEP deck 797 and NCEI deck 995 tide gauge duplicates, DUPS 11 (time/space/id). DUPS was less certain because only a two tenths difference allowance was permitted for NCEP deck 797 wind speed and SST.

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 1797 29295 1225     5MGIP4      5 37  18      10174      253                                    165 279777971031611
2013 1 1   0 1797 29295 1325     5MGIP4      0 404 21      101743 100 253           1 272    A               165 2797799511416 1
                                                                                                                                
                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1  50 4791 23536 1225     5LAPW1                                                55                    165 165747971031611
2013 1 1  50 4791 23536 1325     5LAPW1      0 704 10      10274    0  40           1  58    A               165 1657499511416 1

NCEP deck 794 and NCEI deck 994 drifting buoys duplicates, zero weather elements in common. Not complement duplicates.

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 8350 30056 1225     347545                                                                      165  2839994114 713
2013 1 1   0 8350 30056 1225     347545                                                                      165  2839794103 7 1
ZZYY01010120130011SSVX02CWAO010000B7H////// 47545 01013 0000/ 783498 059435 6113/ 444 20130 01013 0000/ 50001 80136 80001 9////
                                                                                                                                
                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 8331 31013 1225     347537                                                                      165  2939994114 713
2013 1 1   0 8331 31013 1225     347537                                                                      165  2939794103 7 1
ZZYY01010120130011SSVX02CWAO010000B7H////// 47537 01013 0000/ 783308 049865 6113/ 444 20130 01013 0000/ 50001 80134 80001 9////

NCEP deck 797 and NCEI deck 995 tide gauge duplicates, zero weather elements in common. Complement duplicates where in the first set the selection of an NCEP deck 795 report eliminated only the units digit (in deck 797) of wind direction (degraded to tens of degrees in all but the CREX format). In the second and third sets winds were eliminated, and in the fourth set A, PPP and SST were eliminated.

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 4690 23589 1225     5WPTW1      5 81  69      10275       32                                    165 165647971031613
2013 1 1   0 4690 23589 1325     5WPTW1      0 804 67                                        A               165 165649951141613
2013 1 1   0 4690 23589 1325     5WPTW1                    102752 100  32           1  84    A               165 165649951141613
2013 1 1   0 4690 23589 1325     5WPTW1      0 804 67      102752 10   32           1  84                    165 1656479510316 1

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 4791 23536 1225     5LAPW1      5 57  21      10273       42                                    165 165747971031613
2013 1 1   0 4791 23536 1325     5LAPW1      0 604 21                                        A               165 165749951141613
2013 1 1   0 4791 23536 1325     5LAPW1                    102733 190  42           1  62    A               165 1657499511416 1

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 4813 23656 1225     5PTAW1      5196  21      10284       37                                    165 165837971031613
2013 1 1   0 4813 23656 1325     5PTAW1      02004 21                                        A               165 165839951141613
2013 1 1   0 4813 23656 1325     5PTAW1                    10284    0  37           1  86    A               165 165839951141613
2013 1 1   0 4813 23656 1325     5PTAW1                    102843 170  37           1  86    A               165 1658399511416 1

                         A                                               W    D                              A A               D
                L     L  T     N                               S   P     B   WP   D     S                    T TB  B    D  S   U
   Y M D   H    A     O ITTLDV I I        I CD   W   V V WW    L   PI   AT   BT   P S   S NCH CC W W W S S S T TS  1 B  C  I P P
   R O Y   R    T     N MCIISS D I        D 1I  DI  WI V W1    PA  PT   TI   TI   T I   TNHLIHMH D P H D P H I LI  0 1  K  D T S
2013 1 1   0 4855 23699 1325     5FRDW1      03204  5               0  28                    A               165 165839951141613
2013 1 1   0 4855 23699 1325     5FRDW1                    102843 110               1  82    A               165 165839951141613
2013 1 1   0 4855 23699 1225     5FRDW1      5322   5      10284       28                                    165 1658379710316 1

C-MAN/Tide Gauge

We are expanding ICOADS to include meteorological measurements from Tide Gauge (TG) stations, as well as considering the additional possibility (linked with extensive IMMA format changes planned under IVAD) to eventually extend IMMA to include TG water level data. Some TG data are transmitted over the GTS in WMO's CREX format, others are transmitted in the C-MAN format (see Table 2-1 for a breakdown of C-MAN-format and CREX-format receipts by station categories, and Fig. 2-1 for the geographic distribution of all the deck 795/797 stations listed in Table 2-1). The NOAA National Data Buoy Center (NDBC) manages data from many fixed-location near-coastal stations that are transmitted over GTS in C-MAN format. Possibly originally designed with the view to save GTS transmission costs, this C-MAN format lacks latitude and longitude (thus downstream users need to add location information independently; a longstanding issue currently under discussion with NDBC). Some of the C-MAN-format stations are "official" NDBC-operated Coastal-Marine Automated Network (C-MAN) stations, delayed-mode (quality checked) data from which are archived by the NOAA National Oceanographic Data Center (NODC). However a variety of additional station types, including but not limited to TG stations operated and archived in delayed-mode by the NOAA National Ocean Service (NOS), also utilize the C-MAN format for circulating data on the GTS.

Table 2-1. Embedded plot: amounts of ICOADS preliminary (NCEP-based) data for Feb. 2012 by deck (792-795), plus CREX data (deck 797; not presently utilized for the ICOADS preliminary product). The left side of the table shows the breakdown of total Feb. 2012 NCEP GTS receipts (numbers of reports) in decks 795 and 797 according to station categories, as listed by NDBC here: http://www.ndbc.noaa.gov/to_station.shtml (webpage accessed on 15 May 2012; webpage stated: "Page last modified: July 16, 2007") and where "(blank)" indicates ID not found. The right side of the table similarly shows the breakdown of total receipts: (i) for the only two individual station IDs OCIM2 and WPTW1 (both NOS TG stations) that overlap between decks 795 and 797, and (ii) for each individual ID within the complete set of unrecognized "(blank)" IDs. These "(blank)" IDs also were not found here: http://www.ndbc.noaa.gov/activestations.xml (created="2012-05-15T20:40:02UTC"). Presently, setting of platform type (PT) within decks 795 and 995 (also decks 797 and 997) is not well resolved (and further coordination may be needed with NDBC), since possibly only the station category "National Data Buoy Center Stations" should be set to PT=13 (13 - Coastal-Marine Automated Network (C-MAN) (NDBC operated)). For January 2013 data forward, however, we have been utilizing operational downloads of the continuously updated activestations file to make some improvements in the assignment of PT. station inventory

Figure 2-1. Map showing the geographical coverage of deck 795 and 797 Feb. 2012 NCEP data, as listed in Table 2-1, stratified according to these categories: C-MAN-format receipts (green), CREX-format receipts with IDs listed by NDBC (blue), and CREX-format receipts with IDs not listed by NDBC (red; "(blank)" in Table 2-1).

Table 2-2. Number of extant CREX code data elements for 15 February 2012 in NCEP BUFR (file sfcshp.ibm.20120215.gz).

BUFR number and description	IMMA abbrev.	NCEP extant
0 01 198 REPORT IDENTIFIER	ID	4001
0 02 001 TYPE OF STATION		4001
0 04 001 YEAR	YR	4001
0 04 002 MONTH	MO	4001
0 04 003 DAY	DY	4001
0 04 004 HOUR	HR	4001
0 04 005 MINUTE	HR	4001
0 04 015 TIME INCREMENT IN MINUTES		3977
0 04 025 TIME PERIOD OR DISPLACEMENT		3977
0 04 200 YEAR - TIME OF RECEIPT		7286
0 04 201 MONTH - TIME OF RECEIPT		7286
0 04 202 DAY - TIME OF RECEIPT		7286
0 04 203 HOUR - TIME OF RECEIPT		7286
0 04 204 MINUTE - TIME OF RECEIPT		7286
0 05 002 LATITUDE (COARSE ACCURACY)	LAT	4001
0 06 002 LONGITUDE (COARSE ACCURACY)	LON	4001
0 07 001 HEIGHT OF STATION		4001
0 08 202 RECEIPT TIME SIGNIFICANCE		7286
0 10 051 PRESSURE REDUCED TO MSL	SLP	3219
0 11 001 WIND DIRECTION	D	2246
0 11 002 WIND SPEED	W	2189
0 12 101 TEMPERATURE/DRY BULB TEMPERATURE	AT	3004
0 22 038 TIDAL ELEV WITH RESPECT TO CHART		7954
0 22 043 SEA TEMPERATURE	SST	2966
0 22 120 TIDE STATION AUTOMATED WATER DATA CHECK		3977
0 22 121 TIDE STATION MANUAL WATER DATA CHECK		3977
0 22 122 TIDE STATION AUTOMATED METEOROLOGICAL DATA CHECK		3309
0 22 123 TIDE STATION MANUAL METEOROLOGICAL DATA CHECK		3309
0 33 195 SDMEDIT/QUIPS QUALITY MARK FOR WIND		1
0 33 215 CORRECTED REPORT INDICATOR		4001
0 35 021 BULLETIN BEING MONITORED (TTAAii)		7286
0 35 022 BULLETIN BEING MONITORED (YYGGgg)		7286
0 35 023 BULLETIN BEING MONITORED (CCCC)		7286
0 35 194 BULLETIN BEING MONITORED (BBB)		7286
0 35 195 CHANNEL SEQUENCE NUMBER		7286
0 58 008 RAW REPORT STRING	SUPD	75532

Coast Guard coastal stations

Since mid-February 2012, some Coast Guard coastal station reports have been included in the NCEP BUFR files (BUFR subtype NC001007; presently about 200 such reports per day). Following are examples of the raw report strings (see format discription), and Figure 3-1 and Table 3-1 provide additional information. Because of their near-coastal nature and limited geographic coverage, these data are not considered a priority for ICOADS at this time.

L27 C10 / CALM / FLAT / 58 / 59 / AVALON HARBOR
L58 CYRW-6 / SW12 / 0508 / 57 / 57 / 1016 / MX6 MISSION BEACH
L79 C10 / SW10 / 0408 / 56 / 61 / W/MX5 CHANNEL ISLANDS HBR
9L0 CY10 / S05 / / 59 / 58 / 1016 / DANA POINT

Figure 3-1. Map showing the geographical coverage of May 2012 NCEP Coast Guard coastal stations data.

Table 3-1. Number of extant Coast Guard station data elements for 15 May 2012 in NCEP BUFR (file sfcshp.ibm.20120515.gz).

BUFR number and description	IMMA abbrev.	NCEP extant
0 01 198 REPORT IDENTIFIER	ID	208
0 02 002 TYPE OF INSTRUMENTATION FOR WIND MEASUREMENT	WI	208
0 04 001 YEAR	YR	208
0 04 002 MONTH	MO	208
0 04 003 DAY	DY	208
0 04 004 HOUR	HR	208
0 04 005 MINUTE	HR	208
0 04 200 YEAR - TIME OF RECEIPT		208
0 04 201 MONTH - TIME OF RECEIPT		208
0 04 202 DAY - TIME OF RECEIPT		208
0 04 203 HOUR - TIME OF RECEIPT		208
0 04 204 MINUTE - TIME OF RECEIPT		208
0 05 002 LATITUDE (COARSE ACCURACY)	LAT	208
0 06 002 LONGITUDE (COARSE ACCURACY)	LON	208
0 07 001 HEIGHT OF STATION		208
0 08 202 RECEIPT TIME SIGNIFICANCE		208
0 10 004 PRESSURE		14
0 10 051 PRESSURE REDUCED TO MSL	SLP	11
0 11 001 WIND DIRECTION	D	208
0 11 002 WIND SPEED	W	208
0 11 041 MAXIMUM WIND SPEED (GUSTS)		2
0 12 101 TEMPERATURE/DRY BULB TEMPERATURE	AT	73
0 20 001 HORIZONTAL VISIBILITY	VV	100
0 20 003 PRESENT WEATHER	WW	33
0 20 010 CLOUD COVER (TOTAL)		89
0 20 011 CLOUD AMOUNT	N	89
0 22 003 DIRECTION OF SWELL WAVES	SD	12
0 22 011 PERIOD OF WAVES	WP	61
0 22 021 HEIGHT OF WAVES	WH	62
0 22 043 SEA TEMPERATURE	SST	48
0 22 073 MAXIMUM WAVE HEIGHT		5
0 33 215 CORRECTED REPORT INDICATOR		208
0 35 021 BULLETIN BEING MONITORED (TTAAii)		208
0 35 022 BULLETIN BEING MONITORED (YYGGgg)		208
0 35 023 BULLETIN BEING MONITORED (CCCC)		208
0 35 194 BULLETIN BEING MONITORED (BBB)		208
0 35 195 CHANNEL SEQUENCE NUMBER		208
0 58 008 RAW REPORT STRING	SUPD	1082

GTS header information

Since 2003 when NCEI began providing marine GTS data for ICOADS, an NCEI-defined 18-character header has been included, which (together with a blank separating character) prefixes the originally reported GTS message stored in SUPD. For the NCEP GTS data, this header has been replicated in SUPD. The header contains selected GTS bulletin header information and data receipt timestamp information (note: derived independently by NCEI and NCEP, and not part of the GTS bulletin header). Effective with January 2013 data, the NCEP header has been extended to 43 characters, to include additional potentially useful "abbreviated heading" (ref.: WMO No. 386) and quality mark information; however, a planned similar extension to 37 characters of the NCEI header has been postponed indefinitely owing to unresolved technical issues at NCEI. Table 4-1 documents the content of the 18- and 37(43)-character header fields.

Table 4-1. The field content of the 18-character header (e.g. "CMAN01042020122100") in SUPD (i.e. fields 1-7 in this table), extended by the addition of fields 8-11(17) to create the new 37(43)-character header (e.g. "CMAN01042020122100SXUS22KWNB202200RRC//////"). Columns 3-7 are similar receipt timestamp information, but derived independently by NCEI and NCEP, and the information may differ, but generally these differences are restricted to minute, for a given original GTS report. The NCEP 18-character header used observation instead of receipt time.

No.	Field	E.g. segment	NCEI	NCEP
1	MjMjMjMj	CMAN	BBXX, ZZYY or CMAN	BBXX, ZZYY, CMAN or CREX
2	"01"	01	sequence number for split messages¹	"01"
3	Month	04	timestamp month²	MONTH - TIME OF RECEIPT
4	Day	20	timestamp day²	DAY - TIME OF RECEIPT
5	Year	2012	timestamp year²	YEAR - TIME OF RECEIPT
6	Hour	21	timestamp hour²	HOUR - TIME OF RECEIPT
7	Minute	00	timestamp minute²	MINUTE - TIME OF RECEIPT
8	TTAAii	SXUS22	WMO abbreviated heading: data type/geographic/time/etc. designators	same
9	CCCC	KWNB	WMO abbreviated heading: originating/compiling station indic.	same
10	YYGGgg	202200	WMO abbreviated heading: international date-time group	same
11	BBB	RRC	WMO abbreviated heading: indic. for corrections/amendments/etc.	same
12	QMAT	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR TEMPERATURE³
13	QMDD	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR MOISTURE³
14	QMPR	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR PRESSURE³
15	QMST	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR SEA SURFACE TEMPERATURE³
16	QMWH	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR WAVE HEIGHT³
17	QMWN	/	n/a	SDMEDIT/QUIPS QUALITY MARK FOR WIND³

¹ obsolete; unused
² GTS file creation date; could possibly change if the GTS file has to be recreated. The NCEP time of receipt information is determined independently, and therefore may differ, generally to a small extent, such as shown in the following examples.
³ NCEP 43-character header was extended by quality marks, or "/" is assigned in our processing if the quality marks were missing in BUFR (note: more information about NCEP's processing that sets the quality marks is provided here).

Examples of NCEI and NCEP GTS reports (in that order) with 37(43)-character headers:

CMAN01042020122100SXUS22KWNB202100RRX 20214 NPDW3 46/// /3420 10057 40149 92110 333 91226 555 11020 22021
CMAN01042020122149SXUS22KWNB202100RRX////// 20214 NPDW3 46/// /3420 10057 40149 92110 333 91226 555 11020 22021
                ^^

GTS messages — systematic differences

(using coads_20120205_gts_jan.gz)

Possible explanations:

NCEI in their earlier ingest process
centers that are providing NCEI with the GTS data
Local User Terminals vs. main GTS receipt stations

NCEP cancels (Xs) out the ID in the original messages.

NCEI BBXX Section 3 and Section 5 unreadable because of "/"s.

NCEI 99 0 BBXX               44040 01001 99410 70736 46/// /2103 10092 20013 40182 92345 22200 1//00 70001 333/91204 555/11041 22046
NCEP 99 0 BBXX               44040 01001 99410 70736 46/// /2103 10092 20013 40182 92345 22200 1//00 70001 333 91204 555 11041 22046
                                                                                                              ^         ^

BBXX difference with no effect.

NCEI 99 0 BBXX               WAHV 01063 99376 10105 43599 52930 10160 2013/ 40208 53005 82606 22274 00162 20905 80140
NCEP 99 0 BBXX               XXXX 01063 99376 10105 43599 52930 10160 20130 40208 53005 82606 22274 00162 20905 80140
                                                                          ^

NCEI CMAN hour in group YYGGi not rounded up when minutes in group 9GGgg are 30.

NCEI 99 0 CMAN               01054 ANCF1 46/// /0000 10157 40228 90530 222// 00189 333 91201 555 TIDE1123
NCEP 99 0 CMAN               01064 ANCF1 46/// /0000 10157 40228 90530 222// 00189 333 91201 555 TIDE1123
                                ^

NCEI ZZYY Section 3 deleted.

NCEI 99 0 ZZYY               23010 18012 1200/ 303987 080511 6112/ 444 20120 18012 1047/ 521//
NCEP 99 0 ZZYY               23010 18012 1200/ 303987 080511 6112/ 33311 88872 20001 32865 20020 32850 43405 20040 32822 43428 20060 32388 43508 \
20080 32050 20100 31946 20120 31871 20140 31764 20180 31448 20300 31143 20500 30974 444 20120 18012 1047/ 521//
                                                                   ^^^

NCEI ZZYY Section 4 deleted.

NCEI 99 0 ZZYY               25615 10012 2259/ 785271 169671 6113/ 11119 49991 57005 22219 01097
NCEP 99 0 ZZYY               25615 10012 2259/ 785271 169671 6113/ 11119 49991 57005 22219 01097 444 20130 10012 2054/ 50101 80136 8//// 9/015
                                                                                                 ^^^

ZZYY latitude longitude differences.

NCEI 99 0 ZZYY               21210 02012 0700/ 138112 146475 6111/ 11119 10106 29082 49971 22219 00000 444 20110 02012 0749/ 522//
NCEP 99 0 ZZYY               21210 02012 0700/ 138109 146476 6111/ 11119 10106 29082 49971 22219 00000 444 20110 02012 0749/ 522//
                                                   ^^      ^

ZZYY differences with no effect.

NCEI 99 0 ZZYY               46642 12012 0359/ 752500 130970 6110/ 11119 40272 57015 22219 00074 444 201// 12012 0406/ 80080 80005
NCEP 99 0 ZZYY               46642 12012 0359/ 75250/ 13097/ 6110/ 11119 40272 57015 22219 00074 444 201// 12012 0406/ 80080 80005
                                                    ^      ^

Unresolved issues

Platform type (PT) and ID indicator (II)
1. One issue for eventual ICOADS delayed-mode processing is that WMO Pub. 47 contains the REAL callsign, not a masked ID. In the original GTS data (FM 13), masked IDs generally take two forms: (i) the generic ID "SHIP," or (ii) a ship-specific alias referred to as "MASK" (see this article for additional information). JCOMMOPS maintains a secured database of REAL vs. masked callsigns, which could be used (resources permitting and assuming access was authorized by WMO) to sort out the problem. As an additional complication, since Dec. 2007, most NCEP BUFR ship reports (deck 792) have been uniformly remasked by NCEP to "MASKSTID," and with "X...X" overwriting any real FM 13 callsign information. (Note: MASKSTID is considered a generic ID, in addition to SHIP, BUOY, and some other non-ship forms.)
2. MASK is presently classified incorrectly as callsign in the ICOADS preliminary product (see Figure 6-1B), partly because we have no definitive list of what are all legal MASK aliases, and it is unknown whether any might resemble REAL callsigns (see also Table 6-1).
3. More work is needed to ensure that official NDBC C-MAN stations, tide gauge stations, etc. are properly assigned platform type (PT) and ID indicator (II) (possibly using additional information available in: http://www.ndbc.noaa.gov/activestations.xml). As part of this issue, II configurations currently defined in the IMMA format, including "5 - Coastal-Marine Automated Network (C-MAN) ID (US NDBC operated)" may need adjustment/amplification.
Figure 6-1. April input (A) ID indicator (II) (horizontal axis) stratified by deck (colors); (B) deck 992 stratified by estimated improved ID categories (see Table 6-1). (Note: following is the list of IDs currently defined for contemporary data types as generic: SHIP, MASKSTID, BUOY, DRIB, RIGG, PLAT, and NNXX.)

Table 6-1. April input deck 992 IDs categorized as MASK, i.e. identified simply as a pattern of five letters and two numbers, and their frequencies.

Appendix A: Changes in the Processing of Relative Humidity (RH) and Dew Point Temperature (DPT) in NCEP and NCEI GTS Data

While both the FM 13 SHIP code and the FM 18 BUOY code have the capacity to report RH, the IMMA format (version IMMA0) lacked a regular field space for relative humidity (RH). Recently however, an RH field was added to IMMA1. Presently for FM 13 (apparently applicable also for FM 18), the reporting of DPT (T_dT_dT_d; to tenths Celsius) or RH (UUU; whole percent) is governed by Regulation 12.2.3.3.1 (ref. WMO-No. 306, Manual on Codes):
"Under unusual conditions, when the dew-point temperature is temporarily unavailable (e.g. because of instrument failure) but relative humidity is available, the group 29UUU shall replace the group 2snT_dT_dT_d. Every attempt shall first be made, however, to convert relative humidity to dew-point temperature, and the relative humidity included only as a last resort."

For NCEP GTS data starting from March 1997, Table A1 provides a timeline of processing changes impacting RH and DPT. Similarly, NCEI data extending back to at least Nov. 2003 that have been already translated into IMMA are impacted by some such changes, but those data have not yet been utilized for ICOADS and may be reconverted to IMMA1 so as to alleviate processing inconsistencies.

Figure A1 (scatter plot) and Figure A2 (distribution plot) show how at mid reported humidity values recomputed humidity values are slightly larger.

Table A1. Timeline of decoding and other processing changes in NCEP GTS data applicable to RH, DPT, and RF (the trimming flag for RH, which is also stored in IMMA but set during a later QC processing step). The algorithm used for computing missing DPT conforms with ICOADS preconditioning rules (e.g. here), and the algorithm used for computing missing RH conforms with that used for ICOADS monthly summary processing, i.e. the codes available here.

Effective month	Processing modification
Mar. 1997	IMMA fields (including DPT) decoded from original GTS messages, but reported RH values omitted.
Oct. 1999	IMMA fields (including DPT) translated from BUFR instead, but reported RH values still omitted.
Jan. 2005	When available instead of DPT, RH was translated from BUFR and converted to DPT for storage in IMMA0. Then RH was temporarily recomputed from DPT (and reported air temperature) only for setting RF.
TBD	Transition to IMMA1 allows direct storage of RH. Now when RH is reported, missing DPT is computed from RH; conversely, when DPT is reported, missing RH is computed from DPT (and associated fields are set to indicate which field was originally reported versus computed). Thus it is no longer necessary to temporarily recompute RH for setting RF. However, the RH values being flagged represent a mixture of originally reported and computed RH data, between which systematic differences may exist.

Figure A1. Scatter plot of September 2012 RH measured (originally in whole %) vs. the same RH recomputed (i.e. in accordance with the procedure effective from Jan. 2005 in Table A1, such that RH was translated from BUFR and converted to DPT, then RH was temporarily recomputed from DPT for setting RF).
RH histogram

Figure A2. Distribution plot of September 2012 RH measured vs. the same RH recomputed (i.e. in accordance with the procedure effective from Jan. 2005 in Table A1, such that RH was translated from BUFR and converted to DPT, then RH was temporarily recomputed from DPT for setting RF).

Appendix B: Great Lakes CREX data LST time zone

For local operational reasons, CREX bulletins for the Great Lakes were going out (through 14Z 24 Oct 2012) in LST time zone. This accounts for some of the large data differences seen on this webpage (in its Examples 3), i.e. the apparent CREX duplicates are displaced by five or six hours, since any reports with LST times were interpreted by our then current translation programs as UTC times.

time gap

Figure B1. 24 October 2012 NCEP Great Lakes data: year, month, day, hour in hundredths (horizontal axis) vs. number of CREX bulletins in Central Standard Time through 8 a.m. (thin red), in Eastern Standard Time through 9 a.m. (thin blue), and in UTC after 14Z (number of non-CREX with same IDs solid blue).

map

Figure B2. Great Lakes tide gauge locations by time zone (colors: red Central, blue Eastern).

Table B1. Great Lakes tide gauge IDs (ref: NOS_SHEF_CREX_05-03-12.xls). CREX ID MI220 appears twice but in the NCEP data has the same location as SHEF ID FWNM4 (42.30N 83.09W).

NOS #	CREX ID	SHEF ID	LST
9087044	IL010	CMTI2	CST
9087057	WI010	MLQW3	CST
9087068	WI015	KWNW3	CST
9087072	WI020	SBCW3	CST
9087079	WI040	GBWW3	CST
9087088		MNMM4	CST
9099064	MN100	DULM5	CST
9099090	MN110	GDMM5	CST

NOS #	CREX ID	SHEF ID	LST	NOS #	CREX ID	SHEF ID	LST	NOS #	CREX ID	SHEF ID	LST
8311030	NY034	OBGN6	EST	9052058	NY041	RCRN6	EST	9075035	MI016	ESVM4	EST
8311062	NY036	ALXN6	EST	9052076	NY042	OCTN6	EST	9075065		LPNM4	EST
9014070	MI220	AGCM4*	EST	9063007	NY043	NGAN6	EST	9075080	MI019	MACM4	EST
9014080	MI250	SCRM4	EST	9063009	NY046	AMFN6	EST	9075099	MI100	DTLM4	EST
9014087	MI260	PHXM4	EST	9063012		NIAN6	EST	9076024		RCKM4	EST
9014090	MI255	MBRM4	EST	9063020	NY050	BUFN6	EST	9076027		WNEM4	EST
9014096	MI256	DUPM4	EST	9063028	NY060	PSTN6	EST	9076033		LTRM4	EST
9014098	MI245	FTGM4	EST	9063038	PA012	EREP1	EST	9076060	MI110	USSM4	EST
9034052	MI230	SCSM4	EST	9063053	OH010	FAIO1	EST	9076070	MI120	SWPM4	EST
9044020	MI200	GRTM4	EST	9063063	OH020	CNDO1	EST	9087023	MI025	LDTM4	EST
9044030	MI210	WDTM4	EST	9063079	OH025	MRHO1	EST	9087031	MI290	HLNM4	EST
9044036	MI220	FWNM4	EST	9063085	OH030	THRO1	EST	9087096	MI020	PNLM4	EST
9044049	MI221	WMPM4	EST	9063090	MI010	FPPM4	EST	9099004	MI130	PTIM4	EST
9052000		CAVN6*	EST	9075002	MI257	LKPM4	EST	9099018	MI140	MCGM4	EST
9052030	NY040	OSGN6	EST	9075014	MI013	HRBM4	EST	9099044	MI145	OGOM4	EST

* See Figure B3 for possible corresponding CREX ID.

map

Figure B3. Plot of CREX format IDs 3 Oct. 2000 through 31 Dec. 2012 after removing IDs in Table B1; with only CREX IDs MI270, MI240 and NY038 remaining in the Great Lakes region. MI240 and NY038 may have Table B1 SHEF ID counterparts:

ID	NCEP location	NOS_SHEF_CREX_05-03-12.xls location
MI270	44.66N 83.29W
MI240	42.62N 82.53W
AGCM4	42.62N 82.57W	42.621000N 82.526900W
NY038	44.13N 76.34W
CAVN6	44.13N 76.33W	44.130190N 76.331970W

[Documentation and Software][Links to additional]

U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads@noaa.gov
Updated: May 11, 2016 19:04:19 UTC
http://icoads.noaa.gov/merge.html