D R A F T 13 April 1998 
Weekly inventories (report observation times per timeblock) of NOAA/NCEP ON124 surface marine (SFCSHP) data for 1980-85

1. Background

Inventories (color figures) are presented for 1980-85 of surface marine (SFCSHP) reports from NCAR's archive (ds464.0) of NCEP (formerly NMC) "ADP Global Sfc Obs," containing Office Note 124 (ON124) data gathered from the Global Telecommunication System (GTS). ON124 data were gathered at NCEP into four synoptic timeblock (00, 06, 12, and 18 UTC) "files."

In NCAR's archive of ON124 data, there are header and trailer blocks instead of actual file marks. Each such header block begins with a three (10-character) "word" header providing the synoptic timeblock hour, year, month, and day in the first word (HHHHYYMMDD). Each record within the timeblock file is then labeled with the precise hour (to hundredths) but NOT the year, month, or day (positions are provided in ON124 for both observation and receipt times).

The inventory plots are presented with a weekly time increment, because ON124 data are stored at NCAR in weekly files.

2. Weekly timeblock inventory results

The following four groups of color figures provide weekly inventory results for each timeblock; within each such group, six figures display the inventory results separately for each year 1980-85:

 

Figures 1- 6 :	timeblock 00, 1980-85
Figures 7-12 :	timeblock 06, 1980-85
Figures 13-18 :	timeblock 12, 1980-85
Figures 19-24 :	timeblock 18, 1980-85

 

Truncation to tenths, rather than rounding to the nearest tenth of hour, was used to preserve the separation that normally exists between the "upper-cutoff" of one timeblock, and data in the next timeblock. Otherwise, for example, reports in timeblock 00 with observation times ranging from 2.95-2.99 (depending on the precise rule used for rounding) would be plotted at hour 03.0, and would appear to overlap with data from timeblock 06 at hour 03.00, which is not ordinarily the case. The following discussion refers to upper- and lower-cutoffs, and to plotted data that appear to have been reported at whole hours.  Due to the truncation to tenths of hours, any reports within a tenth of each whole hour also are included (e.g., 03.0 actually represents the interval 03.00-03.09 inclusive).

Figure 25 is similar to Figures 1-24, but presents the entire 6-year time period on one plot and aggregates data from all timeblocks to illustrate total data coverage.  Figure 26 is a summation of that total data coverage into 4-week blocks (same color-scale as Figure 25),  which provides a smoothed representation of the  variations in temporal coverage.  Figure 27 is a time-series line plot providing a summary of reports per timeblock (four separate curves) per week. Gaps in the archive (also shown in Figures 1-25 by white gaps) reflect weekly files that were unreadable.

Specifically, the following weeks (starting day of week) of data (or sometimes two weeks in a row, indicated by "+") were unreadable, due to an internal error within the complicated ON124 blocking structure, at least by the program that was used for this exercise:

 

1980Jan27	1983Feb06
1981Nov22	1983Apr10+1983Apr17
1982Oct24	1985May19
1982Nov28	1985Jul28
1982Dec26	1985Nov17+1985Nov24

 

3. Discussion

a) It should be emphasized that the inventories presented here are limited to surface marine (SFCSHP) data (excluding land and other non-marine reports) from the ON124 archive. On the other hand, this represents a superset of the 1980-85 data provided to MEDS (i.e., only ON124 "report types" 561 and 562, corresponding to moored and drifting buoys, respectively, were supplied to MEDS). Most or all "off-hourly" reports (reports not at whole hours) in the ON124 surface marine archive during this time period are believed to be from drifting buoys. This is because ordinary ship reports are concentrated at the main (00, 06, 12, 18 UTC) and secondary synoptic hours (03, 09, 15, 21 UTC), and moored buoys are typically hourly.  Therefore, patterns would likely be more uniform if only drifting buoy data were examined, i.e., less pronounced frequencies at the main and secondary synoptic hours.

b) Except for the anomalous period mid Dec 1981-early Nov 1983 (discussed below), the upper-cutoffs for all the timeblocks appear stable at timeblock hour (T) plus 2.99 hours (T+2.99).

c) Earlier, based on spot checks, we had speculated that the lower-cutoff for each timeblock had changed at some point during the 1980-85 period from T-3 to T-4. The inventories reveal a more complex story: For Jan 1980 through early Feb 1980, the lower-cutoff appears to be at T-3. From then until early Sep 1980, there is a (sporadically missing) lower-cutoff at T-4 but without any intervening data from T-4 up to T-3. Henceforth, the lower-cutoff is T-4, except that during the anomalous mid Dec 1981-early Nov 1983 period (discussed below) data at T-4 are again sporadically missing plus there are no intervening data from T-4 up to T-3.

d) During mid Dec 1981-early Nov 1983, large gaps appear in each timeblock for times less than the timeblock central hour T; other anomalous patterns also occur. Specifically, for timeblocks 06, 12, and 18, data at times less than T appear only at (approximately) whole hours except off-hourly data appear from T-2 down to T-3. For timeblock 00, the entire lower-half of the timeblock coverage (20-23.99) is missing, except for whole-hour data at 20, 21, and 23. Also, from about Jan-Jun 1982, data appear at whole hours above the typical upper-cutoffs of T+2.99; it is not known whether these anomalous values might indicate a timeblock misassignment or even a possible misplacement of observation time.  Problems during this period appear to represent a significant loss of data (~2,500 reports per week; ref. Figure 27), possibly previously undetected in the ON124 archives at NCAR and NCDC (to be checked).  Checks against NCAR ON124 land data inventories have not shown any indication of a corresponding increase at the time the surface marine data drop (Dec 1981), but further checks may be needed to be sure that the missing marine data were not mis-labeled as some other data type.

We ran a separate check to confirm the existence of one element of these problems by reading through the output COADS LMRF data for February 1982, and found zero reports from MEDS (deck 714) with hour greater than 20.98 (after "preconditioning" which removes all MEDS reports with position flagged doubtful, i.e., lat/lon flag=3).   A preliminary check of data from NCDC's ON124 archive also showed possible evidence of the problems.

It is possible that the problems were introduced at NCEP during transitions associated with the 1982 WMO code changes. We were aware of other problems in the ON124 archive associated with the 1982 code changes, but the early Nov 1983 ending of these data gaps does not correspond with the 9 May 1984 timing when NCEP addressed other known problems (e.g., the omission of the IX field).

e) There are a few other transient or unexplained features shown by the plots. Mar-Jun 1981 contains relatively sparse receipts, especially in timeblock 12 (ref. Figures 25-26). Other variations in data density may indicate an evolution in data availability (e.g., more drifting buoy data) or other factors such as changes in data retention policies by NCEP.

f) Conclusions with respect to the COADS Release 1a 1980-85 timeblock 00 ON124 hour misassignment problem for deck 714 (MEDS buoy data): Comparisons of Figures 1-6 (timeblock 00) with Figures 19-24 (timeblock 18) generally confirm our earlier assumption that all reports with observation times 21-23.99 (or 24.00) could be mechanically re-assigned to the proper day. However, during Jan-Jun 1982 there may be a small number of reports (from timeblock 18 with observation times above T+2.99) that would be improperly re-assigned. Also during the entire anomalous period Dec 1981-early Nov 1983 there may not be that many reports re-assigned because of the newly discovered data gaps.

[Documentation and Software][Links to additional]