Release 1a (1980-97) Overview

1. Background

An update and extension of Release 1a for 1980-97 (previously 1980-95) was
completed in May 1999.  This involved new data and other revisions for the
previously available 1980-95 period, plus the extension through 1997 (an
initial version of this update was completed in March 1999, but data for
1982-85 subsequently were re-processed to correct a GTS cloud data problem,
discussed in section 3, item d).

226M input records were processed; yielding 67.7M output Long Marine Reports
(LMR), and 66.7M LMRF (fixed-length format) after removal of 1% of the LMR data
as "uncertain" (possible) duplicates and landlocked data.  The input mixture
involved large new redundancies from alternative US GTS datastreams, but most
of these data were limited to test usage only during this update as we explore
data quality issues in more detail.

Data for 1994-97 were impacted by complex changes due to system and format
changes by major data providers, which took extra time to resolve.  Some
problems are not yet fully resolved (as discussed in section 3).


2. Data source additions and modifications

Following are major new data additions, and modifications of previously
available data sources, made to the Release 1a period:

a) Merger of the UK Main Marine Data Bank (MDB) with COADS for the period
   1980-94 (2% of total output LMR, and 3% of 1980-89 period LMR).
b) Replacement of post-1990 TOGA/TAO GTS data with "standard archive" data
   obtained directly from PMEL (deck 144) offering improved diurnal coverage
   and additional quality control.  PIRATA buoys (tropical Atlantic) are now
   part of this archive.  Also, a problem involving undetected duplicates
   (~24K reports) in 1989-90 deck 144 data was resolved.
c) Data were added from the World Ocean Database 1998 (WOD98; deck 780),
   including sea surface temperature estimates derived from the uppermost
   layers of ocean profiles, and some surface meteorological fields.
d) Correction of a problem with MEDS (deck 714) drifting buoys with misassigned
   (+1) day for 1980-1985, only impacting buoys reporting in the last quarter
   of the day.  Also, for 1993-97, our processing of the MEDS data was modified
   to accept reports flagged doubtful by MEDS (and revised 1993 data were
   obtained from MEDS), resulting in large increases in available drifting buoy
   data for 1993-97 (this problem stemmed from changes at ARGOS; see section 3,
   item e for further information).
e) Usage of US Air Force Global Weather Central (GWC; deck 888) GTS data (and
   experimental usage of US Navy GTS datastreams) to fill data gaps and advance
   toward a more complete and higher-quality GTS archive.  Previously missing
   US Navy Autodin (deck 889) data for March 1980 and January 1989 were added
   (an additional gap, likely not resolvable, still exists in April 1988).
f) Russian AARI North Pole (NP) Station (manned drifting ice floe) data from
   the Polar Science Center (deck 733; through 1991).
g) French logbook (deck 926) data through 1988 with corrected longitudes
   replaced mislocated data (the problem was confined to 90E-90W across the
   dateline, and may have extended only back to about 1981).  Tests during
   duplicate elimination (i.e., seeking matches with a mislocated copy of
   the correction data) indicated that the problem was successfully resolved.
h) New Shipboard Environmental (Data) Acquisition System (SEAS; deck 874) data
   were included for 1995-97.
i) Additional Russian MORMET (deck 732) data.


3. Additional data problems

We are still exploring the extent and ramifications of some of the following
problems.  More information will be available at a later date, and we may
consider remedial processing for selected time periods if problems are later
judged serious enough:

a) The quality and completeness of available GTS and US-keyed logbook data
   deteriorated during recent years (1994-97), stemming from a variety of
   system and format changes.  Progress was made in comparison to previously
   available 1994-95 data (e.g., large numbers of buoy reports previously
   misidentified as reports with generic ID "SHIP" were properly assigned
   during 1995); however, more work is needed (note: there is a residual
   problem with misidentified "SHIP" reports in October 1994).  Efforts are
   planned in the future to more fully resolve some problems by re-processing
   of original GTS archives (e.g., UK and NCEP).
b) Extant wind speed indicators (WI; 3 = knot, estimated) in NCEP data (decks
   892-896) are in error for approximately 19 April-21 October 1997, since the
   initial BUFR format used at NCEP during this portion of 1997 failed to
   carry WI.  We continue to examine questions about the reliability of WI and
   other key indicators in ship (GTS and logbook) data sources.
c) Certain temperature values in NCEP data also are subject to small (0.1C)
   errors, due to the method of storage in BUFR (corrected for data starting
   approximately 17 February 1999).
d) In the initial version of the 1980-97 data completed in March 1999, US Air
   Force GWC (deck 888) GTS data frequently were selected over logbook or other
   GTS sources, during late 1982 through mid-1985, for reasons related to the
   handling of WMO Ship Code changes in 1982.  We later discovered that many
   GWC reports during this period also contained erroneous (zero) total cloud
   amounts.  Thus the 1982-85 period was re-processed in May 1999 to entirely
   delete deck 888.  Deck 888 also was frequently selected during the initial
   NCEP BUFR transition period (approximately April through October 1997); we
   are still looking into the reasons for this more recent selection pattern.
e) The change in our handling of MEDS QC flags (1993-97; section 2, item d)
   will result in some additional reports with genuine location or other
   problems being accepted.  Such data may also be accepted into later COADS
   statistics (2-deg and 1-deg) for areas without trimming limits.  Detailed
   background: Prior to 1993, reports were generally flagged due to unrealistic
   movements in the buoy trajectory detected during MEDS track checking.
   Starting in 1993, however, MEDS started flagging large numbers of reports
   because the position was calculated from a different satellite pass (i.e.,
   the time of the buoy position did not correspond to the time of data), in
   response to apparent software changes at ARGOS.  By retaining these reports
   we are now still accepting some data with non-trivial position (track check)
   problems. At present, we do not have enough information in the available
   MEDS format to determine why data were flagged, but work is underway to
   estimate the extent of remaining problems.
f) WOD98 data were "passed through" duplicate elimination (unique data not
   available from other sources).  However, possible duplication may arise,
   e.g., if oceanographers utilized data also reported from the bridge via
   regular synoptic ship reports.


4. Noteworthy transitions in the data

a) Ship data coverage has been declining since peaking in the late 1980s.
   Although influenced by delays in the receipt of keyed ship logbook data
   (e.g., significant amounts of international logbook data became available
   for 1988-89 as part of this update), most of the change apparently stems
   from declines in global shipping or other transitions in the ship data.
b) Conversely, there has been substantial growth in the numbers of drifting
   and moored buoy reports since 1980, including crucial TOGA/TAO arrays.
   Drifting buoys provide wide coverage in the global ocean (generally only
   temperature and pressure data, however).  There was steady growth, e.g.,
   in NDBC (deck 883) moored buoy and coastal (hourly and recently some
   sub-hourly) data until 1996, but a decline in 1997.
c) The PMEL EPOCS data (deck 145) are all daily averages, and PMEL
   TOGA/TAO data prior to the early 1990s have averaging varying from
   2-8 hours.
d) Fishing fleet data (IATTC; deck 667) are subject to differences in observing
   practices compared to ordinary ship data.  Due to the nature of the fishing
   activities, a bias toward conditions associated with calm winds has been
   detected in these data.
e) NCEP changed procedures around the end of 1994 to accept more hourly moored
   buoy data (and possibly hourly data from other platform types).
f) A gap existing in NCDC's NCEP archive for 2-8 November 1986 is filled
   from GWC data (deck 888) and other sources (missing NCEP data have been
   supplied by NCAR to NCDC, but a reconversion from NCEP's format is still
   required).

[Documentation and Software][Delayed-mode (ICOADS.DM) Archive]