Report-rejection (Hour-missing; Flag ZQ) Problem in Initial Release 1c Data

Release 1c monthly summary statistics (1800-1949) that became available in
March 2001 were impacted by a report-rejection problem that significantly
reduced numbers of observations and spatial coverage in early decades
(especially prior to 1860).  This also impacted users of observational data in
Long Marine Reports Fixed-length (LMRF) format, or data subsets derived from
LMRF, which employed software (trimqc0.f) to select LMRF.  The problem and
software were corrected, and data reissued, during August 2001 (netCDF files
were corrected around September 2001).

Summary of products and impacts from the ZQ problem:
  
* LMRF global and subset binary data files (NOT IMPACTED) (caveat: if your
  software calls subroutine trimqc0.f, the data were IMPACTED; the corrected
  subroutine is available at icoads.noaa.gov/software/)
      
* ASCII subsets created at NCAR from the LMRF observations (YES IMPACTED)
   
* Binary and ASCII subsets created at NCAR from the MSG statistics (YES
  IMPACTED)

* netCDF files served from PSD (YES IMPACTED).

The problem stemmed from usage of the report-status (ZQ) flag to reject reports
flagged erroneous by the NCDC-QC.  Our intention in checking ZQ was to
eliminate reports falling over a 1° latitude x 1° longitude landmask.  However,
ZQ was also set if hour (or day) was missing.  Moreover, the Maury Collection,
covering 1784-1863 and providing the bulk of the available Release 1c data
prior to about 1854, included many daily reports, in which hour was set to
missing.  Thus much of the Maury Collection was eliminated through use of the
ZQ flag.

To correct the problem, we have adjusted the report-selection rules for
calculation of monthly summary statistics (and the corresponding trimqc0.f
software) to omit the check for the ZQ flag, applicable only to pre-1950 data.
Reports are still checked for falling over a 2° latitude x 2° longitude
landmask (using the LZ flag).  More permissive landmask checks may actually be
more appropriate for early historical data due to the more approximate nature
of reported ship and port positions.  As a consequence, however, it should be
noted that allowable spatial coverage rules (for statistics and observations
subsetted from LMRF) are different before and after 1950.

Figures 1a-1b show total numbers of Long Marine Reports (LMR) output from
Release 1c duplicate-elimination processing, per year.  Most duplicates were
removed from LMR by this processing, but "uncertain" duplicates were retained
with flags to indicate their status.  As illustrated in Figures 1a-1b, selected
uncertain duplicates, plus reports falling over a 2° landmask, were removed to
create LMRF from LMR.  The remaining data constitute LMRF.

Figures 1a-1b further illustrate LMRF rejected from the statistics through use
of the ZQ flag (in red), prior to the correction (plus non-ship reports
rejected only from the standard statistics).  Many other data sources
occasionally contain missing hour (or day), but systematically missing hours
generally appear to be limited to the Maury Collection.  Thus minor data
differences may arise through 1949 as a result of the ZQ correction, but the
differences are concentrated prior to 1860.

Figures 2a-2e illustrate decadal spatial coverage through the 1860-69 decade
before and after the ZQ correction, for major observed variables.  In the
corrected data, wind speed (Figure 2d) obtains the largest and earliest
increases in coverage, because many Maury reports were limited to wind
observations.  Conversely, total cloudiness (Figure 2e) shows the least
improvement because the Maury Collection contained little or no usable
cloudiness data.

Figures 3a-3e show the numbers of observations and 2° boxes per month
(1820-1859) included in the enhanced statistics, for each of the same major
observed variables as shown in Figs. 2a-2e, before and after the ZQ correction.
Again, cloudiness was not impacted by the problem.

[Documentation and Software][Links to additional]