Comparisons of TDF-11 (Release 1) Decks and MDB Versions

1.  Introduction

Decks from Tape Data Family-11 (TDF-11) (NCDC, 1968) formed a key component of
COADS Release 1 (covering 1854-1979) data prior to 1970, and more recently
for Release 1c (1784-1949).  Selected TDF-11 decks also were included in the
UK Meteorological Office Main Marine Data Bank (MDB).  The blending of COADS
and MDB data completed as part of Release 1c permitted comparison between
TDF-11 decks that were originally used for COADS, and those also available
from MDB (Table 1).


Table 1.  Description, and raw input counts to COADS Release 1c, of TDF-11
decks (plus MDB deck 215) overlapping between COADS and MDB through 1949
(MDB data extend only back to 1854, and in COADS only deck 193 extends prior
to 1854 with a few scattered reports back to 1800).  MDB deck 215 and TDF-11
deck 192 are compared because, as discussed in sec. 3, they are thought to
be derived from the same original data source.
-------------------------------------------------------------------------------
 Deck     Description                                    COADS             MDB
===============================================================================
 110      US Navy Marine                               534,658         443,950
 116      US Merchant Marine                           339,325         239,486
 118      Japanese Ships No. 1 (Kobe keyed in 1961)  1,702,582       1,688,837
 187*     Japanese Whaling Fleet                         2,227           2,224
 188*     Norwegian Antarctic Whaling Factory Ships     21,640          21,578
 189      Netherlands Marine                           115,623          90,895
 192/215  Deutsche Seewarte Mar./Ser. 15 (German)    6,647,416   /   7,622,983
 193      Netherlands Marine                         6,402,700       6,281,052
 195      US Navy Ships Logs                           603,473         609,007
 196*     Deutsche Seewarte Marine (192 extension)      19,805          19,816
 197*     Danish (and Other) Marine (Polar)             20,988          20,770
 281      US Navy Monthly Aerological Record (MAR)     190,585         193,039
-------------------------------------------------------------------------------
* Deck with negligible overall COADS-MDB difference; not compared further
(decks not marked "*" were subject to more detailed comparisons in sec. 2).
----------


2. Temporal and spatial comparisons by deck

Annual time-series comparisons of the numbers of input reports from COADS
and MDB are presented in Figures 1a-1h for the decks identified as having
significant differences in Table 1 (the deck 192/215 differences are the most
significant; presented first in Figure 1a).

Figures 2a-2b illustrate, for two example years with large overall differences
in counts in Figure 1a, input spatial coverage in deck 192 versus that in deck
215.  In Fig. 2a, note in the sparsely sampled Pacific fragmentary ship tracks
in both decks, which would appear to be connected only by combining the unique
segments from the two decks.  Also in Fig. 2a in the Indian Ocean, deck 215
contains some additional tracks.  Similar features exist in Fig. 2b, e.g., a
ship track that is more complete in deck 192 (215) in the northern (southern)
Pacific.

Figures 3a-3b are difference maps of the spatial coverage provided by deck
215 minus that in deck 192 for the same two example years for Figs. 2a-2b.
Note the presence of both positive and negative differences in heavily sampled
regions such as the Atlantic and Indian Oceans.

Figure captions:
     Figure 1a.  Annual input counts for TDF-11: deck 192 + MDB deck: 215.
     Figure 1b.  Annual input counts for TDF-11 + MDB deck: 110.
     Figure 1c.  Annual input counts for TDF-11 + MDB deck: 116.
     Figure 1d.  Annual input counts for TDF-11 + MDB deck: 118.
     Figure 1e.  Annual input counts for TDF-11 + MDB deck: 189.
     Figure 1f.  Annual input counts for TDF-11 + MDB deck: 193.
     Figure 1g.  Annual input counts for TDF-11 + MDB deck: 195.
     Figure 1h.  Annual input counts for TDF-11 + MDB deck: 281.
     Figure 2a.  Spatial distribution by 1-degree boxes of input counts
     for COADS deck 192 (upper panel) versus MDB deck 215 (lower panel)
     for 1923.
     Figure 2b.  As for Fig. 2a, except for 1927.
     Figure 3a.  Spatial distribution by 1-degree boxes of input counts
     for MDB deck 215 minus COADS deck 192 for 1923.
     Figure 3b.  As for Fig. 3a, except for 1927.


3. Discussion

Most of the TDF-11 data used for Release 1 were taken from the "Atlas File,"
which was subject to some editing and compositing of observations, rather
than original TDF-11 data.  In contrast, the UK Met. Office may have received
the original (pre-Atlas) TDF-11 data (see Woodruff, 1990).

Decks 192 and 194 (Great Britain Marine) were deliberately omitted from MDB
(Woodruff, 1990): deck 192 presumably because of the existence in MDB of deck
215, and deck 194 because it was derived (by the US) from more original MDB
data.

Figure 1a illustrates many differences between the annual counts of COADS
and MDB for decks 192 and 215, with deck 215 sometimes significantly higher
(discussed in more detail below).  In contrast, Figures 1b-1g show counts
for the COADS and MDB versions of other TDF-11 decks that generally are
comparable, or, if there are differences, COADS usually is higher.  Deck
189 is an exception during 1938-40, with MDB higher.  The presence of
noticable amounts of data in 1938 and 1940 disagrees with the documented
(Air Weather Service and Weather Bureau, 1957) deck 189 period-of-record
of 1939 and September 1945 through June 1955 ("the missing period of 1/1940-
8/1945 was during the German occupation of the Netherlands during World War
II").

It appears that both MDB deck 215 and TDF-11 deck 192 were derived from the
same form of original German punched cards.  According to the original card deck
reference manual for deck 192 (Air Weather Service and Weather Bureau, 1955):
     "This card deck was punched by the German Meteorological Service during
     the Nazi regime from German ship observations made during the period
     1859-1939.  Two significant card decks of comparable size were captured
     by the Allies during the course of World War II.  This card deck was
     captured by the British and the Kopenhagener Schluessel card deck (191)
     [note: land station data] was captured by the United States.  Agreement
     was reached during 1946 whereby each meteorological service would
     exchange with the other a complete reproduction of the card deck which
     they held.

     Copies of most of the codes and coding instructions used by the German
     Meteorological Service preparatory to the punching of this card deck
     were received with the reproduced cards.  After these instructions were
     translated and studied it was found that they were incomplete.  Some
     additional information was obtained from other sources but it was
     necessary to reconstruct, from the cards, some codes and instructions.
     It is believed that the original records, from which this card deck
     was punched, were destroyed during the course of the war."

Moreover, the reference manual for deck 215 (UK Met. Office, 1980) describes
an identical 80-character punched card field layout to that in the deck 192
reference manual.  However, neither manual provides a card-count, and there
are some differences (usually minor) between the two reference manuals in
the interpretation of the field configurations and their meanings.

Overall (from Table 1) deck 215 contains 15% (975,567) more reports than deck
192.  From Fig. 1a note that deck 192 contains more data for 1868-1904; then
the pattern reverses and deck 215 contains more data through 1914, and also
during 1922-33.  In remaining years the counts for the two decks are generally
closer or data are extremely sparse in both sources (i.e., prior to 1868,
and during 1915-18; with World War I covering 1914-18).  The example maps of
observational density for decks 215 and 192 (Figs. 2a-2b) and their differences
(Figs 3a-3b), reveal many variations in coverage, and as noted in sec. 2 some
fragmentary ship tracks (e.g., in the Pacific) seem incomplete unless the two
decks were combined.

If decks 192 and 215 actually were based on the same original German punched
cards, these discrepancies in temporal and spatial coverage appear to raise
questions about whether omissions might have occurred in the reproduction or
exchange of the cards between the US and UK, or the possibility of subsequent
errors in processing or archival.  It is not known whether any records exist
in the UK or US that might document the number of original punched cards, nor
how much information was shared about the reconstruction of the incomplete or
missing codes and instructions.  These or other factors might have introduced
differences in independent software translations of the original cards into
the separate TDF-11 and MDB formats.  In cases where deck 192 counts are lower
it should be noted that duplicate elimination processing that was previously
applied in creation of the Atlas File might have had an impact in reducing the
counts (e.g., if the original punched cards contained internal duplicates).

In discussing data available for possible incorporation into the HSST project,
Verploegh (1966, p. 8) had these comments about German observations:
     "The punching of observations started in 1941 with the most recent data,
     working backwards.  In 1946 the available deck of cards was reproduced
     for the benefit of U.K., U.S.A. and the Netherlands; the 5.5 million
     observations mainly covered the period 1920-39; earlier data are however
     also included.  The copied data were not administrated.  In 1956 about
     0.6 million cards were reproduced for the benefit of U.S.A.; the cards
     refer to marine observations made at 12 a.m. in all oceans except the
     North Atlantic Ocean, and cover the period 1906-1939.  These two sets
     of data form together U.S.A. deck no. 3 (para 1.1) in which a certain
     unknown set of observations may appear double because of the fact that
     the first set has not been administrated.

     At present about 14.5 million marine observations covering the period
     1889-1960 are preserved at the Seewetteramt in Hamburg.  The systematic
     punching of earlier data continues.  [Note: For comparison, a more recent
     estimate (Wagner, 1999) is 19 million observational records (with some
     16 million digitized) for the period 1829-1944 (but mainly 1851-1944).]

     Thus about 8 to 9 million observations from German ships are available
     only at the Seewetteramt in Hamburg, in addition to the deck of about
     5.5 million German data which is also preserved in other countries.
     The bulk of the additional data is confined to the Atlantic Ocean and
     adjacent seas."

"U.S.A. deck no. 3" is described in paragraph 1.1 of Verploegh (1966, p. 2)
as German Marine Observations covering 1859-1939, with a volume of 6,104,576
cards.  That volume is less than the total for either deck 192 or 215 in
Table 1, and the statement that "5.5 million observations mainly covered
1920-39" seems difficult to reconcile with Figure 1a.  In agreement with
Verploegh, however, it appears that some deck 192 data are limited to local
noon observations.  Using a 10-deg box west of the US (30-40N, 130-140W),
Woodruff (1990, Fig. 10) traced to deck 192 a concentration of reports at
2100 UTC (i.e., local noon), particularly around 1900.  More research would
be needed to assess the full extent of this observational bias in deck 192
(may only apply to reports with type of logbook labeled "sailing vessel").

About 6.2M of NCDC's original deck 192 cardimages were recently recovered
via FOSDIC (Film Optical Sensing Device Input to Computer) from microfilm
(i.e., about 400K fewer reports than listed in Table 1 for deck 192).  We
completed some limited cross checks between these data and matching reports
from decks 192 and 215 (Table 2).  Sea level pressure (SLP) in deck 215 is
frequently lower than that in deck 192 by about 0.2 hPa (Figure 4).  As
detailed in Table 2, we speculate that most of the differences were due to
use of a truncated constant for conversion from millimeters to hPa.

Figure caption:
     Figure 4.  Histogram of differences of SLP for apparent duplicate report
     matches between decks 192 and 215 (deck 192 minus 215).  These differences
     were based only on undetected duplicates in the beta (preliminary) Release
     1c output LMRF (i.e., the distribution of SLP differences between detected
     duplicates, which was non-zero about 95%-99% of the time, is not shown).
     [Note:  In the beta run, many duplicates appear to have gone undetected
     for these two decks (plus a few other deck combinations) due to unequal
     SLP.  Dupelim rules were adjusted for the re-run to consider some small
     SLP differences as equivalent for purposes of detecting dups.]

Example report matches were also examined between the COADS and MDB versions of
Dutch deck 193.  These contained larger pressure differences (e.g., 1-2 hPa).
As was discussed in Woodruff (1990), the MDB values likely were not adjusted
for gravity.


Table 2.  Example report matches between decks 192 and 215 (selected fields as
abbreviated*), compared to a partial conversion of the original cards used for
deck 192 from FOSDIC.  In this set of examples, other major data elements such
as temperature and wind agreed precisely between decks 192 and 215.  Separate
fields for pressure were available in the original cards in mm and mb (hPa).
The last two columns show the results of converting the original mm value to
hPa.  In the first example, the original mm field was missing, and the tenths
hPa position was blank.  Only in this case does pressure agree between decks
192 and 215.  If these patterns are representative, a biased conversion from
mm to hPa may have been applied in creation of MDB deck 215.**
-------------------------------------------------------------------------------
  YR MO DY HR# LON LAT DCK   SLP II       ID     Original mm  Conversion->hPa
===============================================================================
1863  3 13  6 3535 595 192 10140 10 10000049
1863  3 13  6 3535 595 215 10140  9 00010
original:  06              1014                    missing          ---

1864  7 20 16 3595 505 192 10147 10 62030794
1864  7 20 16 3595 505 215 10145  9 00062
original:  66              10147                    761.1         1014.72 

1864  7 20 20 3595 505 192 10127 10 62030794
1864  7 20 20 3595 505 215 10125  9 00062
original:  70              10127                    759.6         1012.72 

1864  9 28 12 3595 505 192 10246 10 62030795
1864  9 28 12 3595 505 215 10244  9 00062
original:  62              10246                    768.5         1024.58 

1864  9 28 16 3585 505 192 10251 10 62030795
1864  9 28 16 3585 505 215 10249  9 00062
original:  66              10251                    768.9         1025.12 

1864  9 28 20 3575 505 192 10230 10 62030795
1864  9 28 20 3575 505 215 10228  9 00062
original:  70              10230                    767.3         1022.98 

1864  9 29  0 3575 505 192 10230 10 62030795
1864  9 29  0 3575 505 215 10228  9 00062
original:  50              10230                    767.3         1022.98 

1865  5 20  8 3595 505 192 10239 10 62031169
1865  5 20  8 3595 505 215 10237  9 00062
original:  58              10239                    768.0         1023.92 

1865  5 20 12 3595 505 192 10228 10 62031169
1865  5 20 12 3595 505 215 10227  9 00062
original:  62              10228                    767.2         1022.85 

1865  5 20  4 3585 505 192 10227 10 62031169
1865  5 20  4 3585 505 215 10225  9 00062
original:  54              10227                    767.1         1022.72 
-------------------------------------------------------------------------------
* Abbreviations:
     YR: year
     MO: month
     DY: day
     HR: hour (UTC = GMT)
    LON: longitude (10ths deg E)
    LAT: latitude  (10ths deg)
    DCK: deck
    SLP: sea level pressure (10ths hPa)
     II: ID indicator: 9=national ship number
                      10=composite information from early ship data
     ID: identification field
     The ID field 00062 in deck 215 was derived from "logbook type" in the
     original cards, such that 62 is "German and other expeditions."  The ID
     field in deck 192 is the concatenation of the 2-digit logbook type plus
     a 6-digit "code sheet number."  During final Release 1c processing of
     deck 215 we extracted code sheet number, when available from the original
     cardimage, and rearranged the ID to be identical in format to that from
     deck 192.
** If the truncated factor 1.333 was used in creation of deck 215, rather
than 1.333224, this would account for all the above discrepancies.
# Original hour is either:
     00-23: 0000-2300 GMT
     50-73: ship time (LST plus 50)
(this particular set of examples is close to Greenwich meridian).
----------


References

Air Weather Service and Weather Bureau, 1955: Reference Manual: 192 Deutsche
Seewarte Marine 1859.  HQ, Air Weather Service, Data Control Div., and Weather
Bureau, Climatological Services Div., 10 February 1955, 14 pp. [Available from
NOAA/National Climatic Data Center].

Air Weather Service and Weather Bureau, 1957: Reference Manual: 189 Dutch
Marine 1939, 1945-1955.  HQ, Air Weather Service, Data Control Div., and
Weather Bureau, Climatological Services Div., July 1955, 7 pp. [Available
from NOAA/National Climatic Data Center].

NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual.  NCDC,
Asheville, NC.

UK Met. Office, 1980: Reference Manual for Series 15 (German) Data, 1860-1939,
Met. Office, Bracknell, UK, 25 pp. [Compiled from Notes, Coding Manuals, and
WMO Conference Reports by B.S. Fullagar, Met O 22a, June 1980].

Verploegh, G., 1966: Report of the Consultant on the Historical Sea-Surface
Temperature Data Project, 49 pp. plus Annexes.  [Notes: The Consultant was to,
but the report appears not to have been archived by, the World Meteorological
Organization.  Copies were obtained courtesy of the UK Met Office and KNMI.]

Wagner, V., 1999: Historical Data and Its Processing in the Deutscher
Wetterdienst.  Proceedings of the International Workshop on Digitization and
Preparation of Historical Surface Marine Data and Metadata (Toledo, Spain,
15-17 September 1997). H.F. Diaz and S.D.  Woodruff, Eds., WMO/TD-No.957,
MMROA Report No. 43, World Meteorological Organization, Geneva, 19-29.

Woodruff, S.D., 1990: Preliminary comparison of COADS (US) and MDB (UK) ship
reports.  Observed Climate Variations and Change: Contributions in Support of
Section 7 of the 1990 IPCC Scientific Assessment, Parker, D. (Ed.), WMO/UNEP
publication.

[Delayed-mode (ICOADS.DM) Archive][Release 1c (1784-1949)]