ICOADS Web information page (Tuesday, 08-Nov-2005 23:59:41 UTC):

Comparisons of TDF-11 (Release 1) Decks and MDB Versions

1. Introduction

Decks from Tape Data Family-11 (TDF-11) (NCDC, 1968) formed a key component of COADS Release 1 (covering 1854-1979) data prior to 1970, and more recently for Release 1c (1784-1949). Selected TDF-11 decks also were included in the UK Meteorological Office Main Marine Data Bank (MDB). The blending of COADS and MDB data completed as part of Release 1c permitted comparison between TDF-11 decks that were originally used for COADS, and those also available from MDB (Table 1). Table 1. Description, and raw input counts to COADS Release 1c, of TDF-11 decks (plus MDB deck 215) overlapping between COADS and MDB through 1949 (MDB data extend only back to 1854, and in COADS only deck 193 extends prior to 1854 with a few scattered reports back to 1800). MDB deck 215 and TDF-11 deck 192 are compared because, as discussed in sec. 3, they are thought to be derived from the same original data source. ------------------------------------------------------------------------------- Deck Description COADS MDB =============================================================================== 110 US Navy Marine 534,658 443,950 116 US Merchant Marine 339,325 239,486 118 Japanese Ships No. 1 (Kobe keyed in 1961) 1,702,582 1,688,837 187* Japanese Whaling Fleet 2,227 2,224 188* Norwegian Antarctic Whaling Factory Ships 21,640 21,578 189 Netherlands Marine 115,623 90,895 192/215 Deutsche Seewarte Mar./Ser. 15 (German) 6,647,416 / 7,622,983 193 Netherlands Marine 6,402,700 6,281,052 195 US Navy Ships Logs 603,473 609,007 196* Deutsche Seewarte Marine (192 extension) 19,805 19,816 197* Danish (and Other) Marine (Polar) 20,988 20,770 281 US Navy Monthly Aerological Record (MAR) 190,585 193,039 ------------------------------------------------------------------------------- * Deck with negligible overall COADS-MDB difference; not compared further (decks not marked "*" were subject to more detailed comparisons in sec. 2). ----------

2. Temporal and spatial comparisons by deck

Annual time-series comparisons of the numbers of input reports from COADS and MDB are presented in Figures 1a-1h for the decks identified as having significant differences in Table 1 (the deck 192/215 differences are the most significant; presented first in Figure 1a). Figures 2a-2b illustrate, for two example years with large overall differences in counts in Figure 1a, input spatial coverage in deck 192 versus that in deck 215. In Fig. 2a, note in the sparsely sampled Pacific fragmentary ship tracks in both decks, which would appear to be connected only by combining the unique segments from the two decks. Also in Fig. 2a in the Indian Ocean, deck 215 contains some additional tracks. Similar features exist in Fig. 2b, e.g., a ship track that is more complete in deck 192 (215) in the northern (southern) Pacific. Figures 3a-3b are difference maps of the spatial coverage provided by deck 215 minus that in deck 192 for the same two example years for Figs. 2a-2b. Note the presence of both positive and negative differences in heavily sampled regions such as the Atlantic and Indian Oceans. Figure captions: Figure 1a. Annual input counts for TDF-11: deck 192 + MDB deck: 215. Figure 1b. Annual input counts for TDF-11 + MDB deck: 110. Figure 1c. Annual input counts for TDF-11 + MDB deck: 116. Figure 1d. Annual input counts for TDF-11 + MDB deck: 118. Figure 1e. Annual input counts for TDF-11 + MDB deck: 189. Figure 1f. Annual input counts for TDF-11 + MDB deck: 193. Figure 1g. Annual input counts for TDF-11 + MDB deck: 195. Figure 1h. Annual input counts for TDF-11 + MDB deck: 281. Figure 2a. Spatial distribution by 1-degree boxes of input counts for COADS deck 192 (upper panel) versus MDB deck 215 (lower panel) for 1923. Figure 2b. As for Fig. 2a, except for 1927. Figure 3a. Spatial distribution by 1-degree boxes of input counts for MDB deck 215 minus COADS deck 192 for 1923. Figure 3b. As for Fig. 3a, except for 1927.

3. Discussion

Most of the TDF-11 data used for Release 1 were taken from the "Atlas File," which was subject to some editing and compositing of observations, rather than original TDF-11 data. In contrast, the UK Met. Office may have received the original (pre-Atlas) TDF-11 data (see Woodruff, 1990). Decks 192 and 194 (Great Britain Marine) were deliberately omitted from MDB (Woodruff, 1990): deck 192 presumably because of the existence in MDB of deck 215, and deck 194 because it was derived (by the US) from more original MDB data. Figure 1a illustrates many differences between the annual counts of COADS and MDB for decks 192 and 215, with deck 215 sometimes significantly higher (discussed in more detail below). In contrast, Figures 1b-1g show counts for the COADS and MDB versions of other TDF-11 decks that generally are comparable, or, if there are differences, COADS usually is higher. Deck 189 is an exception during 1938-40, with MDB higher. The presence of noticable amounts of data in 1938 and 1940 disagrees with the documented (Air Weather Service and Weather Bureau, 1957) deck 189 period-of-record of 1939 and September 1945 through June 1955 ("the missing period of 1/1940- 8/1945 was during the German occupation of the Netherlands during World War II"). It appears that both MDB deck 215 and TDF-11 deck 192 were derived from the same form of original German punched cards. According to the original card deck reference manual for deck 192 (Air Weather Service and Weather Bureau, 1955): "This card deck was punched by the German Meteorological Service during the Nazi regime from German ship observations made during the period 1859-1939. Two significant card decks of comparable size were captured by the Allies during the course of World War II. This card deck was captured by the British and the Kopenhagener Schluessel card deck (191) [note: land station data] was captured by the United States. Agreement was reached during 1946 whereby each meteorological service would exchange with the other a complete reproduction of the card deck which they held. Copies of most of the codes and coding instructions used by the German Meteorological Service preparatory to the punching of this card deck were received with the reproduced cards. After these instructions were translated and studied it was found that they were incomplete. Some additional information was obtained from other sources but it was necessary to reconstruct, from the cards, some codes and instructions. It is believed that the original records, from which this card deck was punched, were destroyed during the course of the war." Moreover, the reference manual for deck 215 (UK Met. Office, 1980) describes an identical 80-character punched card field layout to that in the deck 192 reference manual. However, neither manual provides a card-count, and there are some differences (usually minor) between the two reference manuals in the interpretation of the field configurations and their meanings. Overall (from Table 1) deck 215 contains 15% (975,567) more reports than deck 192. From Fig. 1a note that deck 192 contains more data for 1868-1904; then the pattern reverses and deck 215 contains more data through 1914, and also during 1922-33. In remaining years the counts for the two decks are generally closer or data are extremely sparse in both sources (i.e., prior to 1868, and during 1915-18; with World War I covering 1914-18). The example maps of observational density for decks 215 and 192 (Figs. 2a-2b) and their differences (Figs 3a-3b), reveal many variations in coverage, and as noted in sec. 2 some fragmentary ship tracks (e.g., in the Pacific) seem incomplete unless the two decks were combined. If decks 192 and 215 actually were based on the same original German punched cards, these discrepancies in temporal and spatial coverage appear to raise questions about whether omissions might have occurred in the reproduction or exchange of the cards between the US and UK, or the possibility of subsequent errors in processing or archival. It is not known whether any records exist in the UK or US that might document the number of original punched cards, nor how much information was shared about the reconstruction of the incomplete or missing codes and instructions. These or other factors might have introduced differences in independent software translations of the original cards into the separate TDF-11 and MDB formats. In cases where deck 192 counts are lower it should be noted that duplicate elimination processing that was previously applied in creation of the Atlas File might have had an impact in reducing the counts (e.g., if the original punched cards contained internal duplicates). In discussing data available for possible incorporation into the HSST project, Verploegh (1966, p. 8) had these comments about German observations: "The punching of observations started in 1941 with the most recent data, working backwards. In 1946 the available deck of cards was reproduced for the benefit of U.K., U.S.A. and the Netherlands; the 5.5 million observations mainly covered the period 1920-39; earlier data are however also included. The copied data were not administrated. In 1956 about 0.6 million cards were reproduced for the benefit of U.S.A.; the cards refer to marine observations made at 12 a.m. in all oceans except the North Atlantic Ocean, and cover the period 1906-1939. These two sets of data form together U.S.A. deck no. 3 (para 1.1) in which a certain unknown set of observations may appear double because of the fact that the first set has not been administrated. At present about 14.5 million marine observations covering the period 1889-1960 are preserved at the Seewetteramt in Hamburg. The systematic punching of earlier data continues. [Note: For comparison, a more recent estimate (Wagner, 1999) is 19 million observational records (with some 16 million digitized) for the period 1829-1944 (but mainly 1851-1944).] Thus about 8 to 9 million observations from German ships are available only at the Seewetteramt in Hamburg, in addition to the deck of about 5.5 million German data which is also preserved in other countries. The bulk of the additional data is confined to the Atlantic Ocean and adjacent seas." "U.S.A. deck no. 3" is described in paragraph 1.1 of Verploegh (1966, p. 2) as German Marine Observations covering 1859-1939, with a volume of 6,104,576 cards. That volume is less than the total for either deck 192 or 215 in Table 1, and the statement that "5.5 million observations mainly covered 1920-39" seems difficult to reconcile with Figure 1a. In agreement with Verploegh, however, it appears that some deck 192 data are limited to local noon observations. Using a 10-deg box west of the US (30-40N, 130-140W), Woodruff (1990, Fig. 10) traced to deck 192 a concentration of reports at 2100 UTC (i.e., local noon), particularly around 1900. More research would be needed to assess the full extent of this observational bias in deck 192 (may only apply to reports with type of logbook labeled "sailing vessel"). About 6.2M of NCDC's original deck 192 cardimages were recently recovered via FOSDIC (Film Optical Sensing Device Input to Computer) from microfilm (i.e., about 400K fewer reports than listed in Table 1 for deck 192). We completed some limited cross checks between these data and matching reports from decks 192 and 215 (Table 2). Sea level pressure (SLP) in deck 215 is frequently lower than that in deck 192 by about 0.2 hPa (Figure 4). As detailed in Table 2, we speculate that most of the differences were due to use of a truncated constant for conversion from millimeters to hPa. Figure caption: Figure 4. Histogram of differences of SLP for apparent duplicate report matches between decks 192 and 215 (deck 192 minus 215). These differences were based only on undetected duplicates in the beta (preliminary) Release 1c output LMRF (i.e., the distribution of SLP differences between detected duplicates, which was non-zero about 95%-99% of the time, is not shown). [Note: In the beta run, many duplicates appear to have gone undetected for these two decks (plus a few other deck combinations) due to unequal SLP. Dupelim rules were adjusted for the re-run to consider some small SLP differences as equivalent for purposes of detecting dups.] Example report matches were also examined between the COADS and MDB versions of Dutch deck 193. These contained larger pressure differences (e.g., 1-2 hPa). As was discussed in Woodruff (1990), the MDB values likely were not adjusted for gravity. Table 2. Example report matches between decks 192 and 215 (selected fields as abbreviated*), compared to a partial conversion of the original cards used for deck 192 from FOSDIC. In this set of examples, other major data elements such as temperature and wind agreed precisely between decks 192 and 215. Separate fields for pressure were available in the original cards in mm and mb (hPa). The last two columns show the results of converting the original mm value to hPa. In the first example, the original mm field was missing, and the tenths hPa position was blank. Only in this case does pressure agree between decks 192 and 215. If these patterns are representative, a biased conversion from mm to hPa may have been applied in creation of MDB deck 215.** ------------------------------------------------------------------------------- YR MO DY HR# LON LAT DCK SLP II ID Original mm Conversion->hPa =============================================================================== 1863 3 13 6 3535 595 192 10140 10 10000049 1863 3 13 6 3535 595 215 10140 9 00010 original: 06 1014 missing --- 1864 7 20 16 3595 505 192 10147 10 62030794 1864 7 20 16 3595 505 215 10145 9 00062 original: 66 10147 761.1 1014.72 1864 7 20 20 3595 505 192 10127 10 62030794 1864 7 20 20 3595 505 215 10125 9 00062 original: 70 10127 759.6 1012.72 1864 9 28 12 3595 505 192 10246 10 62030795 1864 9 28 12 3595 505 215 10244 9 00062 original: 62 10246 768.5 1024.58 1864 9 28 16 3585 505 192 10251 10 62030795 1864 9 28 16 3585 505 215 10249 9 00062 original: 66 10251 768.9 1025.12 1864 9 28 20 3575 505 192 10230 10 62030795 1864 9 28 20 3575 505 215 10228 9 00062 original: 70 10230 767.3 1022.98 1864 9 29 0 3575 505 192 10230 10 62030795 1864 9 29 0 3575 505 215 10228 9 00062 original: 50 10230 767.3 1022.98 1865 5 20 8 3595 505 192 10239 10 62031169 1865 5 20 8 3595 505 215 10237 9 00062 original: 58 10239 768.0 1023.92 1865 5 20 12 3595 505 192 10228 10 62031169 1865 5 20 12 3595 505 215 10227 9 00062 original: 62 10228 767.2 1022.85 1865 5 20 4 3585 505 192 10227 10 62031169 1865 5 20 4 3585 505 215 10225 9 00062 original: 54 10227 767.1 1022.72 ------------------------------------------------------------------------------- * Abbreviations: YR: year MO: month DY: day HR: hour (UTC = GMT) LON: longitude (10ths deg E) LAT: latitude (10ths deg) DCK: deck SLP: sea level pressure (10ths hPa) II: ID indicator: 9=national ship number 10=composite information from early ship data ID: identification field The ID field 00062 in deck 215 was derived from "logbook type" in the original cards, such that 62 is "German and other expeditions." The ID field in deck 192 is the concatenation of the 2-digit logbook type plus a 6-digit "code sheet number." During final Release 1c processing of deck 215 we extracted code sheet number, when available from the original cardimage, and rearranged the ID to be identical in format to that from deck 192. ** If the truncated factor 1.333 was used in creation of deck 215, rather than 1.333224, this would account for all the above discrepancies. # Original hour is either: 00-23: 0000-2300 GMT 50-73: ship time (LST plus 50) (this particular set of examples is close to Greenwich meridian). ----------


Air Weather Service and Weather Bureau, 1955: Reference Manual: 192 Deutsche Seewarte Marine 1859. HQ, Air Weather Service, Data Control Div., and Weather Bureau, Climatological Services Div., 10 February 1955, 14 pp. [Available from NOAA/National Climatic Data Center]. Air Weather Service and Weather Bureau, 1957: Reference Manual: 189 Dutch Marine 1939, 1945-1955. HQ, Air Weather Service, Data Control Div., and Weather Bureau, Climatological Services Div., July 1955, 7 pp. [Available from NOAA/National Climatic Data Center]. NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual. NCDC, Asheville, NC. UK Met. Office, 1980: Reference Manual for Series 15 (German) Data, 1860-1939, Met. Office, Bracknell, UK, 25 pp. [Compiled from Notes, Coding Manuals, and WMO Conference Reports by B.S. Fullagar, Met O 22a, June 1980]. Verploegh, G., 1966: Report of the Consultant on the Historical Sea-Surface Temperature Data Project, 49 pp. plus Annexes. [Notes: The Consultant was to, but the report appears not to have been archived by, the World Meteorological Organization. Copies were obtained courtesy of the UK Met Office and KNMI.] Wagner, V., 1999: Historical Data and Its Processing in the Deutscher Wetterdienst. Proceedings of the International Workshop on Digitization and Preparation of Historical Surface Marine Data and Metadata (Toledo, Spain, 15-17 September 1997). H.F. Diaz and S.D. Woodruff, Eds., WMO/TD-No.957, MMROA Report No. 43, World Meteorological Organization, Geneva, 19-29. Woodruff, S.D., 1990: Preliminary comparison of COADS (US) and MDB (UK) ship reports. Observed Climate Variations and Change: Contributions in Support of Section 7 of the 1990 IPCC Scientific Assessment, Parker, D. (Ed.), WMO/UNEP publication.

[Delayed-mode (ICOADS.DM) Archive][Release 1c (1784-1949)]

U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads@noaa.gov
Updated: Nov 8, 2005 23:59:41 UTC