ICOADS Web information page (Wednesday, 21-Nov-2007 17:56:37 UTC):

UK Royal Navy (RN) WW II Logs and the Marine Data Bank (MDB)

1. Introduction

This webpage (plus Addendum) provides a collection of information about UK Royal Navy (RN) logbooks during the World War II period. This includes results that shed some light on the extent to which meteorological data from the logbooks were previously digitized into the UK Marine Data Bank (MDB)-and thus are already available in ICOADS. Two separate, and partially overlapping, logbook collections are available:

Ship's Logs: archived at The UK National Archives (TNA). In Rhodes (1994) (hereafter R94), these are referred to as "Deck Logs." Under a joint UK/US project, a selection of approximately 260K pages from Ship's Log's for 1938-47 was imaged (in b&w) by early 2006, with digitization of the meteorological data by the NOAA Climate Database Modernization Program (CDMP) planned to begin in the next several months. Monthly Logs from submarines are also included in this project. Significant numbers of the imaged (daily observational) pages were recorded in port, but at least some meteorological observations generally were still made at anchor (e.g., sea level pressure, as shown in Figure 1a; Figure 1b illustrates the format from a submarine Log). Little or no instrumental metadata appear to be available from the Ship's Logs, whereas they contain considerable ship operational and personnel details, in addition to the meteorological observations. However, the cover and following page (Figures 1c-1d) contain some overall ship metadata, such as tonnage and ship type ("Navy List Description"). R94 indicates that the Ship's Logs were "made by professional mariners but with ships instrumentation."

Meteorological (Met) Logs: archived at the UK Met Office. These higher-quality observations, apparently limited to "large warships or the senior ship of a squadron" were taken by "qualified Meteorological Officers...from precision instruments," who "did not normally take observations during the hours of darkness and therefore most logs record observations only three times a day usually 0800, 1200 and 1800. However some logs do contain considerably more observations than this normal standard..." (from R94). Each Met Log includes a standard set of instructions to the observer, and the Logs include space for instrumental metadata plus for some more specialized observation types (discussed below).

Apparently, none of the data from the Ship's Logs have previously been digitized. However, data from selected Met Logs were keyed into MDB deck 204 (plus possibly also into deck 207 for some years, as discussed in sec. 2). Early MDB documentation also refers to these as "British Navy (HM) Ships" or "Admiralty logs." A large handwritten list (UK Met Office, c. 1964) indicates which ships and Met Logs were punched, with the initial instruction: "observations at anchor should not be punched." For the 1936-48 period, R94 also found that not all observations were keyed under a "policy [that] appears to have been to take only 3 or 4 for each day whatever number was available" (perhaps 15% of the observations were undigitized). Among the keyed data, R94 estimated an 8% level of keying errors, most frequently in calculating UTC from local time. Moreover, the original card form (see UK Met Office, c. 1960s) appears to have been limited to 65 characters (later reformatted to 80).

Possibly with the limited space on the card form as one factor, none of the instrumental metadata was keyed from the Met Logs into the MDB (nor was any ship name or other ID information keyed, as discussed in sec. 2). Based on a limited inspection in 2005 (i.e., of the six sample Met Logs described in sec. 3), Philip Brohan provided the following notes on the metadata and ancillary data types:

    "The barometer height and location are usually given, sometimes the thermometer height and location also-in a couple of cases details on the instrument source and calibration is given. There is also a set of standard instructions to observers inside each logbook. But the amount and quality varied dramatically even through this sample of 6 logs.
     The met logs contain a wealth of data-as well as the routine obs there are barograph charts and synoptic plots pasted in, wind records from balloon launches and occasional tephigrams from aeroplane launches. There are daily comments on the weather, and annotations on the barograph charts.
     I didn't copy the logs-the pasted in inserts, size of the log-books, and restricted photocopying facilities in the Archive mean that this would have taken a very long time for a poor result."

Sec. 2 of this webpage describes the coverage during 1930-49 of the MDB data already in ICOADS, including the selected data digitized (possibly around the 1960s) from Met Logs into deck 204. Sec. 3 provides the results of comparisons between Ship's Logs for six sample voyages, and MDB deck 204 data that were reported at the same approximate locations.

2. Temporal and Spatial Distribution of MDB Decks for 1930-49

This section illustrates the spatial (Figures 2a-2t) and temporal (Figure 3, below) distribution of the MDB data (provided during 1998-2000 by Jim Arnott, UK Met Office, in LMR format) during the period 1930-49. These data were blended into the current ICOADS Release (originally as part of Release 1c, covering 1784-1949). However, these plots show the data coverage prior to blending, and thus before any duplicates have been removed. Table 1 lists the applicable decks.

Figures 2a-2t highlight deck 204, which consists of data keyed from the Met Logs. UK Met Office (c. 1960s) describes series 4 (deck 204) as:

"Data from Admiralty logs for years 1930-48. Coded in 1930 code."

That MDB documentation also describes two temporal groups of series 7 (deck 207) data:

"Data for years 1945-48. Coded in 1930 code."
"Data for years 1949-54 including data from Admiralty logs. Coded in 1949 code."

Based on this information, deck 207 would appear to represent an extension of data from "Admiralty logs" (earlier present in deck 204), following the 1949 code change. But whether the indicated 1945-48 portion of deck 207 might have included any data from Admiralty logs seems unclear. Moreover, Table 1 refers to deck 207 as "Selected Ships," and does not even indicate an approximate period extending past 1948.

UK Met Office (c. 1960s) also provides the following information about Folio number (which, if available, might serve as a useful identification field in the absence of ship ID): "The number of the log in which the observations were recorded. For Series 5 these columns were not punched. The name of the ship making the observation was written on the back of the card. For data from Admiralty logs, both Series 4 and those included in Series 7, these columns were not punched, the log number being written on the back of the card." (Note: Very likely, the punched cards referred to were discarded many years ago.)

Otherwise, the original card form contained no space for any ship ID information. Consequently, for the WW II period deck 204 (and 207) lacks any form of ship ID. This is an unfortunate omission, because it complicates the comparison of deck 204 with the overlapping Ship's Logs (as discussed in sec. 3) and can hamper later quality controls, e.g., by precluding track-checking based on the ID.

Table 1: UK Met. Office MDB deck 201-8, 216 assignments (equivalent to MDB series numbers 1-8, 16). Descriptions and approximate time periods from overall MDB documentation (received separately from UK Met Office, c. 1960s; excerpted from Table 6b of LMR documentation). Additional MDB deck numbers in the range 209-255 refer to data outside the 1930-49 period, or non-UK ship (e.g., international exchange) data. Except for deck 216, actual time periods are shown to deviate to some extent, just based on data availability during the 1930-49 period, from those indicated in the overall MDB documentation. For example, no data whatsoever were located in this period for deck 208 (see also Figure 2).

Within 1930-49:
Deck	Description	Approx. period	Data period	No. reports
201	All Ships (1930 code)	1850-1920	1930-1949	180311
202	All Ships (1921 code)	1921-1929	1930-1938	97
203	Selected Ships (1930 code)	1920-1939	1930-1944	403386
204	British Navy (HM) Ships (1930 code)	1930-1948	1930-1949	117179
205	Scottish Fishery Cruisers MARIDS (1930 code)	1946-1956	1945-1949	21901
206	Ocean Weather Stations (OWS) (1930 code)	1947-1949	1947-1948	1642
207	Selected Ships (1930 code)	1945-1948	1945-1949	390923
208	Light Vessels	1949-1956		0
216	UK Merchant Ship Logbooks (METFORMS; keyed in 1996)	1935-1939	1935-1939	458756

In this section, we show results from attempting to match MDB deck 204 (i.e., the existing digital version of the Met Logs; input LMR) with six sample Ship's Logs (Table 2).

Table 2. Sample Ship's Logs photographed (in color by Clive Wilkinson in 2005) for this comparison. Each Ship's Log covers one month, with the observations for each day recorded on a separate page (e.g., Figure 1a; Figures 1c-1d illustrate the cover and page following, which appear at the beginning of each Ship's Log).

Because deck 204 contains no ship ID information, data were extracted from deck 204 based simply on their temporal and spatial proximity to the Ship's Logs data. First, the recorded local noon positions were digitized from the sample Ship's Logs (some daily logbook pages lacked a recorded position, e.g., in port, and were omitted). Then all the local noon reports were extracted from deck 204 that fell within 10° of latitude and longitude of each Ship's Log position, yielding 61 extracted reports. No approximately matching reports from deck 204 were located for the Shropshire (which was in port for the entire month), so that ship was omitted from the following results.

For the five remaining voyages, Figures 4a-4e compare the spatial locations digitized from the Ship's Logs, with the 61 (approximately matching) noon positions extracted from deck 204. Note that in a number of cases, an approximately matching deck 204 report was not located.

Similarly, Figures 5a-5e compare the spatial locations digitized from the Ship's Logs, with the 61 noon positions extracted from deck 204. In this case, latitude and longitude are each plotted separately against day of the month, more clearly revealing differences in some cases between the positions from the two data sources.

Table 3 lists a selection of fields from the 61 extracted deck 204 reports, and provides access (via links) to the corresponding daily page from each Ship's Log. Based on preliminary comparisons between the fields from deck 204 as listed in Table 2, with the local noon reports from the corresponding Ship's Log pages, the meteorological fields from the two sources frequently appear to match up very poorly. However, a more detailed comparison of these 61 cases is planned.

RN94 provides several additional indications that the Ship's Logs and the Met Logs may be independent (or quasi-independent). As discussed in sec. 1, the Ship's Logs were reported by professional mariners with ship's instruments; whereas the Met Logs were recorded by Meteorological Officers with precision instruments.

In addition, as a further complication, it appears that the data in the Met Logs may sometimes be of mixed origin. An example Met Log page that is reproduced in Annex C2 of R94 includes the Remarks: "Entries marked with a red X are read and made by the Met Officer himself. The remainder are taken from the Met Log, and may not be very accurate. Barometer readings from the latter are entered to the nearest mb."

Table 3. The 61 local noon reports from deck 204, whose positions appear in Figures 4a-4e and 5a-5e. These fields represent a subset of the complete deck 204 report, and have been converted from the LMR format back into units closely resembling the form of data as recorded in the Ship's Logs (e.g., local hour, wind speed in Beaufort force, sea level pressure in both mb and inches, and temperatures in Fahrenheit). Each link in the table provides access to the corresponding Ship's Log photograph (All by courtesy of The National Archives, Kew (UK); TNA catalog references for the six voyages are listed below).

Rhodes, Martin H., 1994: Additional Marine Log Book Data, 1936-48. [UK Met Office report contracted (together with two accompanying reports covering 1856-99 and 1911-20) to examine the archives "to see whether all data had been transferred to the Marine Data Bank at the Meteorological Office."] (pdf, 4MB)

UK Met Office, c. 1960s: Reference Manual for Surface Marine Card Form 789 [Series 1-7; 1854-1953]. (pdf, 1.1MB)

UK Met Office, c. 1964: [Large untitled handwritten list of ships with Met Logs in the Met Office.] (pdf; 47MB)