ICOADS Web information page (Wednesday, 11-May-2016 19:47:23 UTC):

US Marine Meteorological Journals Collection (deck 704; 1878-94) (by ESRL)

1. Introduction

The US Marine Meteorological Journals form part of Record Group (RG) 27, (Records of the Weather Bureau, Records of Marine Observations, Entry 127) at the US National Archives and Records Administration (NARA). The MMJ were transferred to NARA from the Weather Bureau in 1936, described by an Identification Inventory documenting that accession as follows:

"Log Books containing extracts of logs of oceangoing ships from 1879 to 1893. Accompanying the log books are three boxes containing a card index of the books, each card giving the name of the ship, master, route, dates en route, and the number assigned to the book. The log books were received from the U.S. Hydrographic Office about thirty years ago when the meteorological work of that office was transferred to the Weather Bureau. The log books were kept by the Master or another official designated by him and the information was sent to the Hydrographic Office where it was used for making up pilot charts and studying weather conditions. It is interesting to note that for a long period these books were sent to Lieutenant M.F. Maury, 'Pathfinder of the Seas,' at the Naval Observatory."

A Location Register obtained from NARA during a visit in 1989 referred to Entry 127 as containing 1,955 volumes (135 feet) from 1879-1893. During that visit, journal No. 2 was examined, resulting in the following description of the format:

"Leather-bound, 3/4" thick, landscape-orientation, 11.5" x 9" volume... A lengthy introduction section, including example journal pages with fictitious data printed in red.... Journal filled with genuine 2-hourly observations.... Stamped in red with barometer correction and 'Records transcribed at Pittsburgh, PA, 1933-34,' and with additional data interpretation marks in red pencil."
Entry 128 was described in the location register as the index to the MMJ (four feet of records, apparently referring to the card index described above).

This report provides additional details about the MMJ collection, gleaned from the 1989 visit, plus two subsequent visits in 1993 and 1997, to NARA. In 1998, agreement was reached between NARA and NCDC to microfilm the collection, as the initial step towards digitization of the marine meteorological observations.

In May 1998 a data exchange agreement was reached between the US and China, for China to digitize the MMJ collection, using paper copies created from the microfilm by NCDC. This document describes the digitization format, and this report from China's National Marine Data and Information Service (NMDIS) describes completion in late 2000 of the digitization and quality controls that were applied by China to the data. The digitized data were provided to NOAA's then-Climate Diagnostics Center (now within the Physical Science Division of the Earth System Research Laboratory) for conversion and blending into ICOADS.

The microfilm was eventually also converted into TIF image format by NOAA's Climate Database Modernization Program (CDMP), and is available from CDMP's archive (186,162 images). It was discovered later that 31 Journals were inadvertently misfiled at NARA among the US Maury Collection, and which have not yet been microfilmed or digitized. These missing Journals, listed here, are estimated to contain approximately 28K observations from 1879-90. Future work has been proposed to CDMP to image and digitize the missing Journals. CDMP work is also planned to make digitally available the card index to the complete collection (Entry 128).

Within the collection, three distinct editions of the Journals, and of the accompanying Instructions, have been located (determining whether other editions exist would require examining every Journal). The column structure (Table 1) for recording the data is the same in the August 1878 and February 1883 editions, but there are differences in the Instructions. The November 1884 edition marks an important change in the data, with the addition of columns for compass correction, true course, and true wind direction.

Table 1. The structure of columns used for recording the data in the three known Journal editions. The November 1884 edition transitions to 20 columns (adding: compass correction, ship's true course, and true wind direction) but the Instructions still group these under only 17 headings (as numbered in the instructions). The (year and month) dates of the editions are from p. 17 of the instructions (the Journal title page also contains a year, which is frequently later than that on page 17 and does not appear to be useful for dating the edition). PDFs of selected high-quality examples of the three editions can be downloaded from here.

No.August 1878, February 1883No.November 1884
2Ship's speed: knots2(same)
3Ship's speed: tenths3(same)
4Courses steered4(same)
5Winds: mean magnetic dir.5Compass correction
6Winds: mean force6Leeway
7Leeway7Ship's true course
8Winds: compass direction
Winds: true direction
Winds: mean force
8Barometer: height in inches9(same)
9Barometer: therm. attached(same)
10Therm.: air, dry bulb10(same)
11Therm.: air, wet bulb11(same)
12Therm.: water at surface12(same)
13State of weather by symbols13(same)
14Form of clouds by symbols14(same)
15Prop. of clear sky in tenths15(same)
16State of the sea16(same)
17Record of sail vessel is under17(same)

2. Translation results

The MMJ collection as digitized by China has now been translated into the Long Marine Report (LMR) and International Maritime Meteorological Archive (IMMA) formats (see specifications), for blending into ICOADS. The actual period of record of the digitized data was discovered to be 1878-94, with 1.76M resultant IMMA reports (58,562 reports, 3%, were rejected from the translation because latitude and longitude were not reported or could not be computed). Figure 1 shows the temporal coverage, and Figure 2 shows the overall, and annual, spatial coverage of the collection.

Figure 3 shows how the different variables were assessed by quality control (QC; "trimming"). The grey regions in the plot thus indicate that sea level pressure (SLP) is the best-sampled variable, followed by wind, sea surface and air temperatures (SST and AT), and relative humidity (RH), respectively. In addition, SLP has larger numbers of data flagged as suspicious by the trimming procedure than the temperatures and wind, whereas the trimming limits are almost entirely missing for RH.

Figure 4 shows the distribution of SLP and other variables with latitude, also indicating whether the data were within or outside the broadest (4.5σ) trimming thresholds (or that the limits were missing). The plot of SLP reveals some suspicious patterns of biased data, in agreement with the relatively high trimming rates shown in Figure 3; and the plots of wet bulb and dew point temperatures (WBT and DPT) show the impact of the missing trimming limits for RH (since those humidity variables are trimmed on the basis of the RH flag).

Figure 5 shows the availability of ship's course and wind direction information in the collection, clearly reflecting the extension of the Journal's structure after the November 1884 edition to include true course, true wind direction, and compass correction (ref. Table 1).

Figure 6 shows a ship's track before and after magnetic correction.

Figure 1
Figure 1. Temporal coverage of the MMJ collection, shown as thousands of reports per year (output IMMA). The bars are colored differently (in 2-year blocks) to correspond with Figure 2.

Figure 2

1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894

Figure 2. The default map shows the overall spatial coverage (composite of all years) of the MMJ collection, or the coverage of individual years is revealed by the links below the map (the colors used for the annual spatial patterns correspond to those used in Figure 1).

Figure 3

Figure 3. For the 1.76M reports in the MMJ collection, the first bar plot shows the numbers of reports made at local nighttime (ND=1) versus daytime (ND=2). The remaining bars show the values of the "trimming" (QC) flags for different variables:

SF:  sea surface temperature flag
AF:  air temperature flag
UF:  U wind flag
VF:  V wind flag
PF:  sea level pressure (SLP) flag
RF:  relative humidity flag
The flag values are defined in Table 2 (e.g., black: data within narrow 2.8σ trimming limits; grey: missing data; and blue: the absence of trimming limits).

Table 2. Description and definition of the trimming flag values, where ai is the individual observation under scrutiny, g is the smoothed median, and σ1 and σ5 are the smoothed lower and upper median deviation (values 8-A are currently unused). See the electronic documentation <stat_trim> for additional information.

1within 2.8σ limitsg - 2.8σ1aig + 2.8σ5
2less than 2.8σ lower limitg - 3.5σ1ai < g - 2.8σ1
3greater than 2.8σ upper limitg + 2.8σ5 < aig + 3.5σ5
4less than 3.5σ lower limitg - 4.5σ1ai < g - 3.5σ1
5greater than 3.5σ upper limitg + 3.5σ5 < aig + 4.5σ5
6less than 4.5σ lower limitai < g - 4.5σ1
7greater than 4.5σ upper limitai > g + 4.5σ5
Blimits missing (ocean/coastal box); MEDS data correct (SF/PF only)[Note: not plotted; not applicable to these data.]
Climits missing (ocean/coastal box)
Dlandlocked 2-degree box
Edata unusable (SF, AF, and PF, only; as defined in <stat_trim>)
Fdata missing or not computable (as defined in <stat_trim>).

Figure 4

SLP (default) or: AT WBT DPT SST

Figure 4. For the 1.6M reports with extant SLP in the MMJ collection, this plot shows the distribution of SLP with latitude. Black (green) indicates data within (outside) the 4.5σ trimming limits (or trimming limits missing). Similarly for AT, WBT, DPT, and SST.

Figure 5

Ship's (default) or: wind

Figure 5. Number of reports per year in the MMJ collection, stratified by the availability of the ship's course and compass correction information. Each digit in the 3-digit code used to identify the eight bar colors has the value 0 (data missing) or 1 (data extant). The three digits thus indicate the presence or absence of the following data, respectively: (1) course steered by compass, (2) ship's true course, and (3) compass correction. Similarly for wind direction, such that the three digits indicate the presence or absence of: (1) wind direction (magnetic), (2) true wind direction, and (3) compass correction.

Figure 6

Uncorrected (default) or: corrected

Figure 6. For several voyages, as detailed in the translation specifications, we found that the course steered by compass and wind direction (magnetic) were actually true. These voyages and wind data were therefore left uncorrected in the translation. For the Theodor Ruger Aug-Dec 1884 this figure shows the uncorrected course steered by compass, versus the impact of applying the compass corrections, which in most cases fragment and distort the ship's course.

3. Unresolved Issues

  • The general absence of trimming limits for RH during the time period of the collection (see Figures 3 and 4) represents a significant problem for the calculation of ICOADS 2° trimmed monthly summaries. Those climatological limits were calculated based on the original COADS Release 1 data (issued in 1985), in which very few humidity data appeared (i.e., during the earliest trimming period, 1854-1909, that was defined for Release 1). One interim solution that could be considered would be to use the RH limits for the next most recent trimming period (1910-49), as QC basis for these early humidity data.
  • Reel numbers (referring to the microfilm copy of the collection) and Journal numbers (referring to the original Journal numbering system) are stored both in the "index records" associated with the CDMP image collection, and in the digital data. In these fields, some errors are known to exist, as well as some unresolved inconsistencies between the index records and the digital data, which it may be possible to correct or improve through future work. In addition, the ship names as separately recorded in the index records and in the digital data contain variations and inconsistencies, which it might also be beneficial to edit for greater uniformity (e.g., to better support ship track checking or linkage with external ship metadata).
  • As discussed in sec. 2, 3% of the data (about 59K reports) were rejected from the translation because latitude and longitude were not reported or could not be computed. The calculation of longitude and latitude was basically limited to location information available on a given calendar day (i.e., one Journal page). The possibility of interpolation of positions across days could be considered in the future, if the collection was reprocessed, as a means of recovering some of the rejected observations.

[Documentation and Software][Translation information]

U.S. National Oceanic and Atmospheric Administration hosts the icoads website privacy disclaimer
Document maintained by icoads&#64;noaa.gov
Updated: May 11, 2016 19:47:23 UTC