=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.1 Data Preconditioning and Duplicate Elimination: 1784-1949 27 February 2004 ================================================================= Document Revision Information (previous version: 9 September 2002): Updates for Release 2.1 and ICOADS. ------------------------------------------------------------------------------- {1. Introduction} This document describes the rules for a sequence of processing steps performed in the duplicate elimination (dupelim) program for 1784-1949 (originally referred to as COADS Release 1c) data. Preconditioning (sec. 2), the first step in this sequence, was used to delete Long Marine Reports (LMR6), or to correct or modify individual data fields within a given report. The second step involved setting the LMR fields for platform type (PT) and ID indicator (II) (sec. 3). The final step was actual QC/dupelim processing (sec. 4). During dupelim, additional reports were eliminated, and a limited number of changes was made to the contents of reports by substitution between duplicates. [NOTE: Because of processing differences, the three original COADS updates that compose ICOADS.DM, and accompanying documentation, are referred to as follows: Release 1a: 1980-97 Release 1b: 1970-79 1946-69 Release 1c: 1784-1949 These four documents describe the "preconditioning" and duplicate elimination processing used to create LMR for the indicated periods. 1946-49 Release 1b data were replaced by Release 1c data.] {2. Preconditioning} Sec. 2.1 gives the rules for report deletions; sec. 2.2 gives the rules for field modifications. Similarly to setting platform type and the ID indicator (sec. 3), deck is the field that initially determines the rules to be used (with some exceptions, such as source IDs 90-93). Decks that are not specified are not subject to preconditioning. Some rules are labelled according to a lower-case letter, which indicates that more than one rule applies to a deck. Dates indicated as part of the rules are inclusive, e.g., "1912-46" refers to the beginning of 1912 through the end of 1946. {2.1 Report deletions} Source IDs 90-93: UK Met. Office Main Marine Data Bank (MDB) Flatfiles Rules: Delete all reports from SIDs 90-93 that contain deck number outside the range 201-255. Background: A number of TD-1100 decks were obtained from the US decades ago, and translated into the MDB Flatfile format, with loss of complete (e.g., supplemental) information. To simplify processing, these copies of the data are deleted. Deck 119: Japanese Ships No. 2 Rules: Delete any report from deck 119. Background: Decks 118-119 contain Japanese Kobe Collection data keyed during 1960-61. According to Uwai and Komura (1992), deck 118 contains data through part of 1953, and the deck 119 period-of-record is supposed to start with the "remaining part of 1953." Deck 119 reports prior to 1953 therefore appear to be incorrectly dated. There is known to be at least one such report, dated 1934, which will be removed by this rule. Deck 128: International Marine (US- or foreign-keyed ship data) Rules: Delete any report from deck 128. Background: Some records well outside the expected period-of-record (~1960-forward) exist that contain incorrect years. Deck 189: Netherlands Marine Rules: Delete any report from deck 189 prior to 1939, and from 1940 through August 1945. Background: Some records that apparently contain incorrect years exist outside the documented period-of-record: "1/1939-12/1939 and 9/1945-6/1955; the missing period of 1/1940-8/1945 was during the German occupation of the Netherlands during World War II" (Air Weather Service and Weather Bureau, 1957). Decks 705-707: US Merchant Marine Collection (1912-46) Rules: Delete any report from decks 705-707 outside 1910-46. Background: Although the nominal period of record is 1912-46, a few records were keyed back to 1910. Some records outside this wider period-of-record exist that contain incorrect years. Deck 731: Russian S.O. Makarov Collection Rules: Delete any report from deck 731 after 1873 containing ID "VITIAZ" and with hour missing. Background: Two input files were received for the Makarov Collection: a) a "19th Century Ships" file containing data from 27 ships, including 153 reports output for the ship VITIAZ: 19 in 1873, and 134 in 1887-89. Hour was always missing in this file. b) a VITIAZ file (2720 reports output covering 1886-89), in which hour was always extant and containing more data elements. Spot checks indicated that the reports in the former file were likely duplicated in the latter. Deck 732: Russian Marine Meteorological Data Set (MARMET) (received at NCAR) Rules: Delete any report from deck 732 prior to 1888. Background: A small number of reports exist prior to 1888 (back to 1804), likely containing incorrect years. Deck 780: Levitus World Ocean Database (WOD) Rules: Delete any report from deck 780 with sea surface temperature missing. Background: (See .) Deck 891: US National Oceanographic Data Center (NODC) Surface Data Rules: Delete all deck 891 reports. Background: The Levitus WOD provides more complete and up-to-date data than are provided by deck 891. {2.2 Field modifications} Field modifications take the form of deleting, modifying, or adding a field, including extraction in some cases of information from the supplemental attachment (Attm4) (see sec. 3 for the rules used for setting the LMR fields for platform type and ID indicator). Erroneous data values already stored in the error attachment (Attm5) of LMR may or may not be affected by preconditioning, as specified under the description of each set of preconditioning rules. In addition, deleted data fields, for example, are not written out to the error attachment. These actions are important to note for any user of the error attachment, in case of unexpected effects. All decks Rules: a) Apply "lmrfix" rules (and some related rules described here) [NOTE: For convenience, all lmrfix rules are described here under "all decks" even though some rules are actually applied only to specific decks]: i) Delete dew point temperature (DPT) from decks 900 (Australian) and 155-156 (HSST), applicable to data in the error attachment. ii) Delete wet bulb temperature (WBT) from deck 899 (South African Whaling), applicable to data in the error attachment. iii) Reset platform type (PT) according to source ID (SID) or deck [NOTE: except for some changes in wording in rule 3, these rules are reiterated unchanged from , even though some SIDs do not fall within the 1784-1949 period]: 1. Release 1 OSV data: If SID=8, 9, or 20, and if PT is not already 3 (ocean station vessel; on station): set PT=2 (ocean station vessel; off station or station proximity unknown). This rule is not applied if PT is in the error attachment. 2. Release 1 data containing original PT information: If SID=1, 5, 6, 10, 11, or 12, leave PT unchanged (including PT in error attachment). 3. Data processed after Release 1 for which the final setting of PT was made during conversion to LMR6 (see sec. 3), leave PT unchanged (including PT in error attachment). 4. For all remaining data from Release 1 (i.e., SID=2-4, 7, 13-17, 19, 21, 24; and SID=18 and SID=23 reports, including those from decks 143 or 876-882) and data processed after Release 1 (SID=22 and SID greater than 24): set PT to missing (PT in error attachment is left unchanged). b) Left-justify ID, with missing right fill. This rule is only applied if leading missing positions are not associated with data in the error attachment. Any ID characters in the error attachment are similarly shifted to ensure that, e.g., they will assume the proper position in a printout. c) For reports converted from TD-1100 (as determined by source ID; see Release 1, supp. F, Table F1-2), extract ID from Attm4 according to deck as specified in Table 1. If a deck not listed in Table 1 that was converted from TD-1100 is encountered, issue an error diagnostic and print the report including the supplemental attachment. Table 1. Location of ID information for 1854-1949 source TD-1100 decks. This table defines the field or concatenation of fields that is to be extracted from Attm4 to form ID (LMR6 fields 57-64) of the length listed, applicable only to reports converted from TD-1100 (i.e., "00" in Fmt column), as determined by SID. When the table contains a question mark for "ID length," no extraction is made but a diagnostic is printed. The "II" column indicates the resultant setting of II (LMR6 field 56), provided extant ID information results from the extraction operation, or refers to sec. 3 if setting is resolved there. Note that in some cases decks may previously have been converted from TD-1100 into TD-1127 or TD-1129, with possible loss of ID information. ------------------------------------------------------------------------------- Deck Doc* Description Fmt Supplemental or regular positions** ID length II =============================================================================== 110 T bb US Navy 00 - 0 - 116 T xk US Merchant 00 90-3 BXN, 114(baro. stn/data source)ON 5 10 118 T bb Japan 1 00 94(class)N, 95-9(ship no.)N# 6 10 128 T tt US/For.Exch 00 90-3 SON 4 9 150- 152 V bb HSST Pacific 00 - 0 - 155- 156 - -- HSST Ind/Atl Ex - - - 186 C -- USSR Ice 00 90-3 U 4 6 187 T bb Japan Whale 00 94-5(ship no.)N, 96-7(season)N 4 10 188 T bb Nor. Whaling 00 97(ship no.)NS 1 9 189 T bb Netherlands 00 110-3(jour.)N, 114-5(code)N 6 10 192 T bb German 00 94-5(book type)N, 96-101(sheet no.)NO*** 8 10 193 T bb Netherlands 00 - 0 - 194 T bb British 00 97(series)N, 98-102(log)N 6 10 195 T bb US Navy 00 97-101(ship no.)N 5 9 196 T bb German ext 00 108-10(ship no.)N 3 9 197 T bb Danish 00 97-100(deck log)ON, 123-5(sheet/source)N 7 10 281 T bb US Navy MAR 00 - 0 - 891 Z x6 NODC 00 90-5(extended "ship no."; from NODC)U 6 7 899 - -- South Af. 29M ? (79-124 contents unknown) - - 900 - -- Aust. 29 - (91-97 used for ID; see ) - - 901 U -- FOSDIC## 00 ? (SST extremes deleted fr other decks) 0 - ------------------------------------------------------------------------------- * Table 1 in describes these columns. ** Table 1 in describes this column. # See Table 1 in for background on this configuration. *** The original card deck ref. manual states that an "X" overpunch in the 3rd position of sheet no. indicated a 4 a.m. temperature in "supplementary cards." ## Table 1 in provides further comments about deck 901. ---------- d) Recover country code (C1) characters 00-40 from Attm5 by accepting an 11 overpunch over a 0-9 in either or both character positions (i.e., "}",J-R, plus "{" (12 overpunch over zero). e) Compute a missing dew point temperature if WBT and AT are extant; if SLP is missing 1015.0 is used as SLP. This rule is not applied if any of the data used for computation of DPT (i.e., SLP, AT, WBT, or T2) are in the error attachment, or if WBT is greater than AT. Constants ACON and BCON are set for computation of DPT relative to water: ACON=7.5 and BCON=237.3. The following Fortran code is then used to attempt computation of DPT [NOTE: this computation actually occurs as a final step in field modifications, since AT and WBT values may be modified by rules given below that are applied to individual decks]: ESW = 6.1078*10.**(WBT*ACON/(WBT+BCON)) E = ESW-(.00066*SLP)*(((.00115*WBT)+1)*(AT-WBT)) IF(E.LT.0.) RETURN CCON = ALOG10(E/6.1078) DPT = BCON*CCON/(ACON-CCON) where the RETURN if vapor pressure (E) is less than zero leads to an error diagnostic, and otherwise the resulting DPT is rounded to the nearest 0.1@C. To indicate that this calculation has taken place during preconditioning, T2 is set to 3, 4, 5, or 6, simply depending on whether the previous value of T2 was missing, 0, 1, or 2. f) Set the lat/lon indicator (LI) according to Release 1, Table K5-1, if the deck of the report is listed in that table. If, however, the report contains an extant LI setting that differs from the Table K5-1 setting, issue an error diagnostic but leave LI unchanged. g) For decks as specified by Release 1, Table K5-1, adjust the tenths positions of longitude and latitude from "the corner" (0,0) to the center (5,5) of the 1-degree box (the location of that corner of the 1-degree box, and the direction of the adjustments, depend on a strict interpretation of 1-degree boxes as discussed below). Tenths positions that are already at the box center (from post-dupelim volumes), or at any other location within the 1-degree box, remain at or are adjusted to the box center. h) Delete an extant temperature indicator (T1) or second temperature indicator (T2) from reports missing all (air, wet bulb, dew point, and sea surface) temperature data. This rule is not applied if any of the temperature data are in the error attachment, but otherwise it is applied even if T1 and T2 are in the error attachment. Background: a) (See .) b) (See .) c) Background is given in Table 1. d) (See .) e) (See .) f) (See .) g) (See .) h) (See .) Deck 118: Japanese Ships No. 1 Rules: If wind speed is extant, set the wind speed (WI) indicator to 5 (Beaufort force, based on documentation). The existing value of WI is expected to be 6 (estimated, units unknown/unknown method). If WI is set otherwise (including missing or in the error attachment), print a diagnostic message followed by a listing of the report, but still set WI to 5. Background: (See .) Deck 150: Pacific (US Responsibility) HSST Netherlands Receipts Deck 151: Pacific (US Responsibility) HSST German Receipts Deck 152: Pacific (US Responsibility) HSST UK Receipts Rules: If sea surface temperature (SST) is extant, set the SST method indicator (SI) to SI=10 to indicate "implied bucket (an HSST SID or any match)." Actual application of this rule is based solely upon testing for SID=2 (i.e., HSST Pacific), which comprises decks 150-152 (plus some reports from deck 192, but only for pre-1946 data). Background: (See .) Deck 155: Indian (Netherlands Responsibility) HSST Deck 156: Atlantic (German Responsibility) HSST Rules: a) If sea surface temperature (SST) is extant, set the SST method indicator (SI) to SI=10 to indicate "implied bucket (an HSST SID or any match)." Actual application of this rule is based solely upon testing for SID=3 (HSST Indian) or SID=4 (HSST Atlantic), which comprise decks 155-156. b) Bracket estimated wind speeds in the HSST Indian and Atlantic decks to the "old" Beaufort midpoints, applicable only to reports not already bracketed, i.e., the wind speed indicator (WI) must be missing and the direction indicator (DI) extant. Also if these conditions are met, WI=missing is reset to WI=6. Actual application of this rule is based solely upon testing for SID=3 (HSST Indian) or SID=4 (HSST Atlantic), which comprise decks 155-156. Background: a) (See .) b) (See .) Deck 187: Japanese Whaling Fleet Rules: If both wind speed (W) and direction (D) are missing (and not in the error attachment), and if 12-point wind direction is "12", and Beaufort force falls in the range 0-9 or is "-" or "+" (fields in the supplemental attachment corresponding to original TDF-11 positions 114- 115 and 116, respectively). If these conditions are all true, make the following settings, ignoring any existing settings of the wind direction indicator (DI) and wind speed indicator (WI): i) DI to missing. ii) D to 360 degrees. iii) WI to 5 (Beaufort force). iv) Convert Beaufort force to W in meters per second using the midpoint of the "old" Beaufort equivalence scale, where 0-9 are mapped to meters per second according to Release 1, Table K5-5, "-" (equivalent to Beaufort force 10) is mapped to 26.8, and "+" (equivalent to Beaufort force 11 or higher) to 30.9. Background: (See .) Deck 188: Norwegian Antarctic Whaling Factory Ships Rules: (As for deck 118.) Background: (See .) [NOTE: NCDC (1968) indicates that "The observations were recorded in Beaufort Force and converted by Norway before punching" using an equivalence scale containing some differences from the "old" scale (ref. Release 1, Table K5-5). The original card deck 188 reference manual states that the equivalent values "are slightly different from the 1929 International code, the differences between successive mean speeds were smoothed."] Deck 189: Netherlands Marine Rules: If wind speed is extant, set the wind speed (WI) indicator to 6 (estimated, units unknown/unknown method). The existing value of WI is expected to be 6 or 7 (measured, units unknown). If WI is set otherwise (including missing or in the error attachment), print a diagnostic message followed by a listing of the report, but still set WI to 6. Background: (See .) Deck 192: Deutsche Seewarte Marine Rules: (As for deck 118.) Background: (See .) Deck 193: Netherlands Marine Rules: a) (As for deck 118.) b) After report deletions (sec. 2.1) all remaining deck 193 reports should contain SID=1, 5, 6, or 10. If a deck 193 report is encountered with another SID, issue an error diagnostic and print the report including the supplemental attachment. Otherwise, recover any extant sea level pressure (SLP; in mm absent leading "7") from Attm4 (TD-1100 positions 101-103), add 700, convert from mm to hPa, and calculate and apply a gravity correction depending on latitude and with gmode=2 using {lmrlib:fwbpgv}. In addition, set the pressure bias flag PB=1 (questionable SLP: level 1: deck). Background: a) (See .) b) NCDC (1968) describes the configuration of SLP in deck 193 as "tens, units, and tenths of millimeters with high-order position assumed. Corrected for temperature but not for gravity. 000-999 = 700.0 - 799.9 millimeters." We are correcting all the deck 193 SLP data for gravity even though some barometers were likely aneroid rather than mercurial (aneroid barometers should not be corrected for gravity), because the reports lack any metadata regarding the type of barometer or ship. The Royal Netherlands Meteorological Institute (KNMI) continues to look into this problem. A missing link being explored by KNMI is a set of logbooks that was confiscated during World War II, which is expected to contain more metadata. "The logbooks may have been left behind in the former Russian zone of Eastern Germany and even may have been transported to Russia.... We have some reasons to believe that most of the Dutch merchant ships used mercury barometers throughout the whole period. Navy ships, however, seemed to have been using both mercury and aneroid barometers, especially after approximately 1880. The number of navy ships in the database was, on average, only 3% of all the observations. This could not be the main cause of the bias, we think. On the other hand, the percentage of naval observations before say 1870, was higher than after that year." (Frits Koek and Hendrik Wallbrink, KNMI, personal communication.) Deck 194: Great Britain Marine Rules: a) (As for deck 118.) b) Delete extant total cloud amount (N) from any deck 194 report. This rule is applied to data in the error attachment. Background: a) (See .) b) In deck 194 for "series code" 1930 (positions 116-119 in TD-1100) there are not cloud amounts appearing over 7 octas. Apparently, the following 0-9 code for this series was assumed to be in tenths and improperly converted as listed (from NCDC, 1968, sec. 4.5, scale 7): 1930 Improper Proper Code Definition conversion conversion ---- -------------- ---------- ---------- 0 No observation 0* missing 1 0 to 1 tenth 1 0 2 2 tenths 2 1 3 3 tenths 2 2 4 4 tenths 3 3 5 5 tenths 4 4 6 6 tenths 5 5 7 7 tenths 6 6 8 8 tenths 6 7 9 9 or 10 tenths 7 8 ---- -------------- ---------- ---------- * It is unclear whether 0 was converted to 0 or missing. ---------- Based on indirect evidence, the proper conversion appears to have been applied in MDB processing (checks could be made against the original cardimages to verify this). Instead of limiting application of this rule to series 1930 data, all deck 194 cloud amounts are deleted with the expectation that more complete (and presumably better quality) UK MDB reports will be selected during dupelim. Deck 215: UK MDB German Marine Rules: a) For all deck 215 reports from SIDs 91-92 (cardimage attached) reset ID with "book type" and "sheet no." (Attm4 positions 18-19 and 20-25, respectively). b) Add 0.1 hPa to all deck 215 pressures. Background: a) The 80-character cardimages available in the UK MDB are stored in Attm4 positions 13-92, and for deck 215 are believed equivalent in format to the deck 192 original cards. This action should bring the ID into alignment with deck 192 ID, in which these same fields were extracted according to Table 1. Presently, based on comparison of duplicates and examination of the attached cardimages, it appears that only book type was stored in deck 215 ID preceded by three zeros. b) Many deck 215 pressures values appear to be subject to 0.1-0.2 hPa biases, apparently due to the use of a truncated constant when data converted from an original values in mm. This is a compromise adjustment, as discussed in detail on this Webpage: http://www.cdc.noaa.gov/coads/mdb_tdf11.html Deck 701: Maury Collection Rules: If field qc6 (local hour from QC format; positions 249-250 in the supplemental attachment) is zero, change it to 24. Background: As a result of an error in the conversion program, zero was written out to supplementary attachment instead of 24. This can be corrected because zero otherwise should not appear. Deck 702: Norwegian Logbook Collection Rules: If sea level pressure (SLP) is extant, set the pressure bias flag PB=1 (questionable SLP: level 1: deck). Background: The metadata required to make instrument (index) corrections were very sporadically available and often appeared to be of questionable quality or interpretation. Thus no index corrections were applied. Deck 732: Russian Marine Meteorological Data Set (MARMET) (received at NCAR) Rules: a) Delete an extant station/weather indicator (IX) from any deck 732 report. This rule is applied to data in the error attachment. b) Delete an ID consisting of "000000." c) Delete extant sea surface temperature method indicator (SI), applicable to data in the error attachment. Background: a) (See background under deck 926.) b) (See .) c) (See .) Deck 780: Levitus World Ocean Database (WOD) Rules: Delete any wet bulb temperature (WBT) value of zero (not applicable to data in the error attachment). Background: (See .) Deck 762: Japanese Kobe Collection Data (keyed after decks 118-119) Rules: If sea level pressure (SLP) is extant, set the pressure bias flag PB=1 (questionable SLP: level 1: deck). Background: Manabe (1999) indicated: "When mercury barometers were used, the readings of the attached thermometers were also written in the logsheets. It is assumed that temperature, gravity, and scale corrections were applied before reporting, in accordance with the Manual on the Marine Meteorological Observation (Imperial Marine Observatory, 1921). {3. Rules for assignment of platform type and ID indicator} See for description of the notation used in this section and implementation details. Rules for setting PT and II, by deck, are as follows: Deck 110: US Navy Marine& PT = 0; II = missing Deck 116: US Merchant Marine& PT = 0 or 3; if ID is not missing II = 10 Deck 118: Japanese Ships No. 1& PT = 0; if ID is not missing II = 10 Deck 150: Pacific (US Responsibility) HSST Netherlands Receipts& Deck 151: Pacific (US Responsibility) HSST German Receipts& Deck 152: Pacific (US Responsibility) HSST UK Receipts& PT = missing; II = missing Deck 155: Indian (Netherlands Responsibility) HSST# Deck 156: Atlantic (German Responsibility) HSST# PT = missing; II = missing Background: (See .) Deck 186: USSR Ice Stations& PT = 9; if ID is not missing II = 6 Background: (See .) Deck 187: Japanese Whaling Fleet& PT = 0; if ID is not missing II = 10 Deck 188: Norwegian Antarctic Whaling Factory Ships& PT = 0; if ID if not missing II = 9 Deck 189: Netherlands Marine& PT = 0; if ID is not missing II = 10 Deck 192: Deutsche Seewarte Marine& PT = 0; if ID is not missing II = 10 Deck 193: Netherlands Marine& PT = 0; II = missing Deck 194: Great Britain Marine& PT = 0; if ID is not missing II = 10 Deck 195: US Navy Ships Logs& PT = 0; if ID is not missing II = 9 Deck 196: Deutsche Seewarte Marine (192 extension)& PT = 0; if ID is not missing II = 9 Deck 197: Danish Marine& PT = 0; if ID is not missing II = 10 Deck 201-255: UK Met. Office Main Marine Data Bank (MDB) (except 215)* PT and II settings vary depending on deck and other factors Deck 215 UK MDB German Marine If ID is not missing, II = 10 Background: Deck 215 is believed to be derived from the same punched card format as deck 192. Based on spot comparisons of deck 192/215 duplicates, the 2-digit "book type" included as part of the deck 192 ID (ref. Table 1; with II=10) appears in deck 215 ID elements 4-5 (preceded by three zeros), and with II=9 (national ship number). Setting II=10 is more meaningful and matches deck 192. [NOTE: Some of the original German data used to construct decks 192/215 seem to have also gotten into MDB deck 254, but the amounts of data are smaller during this time period and the situation is complicated by the inclusion of other data sources in deck 254. Final assignment of PT was made in conversion from the original input format into LMR6.] Deck 281: US Navy Monthly Aerological Record (MAR)& PT = 0; II = missing Deck 701: Maury Collection* PT = 5; if ID is not missing II = 10 Deck 702: Norwegian Logbook Collection* PT = 5; if ID is not missing II = 10 Decks 705-707: US Merchant Marine Collection (1912-46)* PT = 5; if ID is not missing II = 10 Deck 731: Russian S.O. Makarov Collection PT = 5; if ID is not missing II = 10 Deck 732: Russian Marine Meteorological Data Set (MARMET) (received at NCAR) if not OSV_ID, PT = 5; if ship_ID, II = 1 else PT = 2 and II = 1 Background: (See .) Deck 733: Russian AARI North Pole Stations* PT = 9; if ID is not missing II = 6 Deck 734: Arctic Drift Stations* PT = 9; if ID is not missing II = 10 Deck 762: Japanese Kobe Collection Data (keyed after decks 118-119) PT = 5; if ID is not missing II = 10 Background: A (nominally) 5-digit "serial number of the log sheet" was transferred from the NF interim file format (the bulk of the data). For the JMA95 interim format, apparently no ID information was transferred. Spot checks showed that the serial number occasionally contained letters in addition to digits and varied in length. Deck 780: Levitus World Ocean Database (WOD)* PT = 10, 11, or 12; if ID is not missing II = 7 Deck 899: South African Whaling PT = missing; II = missing Deck 900: Australian PT = 5; if AUS_number, II = 9 Background: (See .) Deck 901: FOSDIC Reconstructions (card images from 16mm film)& PT = 0; II = missing Background: (See .) {4. Duplicate elimination} See for general background information. a) QC processing (See .) b) Bathythermographs (See .) c) Hour and day crosses (See .) d) Allowances Ten allowances were defined for the Release 1 pre-1970s version of dupelim (see Release 1, pp. K21-K22 and K27-K29) but allowance #7 was not applicable as discussed on p. K28 of Release 1, and allowance #8 is implemented as part of preconditioning of decks 155-156 (see sec. 2.2). The remaining allowances apply to pre-1950 data in the revised dupelim program, as follows: #1 Temperatures off by less than 1 degree (any match with decks 116, 119, 555, 888, 899). [NOTE: Decks 119, 555, and 888 are post-1949 data.] #2 Present weather off by units digit (any match with the Atlas). #3 Wind ranges used to test for equality (all decks; listed in Release 1, Table K5-4), applied to both measured and estimated wind before July 1963 (exclusive). After July 1963, the ranges are applied unless the two winds being compared both have a wind speed indicator (WI) showing wind was measured; in this case the two winds are given a tolerance of 0.6 m/s (approximately 1 knot) for equality. [NOTE: The latter portion of the allowance does not apply to pre-1950 data. See for discussion of different wording compared to Release 1, p. K28, and of expanded WI values defined in LMR6 versus LMR5.] #4 Pressure to whole millibars (deck 192 vs. HSST or MDB, deck 193 vs. HSST, or HSST vs. MDB; and any location and time match with deck 555 or 899). One of two slightly different checks is used (with the second providing more latitude for pressure differences). For the following cases, pressure is first tested to whole hPa; if pressures are still not equal, then for: --deck 192 vs. 150-152 or 155-156 (HSST), 0.1 hPa is added to HSST --deck 192 vs. 215 or 254 (MDB) , 0.1 hPa is added to MDB --deck 193 vs. 150-152 or 155-156 (HSST), 0.1 hPa is added to HSST and then another test for equality is made to tenths hPa. Or, for the following cases: --deck 150-152 or 155-156 vs. 215 or 254 --location and time matches with deck 555 or 899 0.1 hPa is added to the lower value and then a test for equality is made to whole hPa. [NOTE: This allowance has been expanded in scope from the original rule to also cover deck 193, in which pressure was previously missing but is now being recovered from supplemental data, plus MDB decks of non-UK origin in which pressure differences have been identified. Deck 555 is post-1949 data.] #5 Temperatures lost digits (deck 193 vs. HSST decks other than 192). Air and sea surface temperature problems occur in some HSST reports (i.e., SID=2-4) that match deck 193. If air temperatures are equal between reports and sea temperatures are also equal no further action is needed. When one or both temperatures do not match, the values must be tested digit by digit. The temperatures are considered exact when two digits out of three are equal and the unequal digit in the HSST report is zero (signs must also be the same). Otherwise, they are unequal. For example, temperatures 20.3 and 0.3 would be equivalent using this scheme. [NOTE: The original wording of this allowance stated that "the unequal digit in the HSST report is zero or blank," but the actual implementation involved recovering the digits from an integer such that zero appeared instead of blank, e.g., 20.3 and 00.3.] #6 A 6- or 7-hour time difference (deck 193 vs. HSST Indian). A 6- or 7-hour difference occurs in almost all HSST-deck 193 matches in octant 8 of the HSST Indian basin. A special hour check is performed when the location, year, and month of the HSST report matches that of a report in deck 193. When this type of match occurs and hour is greater than the tolerance in the main dupelim plan (+ or - one hour), hour is given a new tolerance of +6 or +7 (hour of deck 193 minus hour of HSST, taking into account day crosses). This tolerance is allowed for all deck 193-HSST matches in the Indian Ocean regardless of octant. #9 Automatic retention of deck 193 (deck 193 vs. HSST Indian and Atlantic). [NOTE: The justification for this allowance on p. K29 of Release 1 is no longer applicable, because deck 193 pressures are now being recovered and gravity corrected (sec. 2.2). Also, there is evidence that at least some HSST pressures matching deck 193 may have been corrected for gravity. However, the allowance is still applied because of the possibility of other data problems or deficiencies in the HSST data.] #10 Automatic retention of deck 192 (deck 192 vs. HSST decks other than 192). Without this allowance the HSST dups (i.e., SID=2-4) would usually have been retained because they may contain total cloudiness, and wet bulb and dew point temperatures. However, the wet bulb and dew point temperatures were computed from relative humidity (available in deck 192 supplemental), and the total cloudiness was substituted into deck 192, so deck 192 is preferred. e) Exact time/space/ID matches (See .) f) Special deck rules and revised priority structure Table 2 lists special rules that are applied to some decks, and the proposed new priority numbers (according to deck, rather than SID). The special rules, acting independently from the priority codes and other selection criteria, may force a given deck to be selected or not selected, and allow some decks to "pass through" dupelim unchanged. Table 2. Duplicate elimination special deck rules and priority codes. Priority codes are used in the event of a match of two reports with equal quality codes, in which case the deck with the lowest priority code is considered preferable and is selected in that match. (If two reports with equal priority codes match, the second report in sort order is selected.) The special rules may override other duplicate selection criteria, including the quality and priority codes; see for an explanation of the special rules. All decks during this period are considered logbook (or delayed-mode) data for the purpose of setting DS and DC. ------------------------------------------------------------------------------- Rule Priority Deck Description =============================================================================== 2 110 US Navy Marine 2 116 US Merchant Marine 2 118 Japanese Ships No. 1 4 150 Pacific (US Responsibility) HSST Netherlands Receipts 4 151 Pacific (US Responsibility) HSST German Receipts 4 152 Pacific (US Responsibility) HSST UK Receipts 4 155 Indian (Netherlands Responsibility) HSST 4 156 Atlantic (German Responsibility) HSST [k] 3 186 USSR Ice Stations 2 187 Japanese Whaling Fleet 2 188 Norwegian Antarctic Whaling Factory Ships 2 189 Netherlands Marine 2 192 Deutsche Seewarte Marine 2 193 Netherlands Marine 2 194 Great Britain Marine 2 195 US Navy Ships Logs 2 196 Deutsche Seewarte Marine (192 extension) 2 197 Danish Marine 1 201-255 UK Met. Office Main Marine Data Bank (MDB) (except 215) [z] 6 215 UK MDB German Marine 5 281 US Navy Monthly Aerological Record (MAR) [a] 1 701 Maury Collection 1 702 Norwegian Logbook Collection 1 705-707 US Merchant Marine Collection (1912-46) [a] 1 731 Russian S.O. Makarov Collection [z] 6 732 Russian Marine Meteorological Data Set (MARMET) 2 733 Russian AARI NP Stations 1 734 Arctic Drift Stations 1 762 Japanese Kobe Collection Data [a] 1 780 Levitus World Ocean Database (WOD) 2 899 South African Whaling 2 900 Australian 3 901 FOSDIC Reconstructions (card images from 16mm film) ------------------------------------------------------------------------------- g) Revisions for dup status (DS) (See .) h) Revisions for dup check (DC) (See .) i) Substitutions between duplicates Rules: Three substitutions are defined on p. K27 of Release 1 (see for background on 1-2, and implementation details): 1. Longitude and latitude tenths positions and associated indicator from matching HSST into deck 194. 2. Calm wind speeds from matching HSST into deck 194 (wind speed in deck 194 must be missing). 3. Total cloudiness matched under allowance #10 was substituted from HSST into any deck 192 report with total cloudiness missing. One additional action related to substitution was defined on p. K35 of Release 1: "if one report of a pair of dups has a bucket indicator of 2 and the other report has sea surface temperature present but the bucket indicator is missing, it is replaced with a value of 2." This was handled as discussed in . {References} Air Weather Service and Weather Bureau, 1957: Reference Manual: 189 Dutch Marine 1939, 1945-1955. HQ, Air Weather Service, Data Control Div., and Weather Bureau, Climatological Services Div., July 1955, 7 pp. [Available from NOAA/National Climatic Data Center]. Manabe, T., 1999: The Digitized Kobe Collection, Phase I: Historical Surface Marine Meteorological Observations in the Archive of the Japan Meteorological Agency. Bull. Amer. Meteor. Soc., 80, 2,703-2,716. NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual. NCDC, Asheville, NC. Uwai, T. and K. Komura, 1992: The Collection of Historical Ships' Data in Kobe Marine Observatory. Proceedings of the International COADS Workshop, Boulder, Colorado, 13-15 January 1992. H.F. Diaz, K. Wolter, and S.D. Woodruff, Eds., NOAA Environmental Research Laboratories, Boulder, Colo., 47-59.