=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.1 Data Preconditioning and Duplicate Elimination: 1946-69 27 February 2004 ================================================================= Document Revision Information (previous version: 9 September 2002): Updates Release 2.1 and ICOADS. ------------------------------------------------------------------------------- {1. Introduction} This document describes the rules for a sequence of processing steps performed in the duplicate elimination (dupelim) program for 1946-69 (originally referred to as COADS Release 1b) data. Preconditioning (sec. 2), the first step in this sequence, was used to delete Long Marine Reports (LMR6), or to correct or modify individual data fields within a given report. The second step involved setting the LMR fields for platform type (PT) and ID indicator (II) (sec. 3). The final step was actual QC/dupelim processing (sec. 4). During dupelim, additional reports were eliminated, and a limited number of changes was made to the contents of reports by substitution between duplicates. [NOTE: Because of processing differences, the three original COADS updates that compose ICOADS.DM, and accompanying documentation, are referred to as follows: Release 1a: 1980-97 Release 1b: 1970-79 1946-69 Release 1c: 1784-1949 These four documents describe the "preconditioning" and duplicate elimination processing used to create LMR for the indicated periods. 1946-49 Release 1b data were replaced by Release 1c data.] {2. Preconditioning} Sec. 2.1 gives the rules for report deletions; sec. 2.2 gives the rules for field modifications. Similarly to setting platform type and the ID indicator (sec. 3), deck is the field that initially determines the rules to be used. Decks that are not specified are not subject to preconditioning. Some rules are labelled according to a lower-case letter, which indicates that more than one rule applies to a deck. Dates indicated as part of the rules are inclusive, e.g., "1966-69" refers to the beginning of 1966 through the end of 1969. {2.1 Report deletions} Deck 119: Japanese Ships No. 2 Rules: Delete any report from deck 119 after 30 June 1961. Background: (See .) Deck 732: Russian Marine Meteorological Data Set (MORMET) (received at NCAR) Rules: Delete any report from deck 732 in the region 270E through 280E and 10S through 0 (inclusive boundaries). Background: The data for this 10-degree region (near Ecuador) are known to be mislocated in the MORMET archive (these reports likely belong in the Indian Ocean, but further information is needed). Deck 780: Levitus World Ocean Atlas (WOA) Rules: Delete any report from deck 780 with sea surface temperature missing. Background: (See .) Deck 891: US National Oceanographic Data Center (NODC) Surface Data Rules: Delete all deck 891 reports, except retain station data (SD) reports as indicated by PT=10 (oceanographic station data; SD/Co22) provided they contain at least one of the following elements: wind speed, wind direction, present weather, past weather, sea level pressure, air temperature, wet bulb temperature, or total cloud amount (delete reports containing only sea surface temperature, or other elements such as dew point temperature that are not specified above). Background: (See .) {2.2 Field modifications} Field modifications take the form of deleting, modifying, or adding a field, including extraction in some cases of information from the supplemental attachment (Attm4) (see sec. 3 for the rules used for setting the LMR fields for platform type and ID indicator). Erroneous data values already stored in the error attachment (Attm5) of LMR may or may not be affected by preconditioning, as specified under the description of each set of preconditioning rules. In addition, deleted data fields, for example, are not written out to the error attachment. These actions are important to note for any user of the error attachment, in case of unexpected effects. All decks Rules: a) Apply "lmrfix" rules (and some related rules described here) [NOTE: For convenience, all lmrfix rules are described here under "all decks" even though some rules are actually applied only to specific decks]: i) Delete dew point temperature (DPT) from decks 900 (Australian) and 155-156 (HSST), applicable to data in the error attachment. ii) Delete wet bulb temperature (WBT) from deck 899 (South African Whaling), applicable to data in the error attachment. iii) Reset platform type (PT) according to source ID (SID) or deck: 1. Release 1 OSV data: If SID=8, 9, or 20, and if PT is not already 3 (ocean station vessel; on station): set PT=2 (ocean station vessel; off station or station proximity unknown). This rule is not applied if PT is in the error attachment. 2. Release 1 data containing original PT information: If SID= 1, 5, 6, 10, 11, or 12, leave PT unchanged (including PT in error attachment). 3. Data processed after Release 1 for which PT was set during conversion to LMR6: For decks 145 (PMEL Daily Equatorial Mooring), 749 (FGGE Level IIb), and 780 (Levitus Atlas), leave PT unchanged (including PT in error attachment). [NOTE: All but deck 780 are post-1969 data.] 4. For all remaining data from Release 1 (i.e., SID=2-4, 7, 13-17, 19, 21, 24; and SID=18 and SID=23 reports, including those from decks 143 or 876-882) and data processed after Release 1 (SID=22 and SID greater than 24): set PT to missing (PT in error attachment is left unchanged). b) Left-justify ID, with missing right fill. This rule is only applied if leading missing positions are not associated with data in the error attachment. Any ID characters in the error attachment are similarly shifted to ensure that, e.g., they will assume the proper position in a printout. c) For reports converted from TD-1100 (as determined by source ID; see Release 1, supp. F, Table F1-2), extract ID from Attm4 according to deck as specified in Table 1. If a deck not listed in Table 1 that was converted from TD-1100 is encountered, issue an error diagnostic and print the report including the supplemental attachment. Table 1. Location of ID information for 1946-69 source TD-1100 decks. This table defines the field or concatenation of fields that is to be extracted from Attm4 to form ID (LMR6 fields 57-64) of the length listed, applicable only to reports converted from TD-1100 (i.e., "00" in Fmt column), as determined by SID. When the table contains a question mark for "ID length," no extraction is made but a diagnostic is printed. The "II" column indicates the resultant setting of II (LMR6 field 56), provided extant ID information results from the extraction operation, or refers to sec. 3 if setting is resolved there. Note that in some cases decks may previously have been converted from TD-1100 into TD-1127 or TD-1129, with possible loss of ID information. ------------------------------------------------------------------------------- Deck Doc* Description Fmt Supplemental or regular positions** ID length II =============================================================================== 110 T bb US Navy 00 - 0 - 116 T xk US Merchant 00 90-3 BXN, 114(baro. stn/data source)ON 5 10 117 C -- US Navy hly 00 90-3 U 4 9 118 T bb Japan 1 00 94(class)N, 95-9(ship no.)N# 6 10 119 T bb Japan 2 00 94-8(year+ship no.)N, 100(class)N 6 10 128 T tt US/For.Exch 00 90-3 SON 4 9 150- 152 V bb HSST Pacific 00 - 0 - 155- 156 - -- HSST Ind/Atl Ex - - - 184 T bo British 00 97(country)ON,101-2(series)N,103-7(log)U 8 10 185 T bb USSR IGY 00 - 0 - 186 C -- USSR Ice 00 90-3 U 4 6 187 T bb Japan Whale 00 94-5(ship no.)N, 96-7(season)N 4 10 189 T bb Neth. 00 110-3(jour.)N, 114-5(code)N 6 10 194 T bb British 00 97(series)N, 98-102(log)N 6 10 195 T bb US Navy 00 97-101(ship no.)N 5 9 196 T bb German ext 00 108-10(ship no.)N 3 9 197 T bb Danish 00 97-100(deck log)ON, 123-5(sheet/source)N 7 10 555 U -- Monterey 00 90-3 U 4 (sec.3) 891 Z x6 NODC 00 90-5(extended "ship no."; from NODC)U 6 7 897 - -- Eltanin 29M - (deck is from one ship) - - 898 - -- Japan 29 - (91-97 used for ID; see ) - - 899 - -- South Af. 29M ? (79-124 contents unknown) - - 900 - -- Aust. 29 - (91-97 used for ID; see ) - - 901 U -- FOSDIC## 00 ? (SST extremes deleted fr other decks) 0 - 902 C -- British## 00 98(country)U,102-3(series)U,104-8(log)U 8 10 926 - -- IMM 29 - (91-97 used for ID; see ) - - 999 U -- ETAC## 00 ? 0 - ------------------------------------------------------------------------------- * Three non-blank sub-columns appear under this column: An upper-case letter in first sub-column indicates the source of NCDC documentation for the TD-1100 format, if any: - = not applicable (not a TD-1100 deck) C = undocumented: not included in NCDC (1968), but original card deck ref. manual exists U = undocumented: not included in NCDC (1968), and no other documentation is known to exist T = NCDC (1968) V = NCDC (1979) Z = NCDC (1983) A lower-case letter in the second sub-column indicates whether "ship number" (SN; TD-1100 positions 90-93) was documented to be blank or non-blank: - = not applicable (not a TD-1100 deck, or undocumented) b = SN documented as blank t = SN documented as full configuration of TD-1100 codes (non-blank) x = SN documented as non-blank A lower-case letter or digit in the third sub-column indicates whether "Ocean Station Vessel or ship indicator" (OSVSI; TD-1100 position 81) was documented to be blank or non-blank: - = not applicable (not a TD-1100 deck, or undocumented) b = OSVSI documented as blank k = OSVSI documented as 2 with "-" overpunch (i.e., K: "OSV on station") or blank ("all other ships") t = OSVSI documented as full configuration of TD-1100 codes (blank or non-blank) 6 = OSVSI documented as 6 o = OSVSI documentation omitted from NCDC (1968) (See for additional background on the OSVSI field, and why it was intended to be blank for many early TD-1100 decks.) ** Individual fields documented in NCDC (1968) to be constrained to certain characters, as indicated by the following codes: B = blank (i.e., with meaning other than "missing") N = numeric O = sign ("+" or "-") overpunch of a numeric*** S = sign only; "-" is the only documented possibility U = undocumented in TD-1100 X = "X" overpunch alone (deck 116) [NOTE: "X" followed by three blanks appears as an alternative configuration to indicate foreign data in deck 116 according to R. Quayle's hand-annotated copy of NCDC (1968).] *** See Table 1 in for a description of overpunches. # The positions listed for deck 118 in Table 1, which were obtained from hand- written notes in R. Quayle's copy of NCDC (1968), appear consistent with the actual format based on limited data examination. In NCDC (1968), the positions of these and other supplemental fields for deck 118 appear to be incorrectly documented; following NCDC (1968) the table entry would have been "100(class)N, 101-105(no.)N." ## Since deck 902 is undocumented in TD-1100, the positions listed in Table 1 were derived from manual examination of its supplemental data and comparison with deck 184, of which deck 902 is an extension. Note that the supplemental fields in deck 902 appear to have been shifted one position to the right in comparison to deck 184. Decks 901 and 999, also undocumented in TD-1100, were also examined but did not appear to contain any recognizable ID information in their supplemental data. ---------- d) Recover country code (C1) characters 00-40 from Attm5 by accepting an 11 overpunch over a 0-9 in either or both character positions (i.e., "}",J-R, plus "{" (12 overpunch over zero). e) Compute a missing dew point temperature if WBT and AT are extant; if SLP is missing 1015.0 is used as SLP. This rule is not applied if any of the data used for computation of DPT (i.e., SLP, AT, WBT, or T2) are in the error attachment, or if WBT is greater than AT. Constants ACON and BCON are set for computation of DPT relative to water: ACON=7.5 and BCON=237.3. The following Fortran code is then used to attempt computation of DPT [NOTE: this computation actually occurs as a final step in field modifications, since AT and WBT values may be modified by rules given below that are applied to individual decks]: ESW = 6.1078*10.**(WBT*ACON/(WBT+BCON)) E = ESW-(.00066*SLP)*(((.00115*WBT)+1)*(AT-WBT)) IF(E.LT.0.) RETURN CCON = ALOG10(E/6.1078) DPT = BCON*CCON/(ACON-CCON) where the RETURN if vapor pressure (E) is less than zero leads to an error diagnostic, and otherwise the resulting DPT is rounded to the nearest 0.1@C. To indicate that this calculation has taken place during preconditioning, T2 is set to 3, 4, 5, or 6, simply depending on whether the previous value of T2 was missing, 0, 1, or 2. f) Set the lat/lon indicator (LI) according to Release 1, Table K5-1, if the deck of the report is listed in that table. If, however, the report contains an extant LI setting that differs from the Table K5-1 setting, issue an error diagnostic but leave LI unchanged. g) For decks as specified by Release 1, Table K5-1, adjust the tenths positions of longitude and latitude from "the corner" (0,0) to the center (5,5) of the 1-degree box (the location of that corner of the 1-degree box, and the direction of the adjustments, depend on a strict interpretation of 1-degree boxes as discussed below). Tenths positions that are already at the box center (from post-dupelim volumes), or at any other location within the 1-degree box, remain at or are adjusted to the box center. h) Delete an extant temperature indicator (T1) or second temperature indicator (T2) from reports missing all (air, wet bulb, dew point, and sea surface) temperature data. This rule is not applied if any of the temperature data are in the error attachment, but otherwise it is applied even if T1 and T2 are in the error attachment. Background: a) (See .) b) (See .) c) Background is given in Table 1. d) (See .) e) (See .) f) (See .) g) Pre-1970s decks (110, 117, 192-196, and 281) that had latitude and longitude available only to whole degrees (e.g., original cards for decks 192-194 and 196 all contained 10-degree and 1-degree Marsden Square numbers), are adjusted by setting the tenths positions to the center from "the corner" of the 1-degree box (1-degree box is synonymous with 1-degree Marsden Square; see Release 1, supp. G). Basically, the two inclusive boundaries of each 1-degree box, which thus define the corner referred to, are those nearest the intersection of the Equator and Prime Meridian (0/0 latitude/longitude). As a result, the direction of the adjustments is quadrant dependent, with exceptions at "location discontinuities" along the Equator, 0 and 180 longitude, and exactly on the Poles (where 1-degree box is undefined). See Release 1, supp. G for more information about 1-degree boxes. The footnote to Table K5-1 in supp. K of Release 1 provides additional background on the adjustments. [NOTE: Decks 192-193 and 281 are pre-1946 data. The sort for dupelim (10-degree box, year, month, 1-degree box, day, hour and deck) is not disordered by changes within a 1-degree box.] h) (See .) All decks Rules: Starting 1 January 1968, delete wave direction (WD) from all reports. This rule is not applied to data in the error attachment, and the following comparison is made with wind direction (D) as a part of this check: WD values 1-36 must equal D/10, or WD=0 must correspond to D=361, or WD=38 must correspond to D=362. If WD does not match D exactly as stated, print a diagnostic message followed by a listing of the report, but still delete wave direction. Background: (See .) Deck 118: Japanese Ships No. 1 Rules: If wind speed is extant, set the wind speed (WI) indicator to 5 (Beaufort force, based on documentation). The existing value of WI is expected to be 6 (estimated, units unknown/unknown method). If WI is set otherwise (including missing or in the error attachment), print a diagnostic message followed by a listing of the report, but still set WI to 5. Background: Documentation of the original card format used for deck 118 plus a few other decks used in NCDC (1968) shows that winds were uniformly stored in the card format in Beaufort force. Deck 150: Pacific (US Responsibility) HSST Netherlands Receipts Deck 151: Pacific (US Responsibility) HSST German Receipts Deck 152: Pacific (US Responsibility) HSST UK Receipts Rules: If sea surface temperature (SST) is extant, set the SST method indicator (SI) to SI=10 to indicate "implied bucket (an HSST SID or any match)." Actual application of this rule is based solely upon testing for SID=2 (i.e., HSST Pacific), which comprises decks 150-152 (plus some reports from deck 192, but only for pre-1946 data). Background: During Release 1 dupelim processing, the "bucket indicator" (BI; LMR5 field 32) was set to BI=2 (implied bucket) according to SID, so that matching reports from other decks could be set to BI=2, provided BI in the matching report was missing and SST extant (see Release 1, p. K35). This rule implements the initial setting of the equivalent value SI=10, for use in later dupelim processing (sec. 4). [NOTE: During conversion from LMR5 to LMR6, any extant BI values were mapped to SI, including BI=2 to SI=10 (BI=2 could exist only in post-dupelim volumes; see ). Depending on SID, this rule harmlessly resets some existing SI=10 values.] Deck 155: Indian (Netherlands Responsibility) HSST Deck 156: Atlantic (German Responsibility) HSST Rules: a) If sea surface temperature (SST) is extant, set the SST method indicator (SI) to SI=10 to indicate "implied bucket (an HSST SID or any match)." Actual application of this rule is based solely upon testing for SID=3 (HSST Indian) or SID=4 (HSST Atlantic), which comprise decks 155-156. b) Bracket estimated wind speeds in the HSST Indian and Atlantic decks to the "old" Beaufort midpoints, applicable only to reports not already bracketed, i.e., the wind speed indicator (WI) must be missing and the direction indicator (DI) extant. Also if these conditions are met, WI=missing is reset to WI=6. Actual application of this rule is based solely upon testing for SID=3 (HSST Indian) or SID=4 (HSST Atlantic), which comprise decks 155-156. Background: a) (See background under decks 150-152.) b) The Exchange format for the Historical SST Data Project was defined to use the "new" or "scientific" Beaufort wind speed equivalence scale (WMO, 1970) for estimated winds. Release 1, pp. K28-K29 describes the the method used to bracket HSST estimated winds that were converted directly from the Exchange format, to midpoints of the "old" Beaufort equivalence scale (WMO code 1100), thus implementing dupelim allowance #8. [NOTE: As explained in the footnote on p. K28 of Release 1, WI was conservatively set to missing during conversion from the Exchange format into LMR5, which inadvertently caused bracketing not to occur during dupelim, and have to be performed instead at a later stage of Release 1 processing. Wind speeds without the required criteria for bracketing (i.e., WI missing and DI extant) are either: i) from post-dupelim data, and thus already bracketed; or ii) measured or with missing indicator information, and thus never subject to bracketing (see for additional information).] Deck 184: Great Britain Marine (194 extension) Rules: (As for deck 118.) Background: (See background under deck 118.) Deck 187: Japanese Whaling Fleet Rules: If both wind speed (W) and direction (D) are missing (and not in the error attachment), and if 12-point wind direction is "12", and Beaufort force falls in the range 0-9 or is "-" or "+" (fields in the supplemental attachment corresponding to original TDF-11 positions 114- 115 and 116, respectively). If these conditions are all true, make the following settings, ignoring any existing settings of the wind direction indicator (DI) and wind speed indicator (WI): i) DI to missing. ii) D to 360 degrees. iii) WI to 5 (Beaufort force). iv) Convert Beaufort force to W in meters per second using the midpoint of the "old" Beaufort equivalence scale, where 0-9 are mapped to meters per second according to Release 1, Table K5-5, "-" (equivalent to Beaufort force 10) is mapped to 26.8, and "+" (equivalent to Beaufort force 11 or higher) to 30.9. Background: In conversion from the original card format into TDF-11, errors appear to have been made in failing to convert any direction code 36 (north in the 36-point scale) and its associated wind speed. However, a 12-point wind direction code and Beaufort force usually seem to be available in the supplemental attachment to provide a correction. [NOTE: This correction is approximate because D and W in TDF-11 were derived from separate fields, when extant, that appeared only in the original card format (36-point direction and speed in knots); the relationship between the two different sets of D and W fields in the original card format is unclear. DI is set to missing because we currently lack a DI value for indication of the 12-point scale. In addition, the 12-point wind direction code of "12" corresponds to a range of 335-4 degrees, whose midpoint is not 360 degrees. Moreover, Beaufort value "+" actually signifies force 11 or greater (up to 17), according to the original card deck reference manual; usage of the midpoint for force 11 is thus definitely an approximation. Assuming that the original card deck reference manual is correct, NCDC (1968) incorrectly documents the Beaufort value "+" as signifying force 11.] Deck 189: Netherlands Marine Rules: If wind speed is extant, set the wind speed (WI) indicator to 6 (estimated, units unknown/unknown method). The existing value of WI is expected to be 6 or 7 (measured, units unknown). If WI is set otherwise (including missing or in the error attachment), print a diagnostic message followed by a listing of the report, but still set WI to 6. Background: The original cards used to construct deck 189 lacked any wind speed indicator information. WI=7 was set by NCDC to indicate reports with wave height corrected. Originally this problem was thought to be confined to deck 189 reports from Old TDF-11 Supplement B (SID=5), but tests have also shown evidence of the problem in Atlas (SID=1) data. [NOTE: Further work is needed to resolve the status of any remaining wave height problems in deck 189, and whether WI=7 values might supply useful information that should be retained elsewhere in the data.] Deck 194: Great Britain Marine Rules: (As for deck 118.) Background: (See background under deck 118.) Deck 555: US Navy Fleet Num. Met. and Oceano. Center (FNMOC; Monterey) Telecom. Rules: a) Delete wet bulb temperature from any deck 555 report, applicable to data in the error attachment. b) Subtract (add) 0.5 degree from (to) any deck 555 positive (negative) air temperature. Prior to making this modification check that the tenths position of AT is "5"; if not, print a diagnostic message followed by a listing of the report, but still implement the adjustment. Background: a) (See .) b) This adjustment reverses an undocumented correction that NCDC applied to deck 555 air temperatures. NCDC applied the correction to compensate for suspected truncation problems that occurred in storing AT to whole degrees in the original Monterey format. However, tests have shown that few ATs during 1966-69 were truncated, including many originally reported in whole degrees. Thus the net effect of the NCDC correction was to produce positive biases in deck 555 AT (averaging approximately +0.4 deg) when compared to other decks. The adjustment is expected to yield much better AT agreement between deck 555 and other decks during 1966-69, whereas during 1970-73 retention of the original correction (i.e., no adjustment) appears to be more appropriate. Deck 732: Russian Marine Meteorological Data Set (MORMET) (received at NCAR) Rules: a) Delete an extant station/weather indicator (IX) from any deck 732 report. This rule is applied to data in the error attachment. b) Delete an ID consisting of "000000." c) Delete extant sea surface temperature method indicator (SI), applicable to data in the error attachment. Background: a) (See background under deck 926.) b) (See .) c) (See .) Deck 891: US National Oceanographic Data Center (NODC) Surface Data Rules: Delete sea surface temperature from all deck 891 station data (SD) reports, applicable to data in the error attachment. Background: (See .) Deck 926: International Maritime Meteorological (IMM) Data Rules: a) Delete an extant station/weather indicator (IX) from any deck 926 report with country code indicating that the country that recruited the ship was France (C1=4). For deck 926 data from other countries, delete IX if IRD is missing, or if IRD indicates that the data were received prior to March 1985 (i.e., tape I3ZG62, which was received in December 1984). This rule is applied to data in the error attachment. b) Change any ID of "IMMT" to entirely missing. This rule is not applied to data in the error attachment. Background: a) (See .) b) (See .) Deck 999: US Air Force Environ. Tactical Applications Center (ETAC) Rules: Delete wet bulb temperature from any deck 999 report, applicable to data in the error attachment. Background: We believe that wet bulb temperature was computed in deck 999, since only dew point temperature could be reported in early telecommunicated data (for additional information, see background given in for this same rule as applied to deck 555). {3. Rules for assignment of platform type and ID indicator} This section describes the rules used to set LMR fields platform type (PT) and ID indicator (II). PT indicates whether the reporting platform is a ship or some other type of platform; II indicates whether a call sign or some other sort of ID is contained in the ID array. As background, PT and II have the following defined values: Platform type (PT) 0 = US Navy or "deck" log, or unknown 1 = merchant ship or foreign military 2 = ocean station vessel--off station or station proximity unknown 3 = ocean station vessel--on station 4 = lightship 5 = ship 6 = moored buoy 7 = drifting buoy 8 = ice buoy 9 = ice station (manned, including ships overwintering in ice) 10 = oceanographic station data (bottle and STD/CTD data) 11 = mechanical bathythermograph (MBT) 12 = expendable bathythermograph (XBT) 13 = Coastal-Marine Automated Network (C-MAN) (NDBC operated) 14 = other coastal/island station 15 = fixed ocean platform (plat, rig) ID indicator (II) 0 = ID present, but unknown type 1 = ship, OSV, or ice station call sign 2 = generic ID (e.g., SHIP, BUOY, RIGG, PLAT) 3 = WMO 5-digit buoy number (possibly followed by A-L for NCEP data) 4 = other buoy number (e.g., Argos or national buoy number) 5 = C-MAN ID 6 = station name or number 7 = NODC platform/cruise 8 = IATTC pseudo ID 9 = national ship number 10 = ship name or composite information from early ship data Part a) gives the rules for setting PT and II. Part b) describes a set of general checks on the form and contents of ID, referenced in part a). In all cases deck is the field that initially determines the group of rules to be used; for a given deck, specific rules may depend further on the contents of the ID/call sign array (ID; fields 57-64, i.e., 8 characters). At this time, neither source ID (SID) nor the IMMT flag for observation platform (OP) are utilized in setting PT and II. a) Rules for setting PT and II: by deck For brevity, the following rules are expressed using pseudo-code similar to Fortran. Generally each line of pseudo-code assigns PT and then II; when two statements appear on a single line separated by a semicolon (;) and the first such statement is a conditional statement that is not true, execution proceeds on the next line. Checks are made in some cases for "generic" IDs; and in other cases for a specific form of ID, i.e., when a construct appears that contains the underline character (e.g., ship_ID). See part b) for details of the generic and general ID checks. Preexisting PT values were either retained or deleted as part of "lmrfix" processing (sec. 2.2). If preexisting PT was retained, PT was generally not subject to change at this stage. Specifically, for decks indicated by symbols: * = Final assignment of PT and II was made in conversion from the original input format into LMR6. # = Derived "ship type" (Release 1, supp. I, sec. 2.11) was transformed into PT during conversion from LMR5 to LMR6 (see ), possibly subject to "lmrfix" modifications described in sec. 2.2 of this document. & = Existing "ship type" (Release 1, supp. I, sec. 2.10) was transformed into PT during conversion from LMR5 to LMR6 (see ), possibly subject to "lmrfix" modifications described in sec. 2.2 of this document; assignment of II according to Table 1 (for the 1946-69 period, we believe all these decks were converted from TD-1100 into LMR5 during Release 1 processing). Deck 186 is an exception in that any pre-existing PT values are ignored, and deck 555 in that both PT and II are reset depending on the contents of ID. [NOTE: Any missing or extant PT values not listed for a given deck will also be retained; such values would appear if: i) PT values originated from OSVSI values undocumented for that deck according to TD-1100 documentation (see Table 1); ii) or, assignment of existing "ship type" was based on SID instead of OSVSI (see Release 1, supp. I, sec. 2.10).] Otherwise, assignment of PT and II is determined at this stage of QC/dupelim. Where assignment is determined here, it is important to note that initially PT is set to missing and if ID is missing, II is set to missing; otherwise II is set to 0. Also a final check for generic ID (II = 2) overrides a previously set II, including the initial setting of 0. Deck 110: US Navy Marine& PT = 0; II = missing Deck 116: US Merchant Marine& PT = 0 or 3; if ID is not missing II = 10 Deck 117: US Navy Hourlies& PT = 0; if ID is not missing II = 9 Background: The TD-1100 format of this deck was undocumented (see Table 1). It is assumed PT = 0 would be set due to a blank (default) in the OSVSI field of TD-1100, but missing or other PT values are retained if present. II is set according to Table 1 if ID information is successfully extracted from TD-1100 fields. [NOTE: II = 9 is assumed to be appropriate based only on the existence of a ship number in the original cards according to the card deck 117 reference manual. This assumption should be checked, if extant ID fields result.] Deck 118: Japanese Ships No. 1& PT = 0; if ID is not missing II = 10 Deck 119: Japanese Ships No. 2& PT = 0; if ID is not missing II = 10 Deck 128: International Marine (US- or foreign-keyed ship data)& PT = 0, 1, 2, 3, or 4; if ID is not missing II = 9 Deck 150: Pacific (US Responsibility) HSST Netherlands Receipts& Deck 151: Pacific (US Responsibility) HSST German Receipts& Deck 152: Pacific (US Responsibility) HSST UK Receipts& PT = missing; II = missing Deck 155: Indian (Netherlands Responsibility) HSST# Deck 156: Atlantic (German Responsibility) HSST# PT = missing; II = missing Background: The HSST Exchange format did not provide fields for ID or PT information (although all data are assumed to be from ships and possibly OSVs). Deck 184: Great Britain Marine (194 extension)& PT = 0; if ID is not missing II = 10 Background: Documentation for the TD-1100 OSVSI field for this deck was omitted (see Table 1). It is assumed PT = 0 would be set due to a blank (default) in the OSVSI field of TD-1100, but missing or other PT values are retained if present. Deck 185: USSR Marine IGY& PT = 0; II = missing Deck 186: USSR Ice Stations& PT = 9; if ID is not missing II = 6 Background: The TD-1100 format of this deck was undocumented (see Table 1). PT = 9 is set for this deck regardless of any pre-existing PT value. II is set according to Table 1 if ID information is successfully extracted from TD-1100 fields. [NOTE: II = 6 is assumed to be appropriate based only on the existence of a North Pole station number in the original cards according to the card deck 186 reference manual. This assumption should be checked, if extant ID fields result.] Deck 187: Japanese Whaling Fleet& PT = 0; if ID is not missing II = 10 Deck 189: Netherlands Marine& PT = 0; if ID is not missing II = 10 Deck 194: Great Britain Marine& PT = 0; if ID is not missing II = 10 Deck 195: US Navy Ships Logs& PT = 0; if ID is not missing II = 9 Deck 196: Deutsche Seewarte Marine (192 extension)& PT = 0; if ID is not missing II = 9 Deck 197: Danish Marine& PT = 0; if ID is not missing II = 10 Deck 555: US Navy Fleet Num. Met. and Oceano. Cntr. (FNMOC; Monterey) Telecom.& if EB_number or "BUOY", PT = 6; if EB_number, II = 4 else if "RIGG" or "PLAT", PT = 15 else if OSV_ID, PT = 2 and II = 1 else PT = 5; if ship_ID, II = 1 Background: These rules are the same as given in , but we expect only ship and OSV data in deck 555 during 1966-69. [NOTE: We believe that all deck 555 data for 1970-73 originated from SID=18 (70s Decade; TD-1129 format), whereas 1966-69 data originated from SID=7 (TD-1100 format). Release 1 rules for handling existing "ship type" yielded missing PT for SID=7.] Deck 732: Russian Marine Meteorological Data Set (MORMET) (received at NCAR) if not OSV_ID, PT = 5; if ship_ID, II = 1 else PT = 2 and II = 1 Background: The default setting of PT = 5 is overridden only if an OSV call sign is found. All OSV data are set to PT = 2 indicating "off station or station proximity unknown" since tests for location relative to the assigned OSV location at any given time were not performed. [NOTE: Few ID fields other than "000000," deleted as part of field modifications (sec. 2.2), exist in deck 732 prior to 1982.] Deck 733: Russian AARI North Pole Stations (from Polar Science Center) PT = 9; if ID is not missing II = 6 Deck 780: Levitus World Ocean Atlas (WOA)* PT = 10, 11, or 12; if ID is not missing II = 7 Deck 891: US National Oceanographic Data Center (NODC) Surface Data& PT = 10; if ID is not missing II = 7 Background: This rule assumes all data were originally source TD-1100. If this is not the case and an ID is present, II is set according to the standard initialization rules (above). [NOTE: Report deletions (sec. 2.1) should eliminate all reports with PT=11-12.] Deck 897: "Eltanin"# PT = missing; II = missing Deck 898: Japanese if not OSV_ID, PT = 5; if ship_ID, II = 1 else PT = 2 and II = 1 (see background under deck 732) Deck 899: South African Whaling PT = missing; II = missing Deck 900: Australian PT = 5; if AUS_number, II = 9 Background: The Australians provided a 3-digit ship number, which is related to ship names in the original Australian documentation. Deck 901: FOSDIC Reconstructions (card images from 16mm film)& PT = 0; II = missing Background: The TD-1100 format of this deck was undocumented (see Table 1). It is assumed PT = 0 would be set due to a blank (default) in the OSVSI field of TD-1100, but missing or other PT values are retained if present. II is set to missing because no extraction of ID information is made according to Table 1. Deck 902: Great Britain Marine (184 extension)& PT = 0; if ID is not missing II = 10 Background: The TD-1100 format of this deck was undocumented (see Table 1). It is assumed PT = 0 would be set due to a blank (default) in the OSVSI field of TD-1100, but missing or other PT values are retained if present. ID extraction, similar to deck 184, is made according to Table 1. Deck 926: International Maritime Meteorological (IMM) Data if not OSV_ID, PT = 5; if ship_ID, II = 1 else PT = 2 and II = 1 (see background under deck 732) Deck 999: US Air Force Environ. Tactical Applications Center (ETAC)& PT = 0; II = missing (see background under deck 901) b) Checks on ID During earlier field preconditioning (sec. 2.2), the contents of the ID field were left justified with missing fill (note that the blank character was translated into missing during conversion into LMR). It is important to note, however, that extraction of ID information from TD-1100 supplemental fields according to Table 1 occurs after the left-justification operation. This is deliberate, so as to preserve any meaningful blanks (e.g., a leading blank may occur in deck 116 ship numbers, with a different meaning than a leading zero). Embedded and trailing missing ID elements may also occur when TD-1100 fields were extracted according to Table 1, and no checks were made to ensure that extracted ID fields conformed with TD-1100 documentation or, in the case of concatenated TD-1100 fields, that more than one of the fields concatenated was extant. Please see for a full definition of the "generic" IDs and of the general tests on ID that are referred to in part a). {4. Duplicate elimination} The pre-1970s version of the COADS Release 1 duplicate elimination (dupelim) program considered reports within the same 1-degree box and within plus or minus one hour ("hour cross") or day ("day cross") as possible dups, and performed a check for seven weather elements (wind speed, visibility, present weather, past weather, sea level pressure, air temperature, and sea surface temperature) to determine whether reports were actually dups. These checks for weather elements included a number of "allowances," which considered weather elements (or hour) to match under some circumstances even though they were not equal. Furthermore, dup status (DS) was set to indicate dup certainty depending on how many elements matched, whether at least one report was from the Global Telecommunication System (GTS), and whether there was an hour or day cross (see Release 1, Table K5-3). Similarly, dup check (DC) was set to indicate the presence of matches between GTS and logbook reports. Quality code, as computed by the NCDC-defined quality control (QC) procedure (Release 1, supp. J), was the basis for the selection of one duplicate report over another, or, if quality codes were identical, a priority list by source ID (SID) was used, or, if priorities were also identical, the second report in sort order was selected. See for background on why certain changes were made for Release 1a. Changes in dupelim similar to those used for Release 1a are used for Release 1b, as follows: a) QC processing (See .) b) Bathythermographs (See .) c) Hour and day crosses Retain the check for duplicates across hours and days, but ensure that hourly data subject to "chain reaction" problems are not lost in the main LMR file by retention of all hour-crosses as uncertain duplicates (see for background information). d) Allowances Ten allowances were defined for the Release 1 pre-1970s version of dupelim (see Release 1, pp. K21-K22 and K27-K29), but allowances #5, #6, #9, and #10 apply only to pre-1946 data; allowance #7 was not applicable as discussed on p. K28 of Release 1; and allowance #8 is implemented as part of preconditioning of decks 155-156 (see sec. 2.2). The remaining allowances apply to 1946-69 data in the revised dupelim program, as follows: #1 Temperatures off by less than 1 degree (any match with decks 116, 119, 555, 888, 899). [NOTE: Deck 888 is post-1969 data.] #2 Present weather off by units digit (any match with the Atlas). #3 Wind ranges used to test for equality (all decks; listed in Release 1, Table K5-4), applied to both measured and estimated wind before July 1963 (exclusive). After July 1963, the ranges are applied unless the two winds being compared both have a wind speed indicator (WI) showing wind was measured; in this case the two winds are given a tolerance of 0.6 m/s (approximately 1 knot) for equality. [NOTE: See for discussion of different wording compared to Release 1, p. K28, and of expanded WI values defined in LMR6 versus LMR5.] #4 Pressure to whole millibars (deck 192 vs. HSST and any location and time match with decks 555 or 899). [NOTE: Deck 192 is pre-1946 data.] e) Exact time/space/ID matches (See .) f) Special deck rules and revised priority structure Table 2 lists special rules that are applied to some decks, and the proposed new priority numbers (according to deck, rather than SID). The special rules, acting independently from the priority codes and other selection criteria, may force a given deck to be selected or not selected, and allow some decks to "pass through" dupelim unchanged. Table 2. Duplicate elimination special deck rules and priority codes. Priority codes are used in the event of a match of two reports with equal quality codes, in which case the deck with the lowest priority code is considered preferable and is selected in that match. (If two reports with equal priority codes match, the second report in sort order is selected.) The special rules may override other duplicate selection criteria, including the quality and priority codes; see for an explanation of the special rules. ------------------------------------------------------------------------------- Rule Priority Deck Description =============================================================================== 2 110 US Navy Marine 2 116 US Merchant Marine 3 117 US Navy Hourlies 2 118 Japanese Ships No. 1 2 119 Japanese Ships No. 2 2 128* International Marine (US- or foreign-keyed ship data) 4 150 Pacific (US Responsibility) HSST Netherlands Receipts 4 151 Pacific (US Responsibility) HSST German Receipts 4 152 Pacific (US Responsibility) HSST UK Receipts 4 155 Indian (Netherlands Responsibility) HSST 4 156 Atlantic (German Responsibility) HSST 2 184 Great Britain Marine (194 extension) 2 185 USSR Marine IGY 3 186 USSR Ice Stations 2 187 Japanese Whaling Fleet 2 189 Netherlands Marine 2 194 Great Britain Marine 2 195 US Navy Ships Logs 2 196 Deutsche Seewarte Marine (192 extension) 2 197 Danish Marine [z] 5 555** US Navy FNMOC (Monterey) Telecom. [z] 6 732 Russian Marine Meteorological Data Set (MORMET) 2 733 Russian AARI NP Stations (from Polar Science Center) [a] 1 780 Levitus World Ocean Atlas (WOA) 1 891 US National Ocean. Data Center (NODC) Surface Data 2 897 "Eltanin" 2 898 Japanese 2 899 South African Whaling 2 900 Australian 3 901 FOSDIC Reconstructions (card images from 16mm film) 2 902 Great Britain Marine (184 extension) 3 926* International Maritime Meteorological (IMM) Data [k] 3*** 926**** International Maritime Meteorological (IMM) Data/SID=58 [z] 6 999** US Air Force Environ. Tactical Appl. Center (ETAC) ------------------------------------------------------------------------------- * A deck that has been classified as ship logbook data for the purpose of setting DS and DC (deck 926 should be entirely logbook data during the 1946-69 time period). All the other non-GTS decks are classified as "delayed mode" data for setting DC, and thus treated equivalently to logbook data. These additional decks include manned ice floe station data (decks 186 and 733), oceanographic data (decks 780 and 891), and logbook data gathered prior to or outside of the routine international exchange since 1963 under WMO Resolution 35. Deck 732 is a mixture of GTS and logbook data but is classified as delayed mode data for setting DS and DC. ** A deck that has been classified as from the Global Telecommunication System (GTS) for the purpose of setting DS and DC. *** Priority code is needed for [k] deck only in order to resolve "best" match selection for the purpose of constructing "common/different" summary tables output from dupelim. Priority codes are chosen so that the same priority code is applied to both [k] and non-[k] reports within the same deck. **** Data from the French correction tape whose longitudes have been mislocated due to decoding via the "add 1000" method (source ID=58); see . ---------- g) Revisions for dup status (DS) (See .) h) Revisions for dup check (DC) (See .) i) Substitutions between duplicates Rules: i) ID, II, and PT: For ship, OSV, or North Pole ice island data the rules are as given in , except that only source decks 555 and 733, and destination decks 128, 555, and 926, exist in the 1946-69 period (see Table 3 in ). However, because deck 555 has been reclassified as a [z] deck, it is no longer a destination deck. ii) Three additional substitutions are defined on p. K27 of Release 1, but only the first two are applicable to 1946-69 data: 1. Longitude and latitude tenths positions and associated indicator from matching HSST into deck 194. [NOTE: At this stage, tenths positions throughout deck 194 have been set to (5,5) as part of the field modifications (sec. 2.2) according to Release 1, Table K5-1.] 2. Calm wind speeds from matching HSST into deck 194 (wind speed in deck 194 must be missing). In addition, WI is set to 5. [NOTE: Prior to substitution, WI should be missing in deck 194 because conversion rules eliminate indicators associated with missing data. To ensure that deck 194 is uniformly set to WI=5, WI information is not substituted from HSST into deck 194; at this stage deck 194 reports with original (non-substituted) wind speeds have WI=5 as a result of field modifications (sec. 2.2).] Rules: Similarly to substitution of ID/II/PT, the two reports must match as "certain" duplicates without an hour cross (DS=8 or greater), such that the report receiving the substitution (deck 194 for substitutions 1 and 2) qualifies for DS=1, which is changed to DS=2 to indicate that substitution has taken place (thus reports with DS already set to a higher value as the result of an earlier match do not receive a substitution). Immediately after substitution (in contrast to ID/II/PT rules), reports are re-QC'd if the field substituted is subject to any QC checks (thus the quality code of a report may change during dupelim processing, and influence subsequent dup selection). Also, in contrast to ID/II/PT implementation, if a report that had a previous substitution as indicated by DS=2 is subject to a change to a higher dup status as the result of a subsequent match, the previously substituted fields are not deleted, and therefore it is not necessarily true that only reports with DS=2 will contain substituted data upon completion of dupelim. [NOTE: Release 1 processing differed in allowing substitution to occur into uncertain duplicates and worse duplicates, in that reports were re-QC'd after each substitution (i.e., substitution from one report into another, re-QC of report, and then the quality codes of the two reports were compared), and in that WI was substituted in addition to the calm wind speed. Future work, e.g., to substitute pressure tendency data into logbook from GTS reports, should reconsider the advantages of using the Release 1 method of performing a re-QC after each substitution, so that substitution results would possibly influence duplicate selection.] iii) One additional action related to substitution was defined on p. K35 of Release 1: "if one report of a pair of dups has a bucket indicator of 2 and the other report has sea surface temperature present but the bucket indicator is missing, it is replaced with a value of 2." For Release 1b, this is handled as a substitution of sea surface temperature method indicator (SI) value 10 (implied bucket) into a matching report, where SI=10 is initially set in the HSST source IDs during field modifications (sec. 2.2). Rules listed for ii) above also apply to this substitution (it is not necessary to re-QC, because SI is not subject to QC checks). {References} NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual. NCDC, Asheville, NC. NCDC (National Climatic Data Center), 1979: TDF-1150-152 reference pages (from R. Quayle; documentation date approximate), 2 pp. NCDC (National Climatic Data Center), 1983: Tape format documentation, tape deck 1148: "Conversion of XBT, MBT, and SD tapes to TDF-11 format" (original documentation date unknown, with February 1983 hand-written additions by P. Steurer), 2 pp. WMO (World Meteorological Organization), 1970: The Beaufort Scale of Wind Force (Technical and Operational Aspects), Reports on Marine Science Affairs, Report No. 3, Geneva, Switzerland, 22 pp.