=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.1 Conversion Action for Release 1 (1800-1979) LMR5 27 February 2004 ==================================================================== Document Revision Information (previous version: 9 September 2002): Updates Release 2.1 and ICOADS. ------------------------------------------------------------------------------- {1. Introduction} This document specifies action to convert Release 1 data from LMR5 into LMR6. LMR6 incorporates a variety of changes in field structure in comparison to LMR5, including some modifications in precision or range, and redefinitions of previously defined values in a few instances. In addition, a number of new fields have been added to LMR6. In some cases data for new LMR6 fields are available in the supplemental attachment (Attm4) of LMR5. For each field that was added or significantly changed, Table 1 summarizes the necessary action. Table 1 also indicates action for the quality control (QC), supplemental, and error attachments (Attm1, Attm4, and Attm5) in conversion from LMR5 to LMR6. Sec. 2 of this document provides details on conversion action for fields marked "b" in Table 1, and sec. 3 discusses conversion action for fields originally in TD-1100 requiring special treatment (marked "c" in Table 1). Specifications for later setting or resetting of some fields during preconditioning/dupelim (marked "p" in Table 1) are provided in separate documentation ( and ). Table 1. Action for conversion from LMR5 to LMR6 for fields added in LMR6, or for previously existing fields or attachments in LMR5 that are modified in LMR6 (excluding changes in field name or changes that are not applicable to Release 1 data). The field is set to a constant numeric value when a single numeric value is specified, or is extracted from the LMR5 supplemental attachment (Attm4) using the indicated range of characters (numbered with respect to the record length of the original format) when a range is given. Other actions are as follows: a = adjustments for increased field precision or new field base value. b = basic conversion action as defined in sec. 2. c = special TD-1100 conversion action as defined in sec. 3. m = usually missing in LMR5, but if extant:* i) 1970-79: set to missing, or always missing in LMR5 (A6). ii) pre-1970s: basic conversion action as defined in sec. 2. M = set to missing (new field in LMR6).* p = preconditioning/dupelim to provide or modify settings. - = unchanged ------------------------------------------------------------------------------- Action per original format (determined by SID) No. Field Description TD-1100 TD-1127 TD-1129 TD-1129M Exchange =============================================================================== **RPTID 6 6 6 6 6 2 YR year a a a a a 5 HR hour a a a a a 6**TI time indic. 0 0 0 0 0 7 LON longitude a a a a a 8 LAT latitude a a a a a 9 LI lat/lon indic. *** *** *** *** *** 12 PT platform type b,p b,p b,p b,p b,p 14 DS dup status m,p m,p m,p m,p m,p 15 DC dup check m,p m,p m,p m,p m,p 20 WI wind speed indic. b,p b,p b,p b,p b,p 28 T1 temp. indic. b,p b,p b,p b,p b,p 33 SI SST method indic. c,p b,p b,p b,p b,p 47**C1 country code c 78-79,b 79-80,b M&& M 48**C2 2nd country code M M M M M 49**SC ship course 83-83# 80-80 81-81 M&& M 50**SS ship speed 84-84# 81-81 82-82 M&& M 51**A barometric tendency 85-85# 82-82 83-83 M&& M 52**PPP amt. of SLP change 86-88# 83-85 84-86 M&& M 53**IS ice accretion 83-83## 86-86 87-87 M&& M 54**ES ice thickness 84-85## 87-88 88-89 M&& M 55**RS ice accretion rate 86-86## 89-89 90-90 M&& M 56**II ID indic. M,p M,p M,p M,p M,p 57- 64**ID(8) ID/call sign M,p 90-96,p 91-97,p M M 65**OS observation source M M M& M M 66**OP observation plat. M M M& M M 67**T2 2nd temp. indic. M,p M,p M,p M,p M,p 68**IX stn/weather indic. M M M& M M 69**WX wave per. indic. b b b b M 70**SX swell per. indic. b b b b M 71**IRD IMM receipt date M M M M M 72 A6 allowance 6 flag m,p m,p m,p m,p m,p Attm1 QC attachment *** *** *** *** *** Attm4 supplemental attachment b b b b b Attm5 error attachment b b,p b b b ------------------------------------------------------------------------------- * (m) Field missing unless Release 1 pre-dupelim volume lost, or (M) field not defined in LMR5. [NOTE: One 70s pre-dupelim volume was lost (box10s: 254-260) and two pre-70s volumes (box10: 102 and box10s: 297-308). Thus corresponding post-dupelim volumes had to be used instead, such that "m" fields (and Attm1) were extant in LMR5 (but A6 was applicable only to the pre-70s). Setting "m" fields to missing was recognized as a loss of useful information after the 70s was completed, hence the difference in processing (the 70s volume should be reprocessed using the pre-70s method for possible use in a future update).] ** New field in LMR6. # When additional data indicator = 6 (TD-1100 position 82). ## When additional data indicator = 1 (TD-1100 position 82). & Field associated with 1982 code change and thus inapplicable to Release 1 data. [NOTE: The TD-1129 format defined in Release 1, Table I1-4 differs in positions 146-148 from that finally adopted by NCDC (1989b). Positions 146- 148, respectively, are defined in Release 1 to contain OS, OP, and SID, and in NCDC (1989b) to contain OP, IMM quality code indicator, and IX. Furthermore, there are typographical errors in Release 1, Table I1-4 for positions 106-109, also involving the 1982 code change; NCDC (1989b) gives corrected information.] && Only source IDs 13 ("Eltanin;" deck 897) and 15 (South African Whaling; deck 899) were converted from TD-1129M into LMR5. NCDC (1989a) indicates that TD-1129M positions 79-124 are blank and unknown, respectively, in decks 897 and 899. [NOTE: Release 1, supp. I does not provide information on the mapping of decks 897 and 899 from LMR5 back into TD-1129M, as it does for other decks converted back into TD-1129M that do not fit a typical pattern. We assume both input and output TD-1129M data for decks 897 and 899 adhered to NCDC (1989a).] *** LI or Attm1 missing in LMR5 unless Release 1 pre-dupelim volume lost; basic conversion action is to omit any extant LI or Attm1 from the output LMR6. ---------- {2. Basic conversion action for individual fields} For each field, LMR5 values (which may reside in a differently named field) are given on the left, if any, and corresponding LMR6 values on the right. When a mapping occurs only under specified conditions, or a further explanation is warranted, such information appears separated by a colon (:) to the right of the resultant value. At the end of this section, basic conversion action is also specified for the attachments (Attm1, Attm4, and Attm5). 9) LI lat/lon indicator LI was missing in LMR5 unless the Release 1 pre-dupelim volume was lost (see Table 1); basic conversion action is to omit any extant LI from the output LMR6. 12) PT platform type Mapping of LMR5 ship type (ST; field 11) to LMR6 PT: 0 0 = US Navy or "deck" log, or unknown 1 1 = merchant ship or foreign military 2 2 = OSV--off station or station proximity unknown 3 3 = OSV--on station 4 4 = lightship 5 = ship 5 6 = moored buoy : see explanation below 7 = drifting buoy 8 = ice buoy 9 = ice station (manned) 6 10 = oceanographic station data (SD/Co22) : deck 891 6 missing : deck 897 7 11 = mechanical bathythermograph (MBT) : deck 891 supp. data type = 1 7 12 = expendable bathythermograph (XBT) : deck 891 supp. data type = 2 13 = Coastal-Marine Automated Network (C-MAN) 14 = other coastal/island station 15 = fixed ocean platform (plat, rig) Background: See Release 1, pp. I13-14 for a description of the original setting of ST. Detailed notes: a) ST=5 (buoy) is mapped into PT=6 (moored buoy) because we believe that all data assigned ST=5 were from moored buoys: NDBC (decks 876-882), and PMEL (deck 143). All ST=5 from decks other than these are also mapped into PT=6, for probable later removal during preconditioning. [NOTE: Most ST=5 from other decks appear to have originated from TD-1129, which does not possess a ship type field. Thus we believe these were introduced during or after conversion from TD-1129 based on an earlier plan for setting ST. Handwritten "5 = buoy" in TDF-11 reference manual is presumed inapplicable to Release 1 TD-1100 input data. However, preconditioning should probably issue diagnostics or retain any ST=5 that originated from TD-1100 data.] b) ST=6 was set only for decks 891 and 897, and ST=7 was set only for deck 891. For NODC data (deck 891), ST=6 (research ship) is mapped into PT=10, and ST=7 (XBT or MBT) is mapped into PT=11 or 12 depending on the deck 891 supplemental field for data type (TD-1100 position 103) [NOTE: deck 891 is not included in NCDC, 1968]. For "Eltanin" data (deck 897), PT becomes missing because "research ship" is no longer specifically defined. [NOTE: Release 1, p. I13 indicates that all NODC data were inadvertently assigned ST=6 during conversion into TD-11, but XBT or MBT data were reset to ST=7. The above action assumes that this resetting took place during conversion from TD-11 into LMR5.] 14) DS dup status Mapping of LMR5 dup status (DS; field 13) to LMR6 DS: 0 0 = unique 1 1 = best duplicate 2 2 = best duplicate with substitution 3 3 = worse duplicate, uncertain: uncertain WE match with hour cross 4 4 = worse duplicate, uncertain: uncertain WE match with no cross 5 5 = worse duplicate, uncertain: uncertain WE match with day cross 6 = worse duplicate, uncertain: time/space match with ID mismatch 7 = worse duplicate, uncertain: certain WE match with hour cross Background: DS was extant in LMR5 only from post-dupelim volumes, and mapped into LMR6 only for the pre-70s (see Table 1). Additional DS values indicating "certain" dup status are listed in , but these values were not output (WE indicates "weather element"). 15) DC dup check 0 0 = GTS and logbook match with SLP and SST match 1 1 = GTS and logbook match without SLP and SST match 2 2 = not GTS and logbook match Background: DC was extant in LMR5 only from post-dupelim volumes, and mapped into LMR6 only for the pre-70s (see Table 1). 20) WI wind speed indicator Mapping of LMR5 wind speed indicator (WI; field 20) to LMR6 WI (additional value WI=2 was defined after the LMR5 to LMR6 conversion took place): 0 0 = meter per second, estimated : possible for TD-1127/TD-1129 1 1 = meter per second, measured : possible for TD-1127/TD-1129 2 3 = knot, estimated : possible for TD-1127/TD-1129 3 4 = knot, measured : possible for TD-1127/TD-1129 5 = Beaufort force (based on documentation) 0,2 6 = estimated (units unknown)/unknown method : otherwise 1,3 7 = measured (units unknown) : otherwise 8 = high resolution measurement (e.g., hundredths of a meter per second) Background: The meaning of WI was changed in LMR6 to refer to the units in which the speed was originally recorded rather than those it was immediately converted from (LMR6 WI values 0, 1, 3, 4 are in accordance with WMO code 1855). The description for WI=6 is modified from current LMR6 documentation to add "/unknown method" due to the inability to distinguish between estimated and unknown in early data. In addition to the numeric mappings, it should be noted that missing WI is mapped to missing. Detailed notes: a) TD-1127 and TD-1129. The "original wind speed units indicator" is defined only in TD-1127 (position 97) and TD-1129 (position 98). If that field is blank (or erroneous), the original units are unknown and LMR5 values 0 and 2 (1 and 3) are mapped instead to 6 (7). Otherwise, LMR5 values 2 and 3 (0 and 1) are mapped to 0 and 1 (3 and 4) if the units in which the wind speed was originally recorded differ from those it was "immediately converted from." [NOTE: Apparently, Release 1 conversions yielded only WI values missing, 0, and 1, despite that fact that TD-11 data were "immediately converted from" knots, i.e., converted at NCAR to LMR5 from those units (in contrast, data in the Exchange format were converted from meters per second, but assigned only WI values missing or 1, for reasons discussed in item b). The above rules should correctly handle TD-1127 and TD-1129 data, assuming the distinction between estimated and measured was properly maintained.] b) TD-1100, TD-1129M, and Exchange. No indicator exists for the original wind speed units, thus LMR5 values 0 and 2 (1 and 3) are automatically mapped to 6 (7). [NOTE: The footnote on p. K28 of Release 1 discusses why WI in HSST data was set to missing. The HSST German, as opposed to the Netherlands, format indicates "wind speed converted from Beaufort force" in values of the f-wind flag (position 44). Contact with the Germans might be helpful to determine if these values warrant mapping into WI=5 as part of future preconditioning.] 28) T1 temperature indicator Mapping of LMR5 temperature indicator (TI; field 27) to LMR6 T1: 0 0 = degrees Celsius and tenths 2 1 = half degrees Celsius 1 2 = whole degrees Celsius 3 = degrees C and tenths, except dew point to whole degrees C 3 4 = degrees Fahrenheit and tenths 5 5 = half degrees Fahrenheit 4 6 = whole degrees Fahrenheit 7 = degrees F and tenths, except dew point to whole degrees F 8 = high resolution data (e.g., hundredths of a degree Celsius) 9 = other (refer to supplemental data) Background: Previously defined TI values were reordered, and new values were added. As noted in Release 1, pg. I15, TI had a different meaning for TD-11 as opposed to Exchange inputs, as is still the case following this one-to-one mapping. 33) SI sea surface temperature method indicator Mapping of LMR5 bucket indicator (BI; field 32) to LMR6 SI: 1 0 = bucket 1 = condenser inlet (intake) 2 = trailing thermistor 3 = hull contact sensor 4 = through hull sensor 5 = radiation thermometer 6 = bait tanks thermometer 7 = others 0 9 = unknown or non-bucket 2 10 = implied bucket (an HSST SID or any match) 11 = reversing thermometer 12 = electronic sensor Background: Previously defined BI values were reordered, and new values were added (SI values 0-7 follow the current WMO code). 47) C1 country code Overpunches are removed and the numeric country code value is retained according to Release 1a conversion specifications, except as amended in Table 2. Note that the original country code characters are retained in the supplemental attachment. Table 2. Mapping of country code characters from positions 78-79 of TD-1127 or positions 79-80 of TD-1129 into C1 (this mapping is also used to handle country codes extracted under certain conditions from TD-1100; see sec. 3). [NOTE: During later preconditioning (field modifications), characters '{' and '}' (interpreted as zero with different overpunches) were allowed over either or both positions of C1 (see and ).] ------------------------------------------------------------------------------- Pos. 78-79 or 79-80 C1 =============================================================================== '0}' or '}}' or '00' 00 = Netherlands* '0J' or '}J' or '01' 01 = Norway '0K' or '}K' or '02' 02 = USA '0L' or '}L' or '03' 03 = UK '0M' or '}M' or '04' 04 = France '0N' or '}N' or '05' 05 = Denmark '0O' or '}O' or '06' 06 = Italy '0P' or '}P' or '07' 07 = India '0Q' or '}Q' or '08' 08 = Hong Kong '0R' or '}R' or '09' 09 = New Zealand '1}' or 'J}' or '10' 10 = Ireland* '1J' or 'JJ' or '11' 11 = Philippines '1K' or 'JK' or '12' 12 = Egypt '1L' or 'JL' or '13' 13 = Canada '1M' or 'JM' or '14' 14 = Belgium '1N' or 'JN' or '15' 15 = South Africa '1O' or 'JO' or '16' 16 = Australia '1P' or 'JP' or '17' 17 = Japan '1Q' or 'JQ' or '18' 18 = Pakistan '1R' or 'JR' or '19' 19 = Argentina '2}' or 'K}' or '20' 20 = Sweden* '2J' or 'KJ' or '21' 21 = Federal Rep. of Germany '2K' or 'KK' or '22' 22 = Iceland '2L' or 'KL' or '23' 23 = Israel '2M' or 'KM' or '24' 24 = Malaysia '2N' or 'KN' or '25' 25 = USSR '2O' or 'KO' or '26' 26 = Finland '2P' or 'KP' or '27' 27 = Republic of Korea '2Q' or 'KQ' or '28' 28 = New Caledonia '2R' or 'KR' or '29' 29 = Portugal '3}' or 'L}' or '30' 30 = Spain* '3J' or 'LJ' or '31' 31 = Thailand '3K' or 'LK' or '32' 32 = Yugoslavia '3L' or 'LL' or '33' 33 = Poland '3M' or 'LM' or '34' 34 = Brazil '3N' or 'LN' or '35' 35 = Singapore '3O' or 'LO' or '36' 36 = Kenya '3P' or 'LP' or '37' 37 = Tanzania '3Q' or 'LQ' or '38' 38 = Uganda '3R' or 'LR' or '39' 39 = Mexico '4}' or 'M}' or '40' 40 = German Democratic Republic* ------------------------------------------------------------------------------- * Release 1a conversion specifications differ as follows: '0 ' or '} ' or '00' 00 = Netherlands '1 ' or 'J ' or '10' 10 = Ireland '2 ' or 'K ' or '20' 20 = Sweden '3 ' or 'L ' or '30' 30 = Spain '4 ' or 'M ' or '40' 40 = German Democratic Republic ---------- 69) WX wave period indicator 70) SX swell period indicator Generation of LMR6 WX/SX (no corresponding LMR5 field): 1 = period converted from code into whole seconds Background: Unless missing, WX and SX indicate that the wave and swell periods were converted from code into whole seconds. WX and SX were not defined in LMR5. However, LMR5 wave and swell periods (WP and SP) in whole seconds were taken in preference to the old codes if both were available (see Release 1, pp. F18-19). TD-1129 includes fields for both wave period (positions 101-102) and swell period (103-104) in seconds, whereas TD-1127 includes wave period, but not swell period, in seconds (positions 100-101). [NOTE: The '70s Decade file, which formed the bulk of the TD-1129 input for Release 1, was converted from TD-1127 such that resultant positions 103-104 were blank filled.] For TD-1127 and TD-1129 inputs, WX (and SX for TD-1129) should be set if WP (SP) is extant, but the corresponding supplemental field for wave (swell) period in seconds is missing. For TD-1100 and TD-1129M inputs, WX (SX) should be set if WP (SP) is extant. For Exchange inputs, WP (SP) should always be missing, as, therefore, should WX (SX). [NOTE: Conversion tested that extant supplemental periods in seconds matched WP or SP, and, for 2,159 reports when this was not the case, issued an error message. All 255 reports with this problem were examined from volume K3496K and found to be buoy data from deck 876, source ID 24 (Buoy data; TD-1129), with WP (and WH) missing in LMR. The Release 1 conversion program would have to be examined to see if, e.g., it required literally that both the code and seconds be present in TD-1129 before using the seconds for WP.] 72) A6 allowance 6 flag Mapping of LMR5 allowance 6 flag (A6; field 46) to LMR6 A6: 0 0 = six hours, or 1 1 = seven hours Background: A6 was extant in LMR5 only from post-dupelim volumes, applicable only to the pre-70s (see Table 1). Attm1. QC attachment This attachment was missing in LMR5 unless the Release 1 pre-dupelim volume was lost (see Table 1); basic conversion action is to omit any extant Attm1 from the output LMR6. Attm4. Supplemental attachment a) Convert EBCDIC characters into ASCII using the standard conversion table. b) For TD-1129 only, the supplemental attachment will be reduced from retention of positions 79-148, to retain only 79-80, 98-124, and 141-148 in conformance with Release 1a processing of TD-1129 data. [NOTE: Overlaying of QC flags as was done for Release 1 '70s data will no longer be possible.] Attm5. Error attachment a) Convert EBCDIC characters into ASCII using the standard conversion table. b) Field renumbering as needed for conversion from LMR5 field numbers into LMR6 field numbers. For fields new in LMR6 for which data are being converted from Attm4, data may have to be inserted into the error attachment if conversion errors are encountered. c) Any report containing Attm5 field numbers outside the defined (1-46) range of LMR5 is written out to the reject file, except that reports containing a "field 104" value in Attm5 (original Marsden Square in TD-11 data) are handled as follows: i) field 104 should refer to Marsden Square (MSQ) only in reports derived from TD-11 formats; therefore, reject any reports derived from the Exchange format as determined by SID (applicable only to pre-1970s data). ii) otherwise, use existing BOX10 to determine quadrant. iii) recompute Marsden Square (MSQ) from quadrant, latitude, and longitude according to the NCDC method for calculation of MSQ (this is described in Release 1a conversion specifications). iv) if MSQ agrees with the original MSQ from field 104, recompute BOX10 from MSQ, delete the field 104 information from Attm5, and do not reject the report. v) if MSQ disagrees with original MSQ from field 104, reject the report. [NOTE: An undocumented feature exists in LMR5 such that 100 was added to the original field number for MSQ, yielding Attm5 "field number 104." This feature was used to retain in Attm5 any MSQ inconsistent with latitude, longitude, and quadrant, primarily from source ID=23 ('70s Mislocated Data), but also from IMMPC data (SID=19), etc. Examination of reports containing field 104 revealed many cases of reports along 180-deg longitude, apparently resulting from a simple mismatch between quadrant and MSQ (i.e., the MSQ carried in TD-1129 was in the quadrant across the dateline from the reported quadrant) introduced by the NCDC method for calculation of MSQ. The above rule should recover these reports. There are also some reports at the North Pole with field 104, which according to Release 1 conventions were assigned to BOX10=1. These may or may not be rejected according to the above rule, depending on whether the quadrant associated with BOX10=1 happens to agree with the reported latitude, longitude, and the field 104 MSQ (in agreement with Release 1a processing, the resulting BOX10 is determined by longitude instead of arbitrary assignment to BOX10=1.] {3. Special TD-1100 conversion action for individual fields} 33) SI sea surface temperature method indicator During Release 1 initial conversion processing, the sea surface temperature method indicator was inadvertently not extracted from the corresponding supplemental field for deck 128 (position 99), which is the only documented TD-1100 deck possessing such information. Starting 1 January 1968, 2 with "+" overpunch (i.e., "B") in position 99 is to be mapped to SI=0 (bucket), and blank to SI=9 (unknown or non-bucket). [NOTE: During pre-1970s Release 1 dupelim processing, the indicator value for "implied bucket" (BI=2 in LMR5, equivalent to SI=10 in LMR6) may have been set in reports converted from TD-1100 (deck 128 or otherwise). To preserve this information in the post-dupelim volumes (see footnotes to Table 1), a two-part rule is adopted: i) any extant BI values in TD-1100 data are mapped into SI as shown in sec. 2; ii) SI information is extracted from the deck 128 supplemental field as described above. Under this rule, if BI=2 at the same time that supplemental information is available from deck 128, we retain the supplemental information (SI results are equivalent to those obtained by conversion of pre-dupelim volumes to LMR5, followed by dupelim processing).] 47) C1 country code Only for deck 128 and only when "Ocean Station Vessel or ship indicator" (OSVSI; position 81) is blank, 0, or 4 (i.e., non-OSV), extract country code from "Ocean Weather Station number" (OWSN; positions 78-79). Overpunches are removed and the numeric portion retained for C1, as specified in Table 2. Any overpunches appearing over the first or second character of the country code field are preserved by retaining OWSN in the supplemental attachment. {References} NCDC (National Climatic Data Center), 1968: TDF-11 Reference Manual. NCDC, Asheville, NC. NCDC (National Climatic Data Center), 1989a: Marine Data Users' Reference: 1854-1969. NCDC, Asheville, N.C. NCDC (National Climatic Data Center), 1989b: Marine Data Users' Reference: 1970-Current. NCDC, Asheville, N.C.