=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.1 French International Maritime Met. (IMM) Longitude Problem 27 February 2004 =================================================================== Document Revision Information (previous version: 9 September 2002): Updates for Release 2.1 and ICOADS. ------------------------------------------------------------------------------- {1. Introduction} French International Maritime Meteorological (IMM) reports during the period 1954-89 (or possibly 1981-88) in the region 90@E-90@W across the dateline may be mislocated (generally) 10@ in longitude toward the dateline. This problem was caused by an ambiguity in WMO's IMM formats for international exchange, which lead to incompatible conversions between countries. For international exchange in IMM formats, 4-digit longitude (LoLoLoLo) and quadrant (QC), as recorded in the ship's logbook and as required in code FM 13-VII SHIP, must be converted to 3-digit longitude (LoLoLo) and octant (applicable until the WMO code change effective 2 November 1994). For longitudes in the range 90.0@ through 180.0@ (i.e., LoLoLoLo is between 0900 and 1800) two methods for this conversion are known to have been used: a) by subtracting 0900 (or 90@) b) by subtracting 1000 (i.e., effectively removing the first digit) Nineteen contributing member countries replied to the WMO on this matter and only one, France, replied that it had used method a) since the commencement of the marine climatological summaries scheme (MCSS). Regarding the time period for the problem, work by the Dutch and by the UK Met Office showed that the problem began about 1981, with a possible transition period in 1981-82, instead of 1954. A 31 May 1989 letter from WMO stated that "the longitude conversion procedure used by France for ships' weather reports contributed under the MCSS will be changed to 'subtract 1000,' for all observations made as from 1 Sep. 1989." As part of a data exchange, France provided NCDC at the end of 1990 with a tape containing IMM data from French ships for 1954-88 and octants 1, 2, 6, and 7 (235,528 reports). This is henceforth referred to as the "French correction tape," although all the longitudes on the tape appear to be encoded according to the "subtract 900" procedure. It is not known why the French did not send data through August 1989. Apparently, some French data received previously to the correction tape were moved to their proper locations by NCDC's data operations branch. Dick Cram stated in a letter of 25 January 1989 to Brian Fullagar that "we adjusted our programs to properly convert the French data in June 1988. Positions of the French data will be correct in future IMMT shipments, unless the French change procedures without informing us." We expect that this special conversion program should have been decommissioned after the French changed procedures in September 1989. Inspection of the French correction tape revealed some problems. For example, some ID fields were invalid call signs (e.g., numeric) and the French indicated that they did not recognize position 79 in the IMMT format as the station/ weather indicator (IX). Insertion of earlier French data into the present IMM format has lead to some additional inconsistencies. For example, prior to 1968 the WMO code called for the wind wave direction to be reported, which does not match the current IMMT format, thus any early directional information is lost. Also, the French converted sea and swell wave period codes into actual seconds to match the current IMMT format. Table 1 shows in detail how the problem apparently works at different actual longitudes. Some important points from Table 1: a) Reports are subject to a 10@ shift towards the dateline, except reports at 180@ end up at 90@ (in the same octant). b) Hopefully, the French used "subtract 900" rather than literally "modulo 900", as the latter method produces identical results at 90@ and 180@. c) Thus regions without data are: 90.1-99.9 E or W (inclusive). Note that these regions form two columns of box10s (7, 43, ..., 619; and 24, 60, ..., 636), except that data should appear on the inclusive boundaries of these box10s at 90@E or W due to item a) above. d) Data from the region 170.1-179.9 E or W "fall off the end." The Release 1a IMMT conversion program considered such reports illegal and wrote them out to the reject file. Past NCDC conversions presumably considered such reports illegal, and recent NCDC conversion programs apparently wrote them out to MSQ 999. In initial Release 1a updates, the French longitude problem was corrected by deletion of all deck 926 reports during 1980-88 containing the French country code (C1=4), confined to the region 90E-90W across the dateline (deck 926 includes the French correction tape, SID=58, among other sources). For the 1980-97 update and extension, and for Release 1b, an improved correction was implemented as described in and . Table 1. IMMT encoding of actual longitudes greater than 89.9* according to the French method (presumably subtraction of 900 from the actual longitude in tenths), versus that used by other countries (modulo 1000), and the erroneous longitudes that result by conversion of French-encoded longitudes back to actual values using the other decoding method. ------------------------------------------------------------------------------- Actual lon. French IMMT Erroneous lon. from French IMMT Other IMMT (E or W) (minus 900) (by adding 1000 only if < 801) (modulo 1000)** =============================================================================== 90.0 000 100.0 900 90.1 001 100.1 901 90.2 002 100.2 902 ... ... ... ... 99.9 099 109.9 999 100.0 100 110.0 000 100.1 101 110.1 001 ... ... ... ... 169.9 799 179.9 699 170.0 800 180.0 700 170.1 801 *** 701 ... ... *** ... 179.9 899 *** 799 180.0 900**** 90.0 800 ------------------------------------------------------------------------------- * Applicable to octants 1, 2, 6, and 7, except reports from 90.0 may be assigned to octants 0, 3, 5, and 8 (?). ** Encoding of actual longitudes according to the method used by other countries yields two discontinuous ranges of meaningful coded values in this region: 000-800 and 900-999. *** Coded values 801-899 are meaningless under the scheme used by countries other than France. **** "Subtract 900" produces the same result as "modulo 900" except at 180.0 where the modulo would produce 000, which is indistinguishable from the results at 90.0. ---------- {2. Duplicate comparisons} In preparation for the original 1980-92 Release 1a update, special 2-year dupelim runs were made for various 10-degree boxes (box10) that illustrated the problem during 1980-81 and 1984-85: a) Uncorrected: By using the Release 1a input to dupelim. Thus among this input the correction tape, SID=58, is in the wrong location. As discussed in sec. 1, for initial Release 1a updates we deleted during preconditioning all reports with the French country code (C1=4), thus deleting the French correction tape and logbook data earlier received from France. b) Corrected: By using the Release 1a input to dupelim, except with the French correction tape data (SID=58) all moved to their proper locations. Results are summarized in Table 2. In addition, the 1970-79 Release 1 input to dupelim was subjected to special dupelim runs for various box10s (Table 3). Table 2. Duplicate comparison results for the French correction tape (SID=58) for selected box10s. Letters indicate whether SID=58 matched exclusively GTS (G) or logbook (L) data, or both GTS and logbook (B) data. The longitudes for SID=58 were corrected, unless results are shown in parentheses indicating that longitudes were uncorrected (uncorrected and corrected results shown only for box10=379). If the test revealed no legitimate matches, "-" is shown, or "?" indicates no available test results. ------------------------------------------------------------------------------- box10 -------------------------------------------------------------------- Year 263 264 296 297 378 379* =============================================================================== 1980 ? ? ? ? (-) B(L) 1981 ? ? ? ? (L) ?(L) 1984 G (-) G (L) ? ? 1985 G (L) ? ? ? ? ------------------------------------------------------------------------------- * The corrected box10=379 run for 1980-81 hit a print limit of 500 before reaching 1981, indicating that there were 500 or more 1980 matches. This is in contrast to the small number, 16, of uncorrected logbook matches in 1980. ---------- Table 3. Duplicate comparison results for 1970-79 input to Release 1 dupelim. "G" ("-") indicates the presence of French logbook data and that matches were found (not found) of the logbook data with GTS (ID1=F) data for selected box10s ("?" indicates no available test results, e.g., due to reaching a print limit). ------------------------------------------------------------------------------- box10 ----------------------------------------------------------------------- Year 261 275 276* 297 312* =============================================================================== 1970 G G G - G 1971 G G G - G 1972 G G** G** G G** 1973 ? G** G** G G** 1974 ? G** G** G G** 1975 ? G G (no data) G 1976 ? ? G*** G ? 1977 ? ? G*** G ? 1978 ? ? G - ? 1979 ? ? ? G ? ------------------------------------------------------------------------------- * French IMM data-void box10s (except for reports at 90.0@), if the longitude problem exists. ** Country code (C1) missing in deck 128/927. *** Country code (C1) missing in deck 926. ---------- {3. Questions/conclusions} a) Apparently WMO has not defined inclusivity rules at the boundaries of the octants, e.g., that require reports at longitude 90.0 to be in one octant or the other. Therefore, the French may have assigned data at 90.0E to, say, octant 4 instead of 2. This raises the question of the exact geographical extent of the data obtained on the correction tape. b) Following is excerpted from the Release 1a tapelist, where F2ZG1 is the French correction tape: #VOLUME SID RECEIVED REPORTS POR USAGE LABEL F2ZG1 58 91/01/30 235528 1954 1988 RSabcde C2=04 IRD=9111 I3ZG11 46 91/12/11 18399 1982 1985 bc C2=04 IRD=8610 I3ZG12 46 91/12/11 16306 1982 1987 R bcd C2=04 IRD=8901 I3ZG13 47 92/03/13 8900 1985 1988 RS cde C2=04 IRD=9109 I3ZG70 47 93/02/16 17814 1988 1992 efg C2=04 IRD=9302 Tapes received directly from France contained data only for the U.S. area of responsibility, which includes only a portion (90W-170W) of the total area affected by the French longitude problem. This explains why conversion outputs for I3ZG11-12, for example, had no rejections due to data actually in the region 170.1-179.9 (as noted above, such data encoded according to the French method and decoded according to the other method "fall off the end," and the Release 1a IMMT conversion program considered such values illegal and wrote them out to the reject file). c) Other responsible members should have received data from France for the remaining portion (i.e., 90E-70W across the dateline) of the total area affected by the French longitude problem (i.e., Hong Kong, India, Japan, Netherlands, and USSR), and may have provided data to NCDC under bilateral agreements. However, examination of the available original IMM tapes (I3ZG01-78) did not reveal any report rejection problems that appear to be related to the French longitude error. This may only mean that, e.g., data from France received with longitudes 801-899 were rejected during processing by other responsible members. d) Problems exist in the country code field (C1). Particularly for data during 1972-74, it appears that significant amounts of French logbook data have C1 set to missing. e) In view of the possibility of mislocated receipts received through other countries, why does the French tape end in 1988, when the problem went through August 1989? (From 31 May 1989 WMO letter: "the longitude conversion procedure used by France...will be changed...for all observations made as from 1 Sept. 1989.") This raises the question of whether Release 1a preconditioning action should have been continued through August 1989. f) How did NCDC handle earlier French reports in the two 10-degree bands 170.1-179.9 (i.e., resultant coded IMMT longitudes 801-899? Unless the data are recoverable, e.g., from MSQ 999, we'll have to make do with what is available on the correction tape (but converted to the current IMMT format). However, if the French problem only goes back to about 1981, we shouldn't have a reject problem for previous years (unless the French at some point provided some delayed data that were subject to the problem). g) A plot was created of numbers of reports per year on the French correction tape, and of rejected reports per year from our conversion into LMR6. The reject curve was relatively constant. This may indicate that the correction tape was uniformly coded using "subtract 900." I.e., perhaps the French stored their data in a different format, and at the time of modifications to the software (e.g., for the 1982 code change) the method of encoding longitudes was inadvertently changed. This plot also showed a significant drop in the number of reports on the French correction tape in 1973, which is unexplained.