Comprehensive Ocean-Atmosphere Data Set; Release 1
Supplement I: Long Marine Report Conversions

0. Introduction

This is a collection of background details for the conversion to LMR (supp. F) from various character-based formats, and for the conversion back into TD-1129(M). The common characteristics and incompatibilities of the various formats in TD-11 and the Exchange format are discussed in sec. 1, as background for the design of LMR. In sec. 2, extensive corrections that were made to correct known data problems and other details of the conversion into LMR are given. Finally, sec. 3 defines changes made in the conversion from LMR back into TD-1129(M).

1. Long Marine Report Background

Brief format layouts give the primary fields in TD-11 (Tape Deck-11); more specific information on TD-11, including the representation of data within fields, will be found in [5], [6], and [7]. Different versions of the Exchange format are described in more detail since documentation of these was not readily available.

1.1 TD-11

TD-11 formats can be grouped into three classes:

1) TD-1100
2) TD-1127
3) TD-1129(M)
These trace sequentially the evolution of the format through time in response to changing observational methods, often as a result of differing WMO code conventions, and a desire (perhaps unrealistic) to arrive at a single format suitable for all such data. The following discussions of each class include lists of data sets processed from each class.

TD-1100
Data sets processed: Atlas, HSST Pacific, Old TDF-11 Supplement B-C, Monterey Telecom., OSV, MSQ 486 and 105 Omissions, NODC Surface, OSV Z.

The original TDF-11 (Tape Data Family-11) now called TD-1100, comprises at least 18 distinct sub-formats, linked by tape deck number "11xx" to a source card deck "1xx." Exceptions are that tape deck 1181 describes card deck 281, and later additions such as card decks 555 and 891 cannot fit this pattern either. Report length is 140 characters, but a variable number of characters at the end could be blank depending on sub-format. The form of any member of TD-1100 can be expressed by

location = 1-26(26) + regular1 = 27-77(51) + irregular1 = 78-140(63).
This notation shows the start position i, end position j, and length k of sections as given by "section = i-j(k)", which when concatenated ("+") depict the whole format. The format layout is given by Table I1-1.


Table I1-1
TD-1100 Format

Field     Char                                                  
number  position  Description                                   
----------------------------------------------------------------
     1      1-3   card deck                                     
     2      4-6   10° Marsden Square                            
     3      7-8   1° Marsden sub-square                         
     4        9   quadrant (1-4)                                
     5    10-12   latitude (degrees N, S)                       
     6    13-16   longitude (degrees E, W)                      
     7    17-20   year                                          
     8    21-22   month                                         
     9    23-24   day                                           
    10    25-26   hour (GMT)                                    
    11    27-29   wind direction and indicator (code)           
    12    30-33   wind speed and indicator (knots)              
    13    34-36   visibility and indicator (code)               
    14    37-38   present weather (code)                        
    15       39   past weather (code)                           
    16    40-44   sea level pressure (mb)                       
    17    45-48   temperature indicator and air temperature (°C)
    18    49-51   wet bulb temperature (°C)                     
    19    52-54   dew point temperature (°C)                    
    20    55-57   sea surface temperature (°C)                  
    21    58-60   air-sea temperature difference (°C)           
    22       61   total cloud amount (oktas)                    
    22       62   lower cloud amount                            
    22       63   type of low cloud                             
    22       64   cloud height indicator                        
    22       65   cloud height                                  
    22       66   type of middle cloud                          
    22       67   type of high cloud                            
    23    68-69   direction of waves (code)                     
    24       70   period of waves (code)                        
    25    71-72   height of waves (1/2 meters)                  
    26    73-74   direction of swell (code)                     
    27       75   period of swell (code)                        
    28    76-77   height of swell (1/2 meters)                  
    29    78-79   ocean weather station number or country code  
    30       80   card indicator                                
    31       81   ship type                                     
    32       82   additional data indicator                     
 33-36    83-88   additional data                               
    37       89   ice indicator                                 
    38    90-93   ship number                                   
    39   94-140   supplemental data                             
----------------------------------------------------------------



Within irregular1 the positions 78-81(4) and 89-93(5) are themselves regular for all sub-formats; these subsections will be denoted standard1 and standard2, respectively. Positions 82-88(7) contain additional data, whose contents depend on the indicator in position 82. Finally, positions 94-140(47) contain supplemental data whose contents depend on the sub-format. Thus irregular1 takes the overall form
standard1 = 78-81(4) + additional = 82-88(7) + standard2 = 89-93(5) + supplemental = 94-140(47).
The supplemental section is used to preserve the original units or form of fields whose conversion might be open to question, or which are unique to a sub-format. Table I1-2 shows the (supposedly) non-blank length of supplemental for each of 17 sub-formats.


Table I1-2
Supplemental Length

Tape deck   Card deck   Supplemental = 94-   Trailing blanks
------------------------------------------------------------
  1110         110           140(47)                0       
  1116         116           120(27)               20       
  1118         118           118(25)               22       
  1119         119           112(19)               28       
  1128         128            101(8)               39       
  1181         281           134(41)                6       
  1184         184           112(19)               28       
  1185         185            100(7)               40       
  1187         187           119(26)               21       
  1188         188             97(4)               43       
  1189         189           116(23)               24       
  1192         192           136(43)                4       
  1193         193           116(23)               24       
  1194         194           120(27)               20       
  1195         195           113(20)               27       
  1196         196           126(33)               14       
  1197         197           125(32)               15       
------------------------------------------------------------



TD-1127
Data set processed: '70s Mislocated Data.

Tape Deck-1127 has the general form

location = 1-26(26) + regular1 = 27-77(51) + regular2 = 78-140(63),
where regular2 takes the place of irregular1 in TD-1100. Quality flags have been added and the format of regular2 is invariant, regardless of deck number. Also, the call sign is usually used in place of ship number. Table I1-3 gives the format layout.


Table I1-3
TD-1127 Format

Field     Char                                                  
number  position  Description                                   
----------------------------------------------------------------
     1      1-3   card deck                                     
     2      4-6   10° Marsden Square                            
     3      7-8   1° Marsden sub-square                         
     4        9   quadrant (1-4)                                
     5    10-12   latitude (degrees N, S)                       
     6    13-16   longitude (degrees E, W)                      
     7    17-20   year                                          
     8    21-22   month                                         
     9    23-24   day                                           
    10    25-26   hour (GMT)                                    
    11    27-29   wind direction and indicator (code)           
    12    30-33   wind speed and indicator (knots)              
    13    34-36   visibility and indicator (code)               
    14    37-38   present weather (code)                        
    15       39   past weather (code)                           
    16    40-44   sea level pressure (mb)                       
    17    45-48   temperature indicator and air temperature (°C)
    18    49-51   wet bulb temperature (°C)                     
    19    52-54   dew point temperature (°C)                    
    20    55-57   sea surface temperature (°C)                  
    21    58-60   air-sea temperature difference (°C)           
    22       61   total cloud amount (oktas)                    
    22       62   lower cloud amount                            
    22       63   type of low cloud                             
    22       64   cloud height indicator                        
    22       65   cloud height                                  
    22       66   type of middle cloud                          
    22       67   type of high cloud                            
    23    68-69   direction of waves (code)                     
    24       70   period of waves (code)                        
    25    71-72   height of waves (1/2 meters)                  
    26    73-74   direction of swell (code)                     
    27       75   period of swell (code)                        
    28    76-77   height of swell (1/2 meters)                  
    29    78-79   country code                                  
    30       80   ship direction (code)                         
    31       81   ship speed (code)                             
    32       82   barometric tendency (code)                    
    33    83-85   amount of pressure change (mb)                
    34       86   type of ice accretion on ship (code)          
    35    87-88   thickness of ice on ship (cm)                 
    36       89   rate of ice accretion (code)                  
    37    90-96   ship, OSV, or buoy call sign                  
    38       97   original wind speed units indicator           
    39       98   original temperature units indicator          
    40       99   sea temperature measurement method indicator  
    41  100-101   wind wave period (seconds)                    
    42      102   description of ice type (code)                
    42      103   effect of ice on navigation (code)            
    42      104   bearing of principal ice edge (code)          
    42      105   distance to ice edge from ship (code)         
    42      106   orientation of ice edge (code)                
    43  107-108   amount of precipitation (code)                
    43  109-110   time period for precip. amount (code)         
    44      111   significant cloud amount (code)               
    45      112   significant cloud type (code)                 
    46  113-114   significant cloud height (code)               
    47      115   ship position - flag                          
    48      116   wind - flag                                   
    48      117   visibility - flag                             
    48      118   present weather - flag                        
    48      119   past weather - flag                           
    48      120   pressure - flag                               
    48      121   air temperature - flag                        
    48      122   dew point/wet bulb - flag                     
    48      123   sea surface temperature - flag                
    48      124   cloud - flag                                  
    48      125   wave - flag                                   
    48      126   swell - flag                                  
    48      127   pressure change - flag                        
    49  128-129   quality code                                  
    50  130-134   Julian date (year, day) of QC                 
    51  135-136   blank                                         
    52  137-140   reserved for NCDC use only                    
----------------------------------------------------------------



TD-1129(M)
Data sets processed: Eltanin, Japanese, South African Whaling, Australian, IMMPC, '70s Decade, Buoy Data.

This format is intended to replace both TD-1100 and TD-1127 as an all purpose character-based marine format. TD-1129 is for recent data, and its variant TD-1129M is for old data from TD-1100 (e.g., the Eltanin and South African Whaling data sets). The notation TD-1129(M) refers to either TD-1129 or TD-1129M. In general the form is

location = 1-26(26) + regular3 = 27-78(52) + irregular2 = 79-148(70).
In comparison with regular1, overpunches have been eliminated and the air-sea temperature difference has been dropped from regular3, so that its contents are essentially equivalent to regular1. Table I1-4 gives the format layout for TD-1129.


Table I1-4
TD-1129 Format

Field     Char                                                           
number  position  Description                                            
-------------------------------------------------------------------------
     1      1-3   card deck                                              
     2      4-6   10° Marsden Square                                     
     3      7-8   1° Marsden sub-square                                  
     4        9   quadrant (1-4)                                         
     5    10-12   latitude (degrees N, S)                                
     6    13-16   longitude (degrees E, W)                               
     7    17-20   year                                                   
     8    21-22   month                                                  
     9    23-24   day                                                    
    10    25-26   hour (GMT)                                             
    11       27   wind direction indicator                               
    11    28-29   wind direction (code)                                  
    12       30   wind speed indicator                                   
    12    31-33   wind speed (knots)                                     
    13       34   visibility indicator                                   
    13    35-36   visibility (code)                                      
    14    37-38   present weather (code)                                 
    15       39   past weather (code)                                    
    16    40-44   sea level pressure (mb)                                
    17       45   temperature indicator                                  
    17    46-49   air temperature (°C)                                   
    18    50-53   wet bulb temperature (°C)                              
    19    54-57   dew point temperature (°C)                             
    20    58-61   sea surface temperature (°C)                           
    21       62   total cloud amount (oktas)                             
    21       63   low or middle cloud amount                             
    21       64   type of low cloud                                      
    21       65   cloud height indicator                                 
    21       66   lowest cloud height                                    
    21       67   type of middle cloud                                   
    21       68   type of high cloud                                     
    22    69-70   direction of waves (code)                              
    23       71   period of waves (code)                                 
    24    72-73   height of waves (1/2 meters)                           
    25    74-75   direction of swell (code)                              
    26       76   period of swell (code)                                 
    27    77-78   height of swell (1/2 meters)                           
    28    79-80   country code                                           
    29       81   ship direction (code)                                  
    30       82   ship speed (code)                                      
    31       83   barometric tendency (code)                             
    32    84-86   amount of pressure change (mb)                         
    33       87   type of ice accretion on ship (code)                   
    34    88-89   thickness of ice on ship (cm)                          
    35       90   rate of ice accretion (code)                           
    36    91-97   ship, OSV, or buoy call sign                           
    37       98   original wind speed units indicator                    
    38       99   original temperature units indicator                   
    39      100   sea temperature measurement method indicator           
    40  101-102   wind wave period (seconds)                             
    41  103-104   swell wave period (seconds)                            

    42      105   concentration of ice (new code 1982)                   
                  description of ice type (code)                         
    42      106   stage of ice development (new code 1982)               
                  effect of ice on navigation (code)                     
    42      107   ice of land origin (new code 1982)                     
                  bearing of principal ice edge from ship (code)         
    42      108   bearing of principal ice edge from ship (new code 1982)
                  distance to ice edge from ship (code)                  
    42      109   situation or trend (new code 1982)                     
                  orientation of ice edge (code)                         

    43  110-111   amount of precipitation (code)                         
    43  112-113   time period for precip. amount (code)                  
    44      114   significant cloud amount (code)                        
    45      115   significant cloud type (code)                          
    46  116-117   significant cloud height (code)                        
    47      118   second past weather (code)                             
    48  119-120   second swell direction (code)                          
    49  121-122   second swell period (seconds)                          
    50  123-124   second swell height (1/2 meters)                       
    51      125   ship position - flag                                   
    52      126   wind - flag                                            
    52      127   visibility - flag                                      
    52      128   present weather - flag                                 
    52      129   past weather - flag                                    
    52      130   pressure - flag                                        
    52      131   air temperature - flag                                 
    52      132   wet bulb temperature - flag                            
    52      133   dew point temperature - flag                           
    52      134   sea surface temperature - flag                         
    52      135   cloud - flag                                           
    52      136   wave - flag                                            
    52      137   swell - flag                                           
    52      138   pressure change - flag                                 
    53  139-140   quality code                                           
    54  141-142   QC - year                                              
    54  143-144   QC - month                                             
    55      145   indicator for wave measurement (1982 code)             
    56      146   source of observation on card (1982 code)              
    57      147   observation platform (1982 code)                       
    58      148   source ID (A-X corresponds to 1-24 in LMR)             
-------------------------------------------------------------------------

Note: In the original published edition of COADS Release 1, field 42 was disordered and incomplete; it is corrected above. Also, NCDC made changes in TD-1129 c. 1989 in fields 56-58.


For recent data (TD-1129), irregular2 assumes the invariant form shown in Table I1-4, which accommodates recent WMO code changes at the expense of adding seven characters. (One character is also added to regular3 so the report length is eight characters longer.)

For older data (TD-1129M), irregular2 contains information that is practically equivalent to that contained in irregular1 according to the following transformation.

a) Standard1 data 78-81(4) in TD-1100 move to 79-82(4) in TD-1129M.
b) Additional data 82-88(7) move to 141-147(7).
c) Standard2 data 89-93(5) move to 83-87(5).
d) Supplemental data 94-140(47) move to 88-124(37).
Clearly, depending on the sub-format, supplemental data may not all fit. Decks 110, 117, 281, 192, 150, 151, and 152 require special treatment:

In practice, these special transformations do not work as stated because supplemental, standard1, and standard2 often contain undocumented or erroneous characters. Some of these characters were "area codes" assigned for special Atlas studies or they were dates when data were added to a data base at NCDC.

1.2 Exchange Format

The United States, Germany (F.R.G.), the Netherlands, and the United Kingdom apparently used this as the format for exchange of reports gathered in the Historical Sea Surface Temperature (HSST) Data Project. In order to minimize processing, data from the German and Netherlands areas of responsibility (Atlantic and Indian Oceans) were translated from the Exchange format directly into LMR, even though a TD-1100 transcription was available from NCDC. For the United States area of responsibility (Pacific Ocean), the TD-1100 was used because it contained additional data not available in the Exchange format.

The Exchange format obtained (Table I1-5) is considerably shorter (46 characters) than any class of TD-11. The format differs slightly depending on which country provided a report (no details were available on the U.K. format), and each area of responsibility contains data merged together from the four countries. Although source ID (and card deck) identify the area of responsibility, no identifier is available in the Exchange format showing which country provided a report. For translation to LMR, a special supplemental was defined for this format (later subject to special treatment as described for deck 152 in translation from LMR to TD-1129M). Characters from columns 33-35 and 42-46 were saved in the supplemental attachment. This includes the wind speed and all flag information from the original format, as shown in Table I1-5.


Table I1-5
Exchange Format

Field     Char                                                  
number  position  Description                                   
--------------------------------------------------------------------------------
     1      1-1   octant                                                        
                  ----------------------------------------------                
                   S. Hemisphere     N. Hemisphere     Longitude                
                  ----------------------------------------------                
                         5                 0            0-90°W                  
                         6                 1           90-180°W                 
                         7                 2           180-90°E                 
                         8                 3            90-0°E                  
                  ----------------------------------------------                
     2      2-3   10° square (tens digit of latitude and longitude)             
     3      4-5   month                                                         
     4      6-9   year                                                          
     5    10-11   latitude (units and tenths digit)                             
     6    12-13   longitude (units and tenths digit)                            
     7    14-15   day                                                           
     8    16-17   hour (GMT)                                                    
     9    18-21   sea surface temperature                                       
    10    22-25   air temperature                                               
    11    26-29   wet bulb temperature                                          
                  in tenths of a degree Celsius, or replaced by 999 if missing, 
                  with the sign as the first position.  In the U.S. and Nether- 
                  lands formats the sign is blank if the value is positive.  In 
                  the German format the sign is "+", "-", or "E" (the latter    
                  used only for wet bulb with ice).                             
    12    30-32   wind direction                                                
                  in whole degrees from north (converted from other units if    
                  necessary) with 000 for calm, 990 for variable, or 999 for    
                  missing.                                                      
    13    33-35   wind speed                                                    
                  in tenths of a meter per second (converted from other units   
                  if necessary) with 999 for missing.                           
    14    36-40   sea level pressure                                            
                  in tenths of a millibar, with 99999 for missing.              
    15       41   total cloud amount                                            
                  in oktas, with 9 for obscured, or blank for missing.  In the  
                  U.S. and Netherlands formats, missing occurs with f-sus2      
                  (field 20) of 2, 3, 6, or 7.                                  
    16       42   f-sea                                                         
                  U.S. and Netherlands formats only, flag for measurement       
                  precision of sea surface temperature and the state of the wet 
                  bulb.  Codes 0 to 9: Codes 0 to 4 (Netherlands for frozen wet 
                  bulb); codes 5 to 9 for unfrozen wet bulb, even when showing  
                  temperature below freezing point (or Netherlands wet bulb     
                  temperature missing):                                         
                  --------------------------------                              
                   Code       Code      Precision                               
                  --------------------------------                              
                    0          5          0.1°F                                 
                    1          6          0.1°C                                 
                    2          7          0.5°F                                 
                    3          8          0.5°C                                 
                    4          9       1°F or 1°C                               
                  --------------------------------                              
    17       43   f-air                                                         
                  U.S. and Netherlands formats only, flag for measurement       
                  precision (as given by f-sea) of air and wet bulb tempera-    
                  tures.  Codes 0 to 9: codes 0 to 4 (Netherlands for wet bulb  
                  temperature missing); codes 5 to 8 for temperatures measured  
                  by an aspirated or whirling psychrometer; code 9 for original 
                  units or precision of temperatures unknown.                   
    18       44   f-wind                                                        
                  Flag for wind observation.  Codes 0 to 9: codes 0 to 4        
                  indicate wind speed measured; codes 5 to 9 indicate wind      
                  speed estimated or converted from Beaufort force, or method   
                  of observation unknown (in the German format, only codes 1,   
                  6, and 7 are documented, with 6 and 7 differing by definitely 
                  indicating a conversion from Beaufort force):                 
                  --------------------------------                              
                   Code     Code    Point compass                               
                  --------------------------------                              
                    0        5           360                                    
                    1        6            36                                    
                    2        7            32                                    
                    3        8            16                                    
                    4        9             8                                    
                  --------------------------------                              
    19       45   f-sus1                                                        
                  U.S. and Netherlands formats only, flag for suspect values of 
                  sea surface and air temperatures, and wind.  Codes 0 to 7:    
                  -----------------------------------------------------------   
                   Code     Condition                                           
                  -----------------------------------------------------------   
                    0       none of the following conditions                    
                    1       suspect sea surface                                 
                    2       suspect air temperature                             
                    4       suspect wind                                        
                  3,5-7     more than one value suspect, codes added together   
                  -----------------------------------------------------------   
    20       46   f-sus2                                                        
                  U.S. and Netherlands formats only, flag for suspect values of 
                  pressure, cloud amount, or additional report.  Codes 0 to 7:  
                  -----------------------------------------------------------   
                   Code     Condition                                           
                  -----------------------------------------------------------   
                    0       none of the following conditions                    
                    1       suspect pressure                                    
                    2       cloud amount not reported                           
                    4       additional report at same time in the same          
                            1° square though not identical.                     
                  3,5-7     more than one value suspect, codes added together.  
--------------------------------------------------------------------------------



1.3 LMR

The three TD-11 classes take the following forms:

TD-1100: location = 1-26(26) + regular1 = 27-77(51) + irregular1 = 78-140(63)
TD-1127: location = 1-26(26) + regular1 = 27-77(51) + regular2 = 78-140(63)
TD-1129: location = 1-26(26) + regular3 = 27-78(52) + irregular2 = 79-148(70)
Note that location remains unchanged. Actually positions 1-45 are invariant. Moreover, the contents of regular1 and regular3 are essentially equivalent as noted previously. Thus, only the final section of each class contains variable information, and the first two sections in each can all be entered into a uniform location and regular section in LMR. Adding a control section and an irregular section at the end completes the format as described fully in supp. F.

2. Corrections and Conversion into LMR

A number of known data problems were corrected at the conversion into LMR, and prior to sorting the data as required by duplicate elimination. These and other conversion details given here impact the LMR, and in some cases also apply to the TD-1129(M). See supp. K for a description of earlier changes made in the translation from miscellaneous formats (e.g., Japanese, Australian) into TD-11, performed at NCDC.

2.1 Character Translation Tables

Some possible overpunch-numeric combinations can result in confusing character conventions. Therefore, the following conventions were always used. For the most part, these are consistent with the most commonly used conversions.

12 overpunches and numbers 1-9 map to letters A-I.
11 overpunches and numbers 1-9 map to letters J-R.
These letters are well defined in all character sets. In the supplemental attachment, ebcdic is used to represent the letters A-Z, and special characters translate into the ship character set as shown in Table I2-1.


Table I2-1
Translation into Ship Characters

------------------------------------------------------------
              |   ebcdic   |    ascii    |   CDC dpc   |Ship
026 punch code|  Hex   Char|  Hex   Char | Octal  Char | Hex
--------------|------------|-------------|-------------|----
     12-0     | C0/4C  {/< | 7B/3C  {/<  |  72     <   |  C0
     11-0     | D0/4F  }/| | 7D/21  }/!  |  66     !   |  D0
       12     |  4E     +  |  2B     +   |  45     +   |  CB
       11     |  60     -  |  2D     -   |  46     -   |  DA
    0-8-7     |  50     &  |  26     &   |  67     &   |  CA
      0-1     |  61     /  |  2F     /   |  50     /   |  E1
   12-8-4     |  5C     *  |  2A     *   |  47     *   |  EA
------------------------------------------------------------



2.2 Watch Number to Hour

For deck 194, if watch number was 6, 1 was added to the day (and month/year if applicable), and watch number was changed to 9.

2.3 Hour

Any time hour was 24, 1 was added to the day (and month/year if applicable), and hour was changed to 00. An hour of 99 was considered missing.

2.4 Pre-July 1963 Wave Fields Applicable to both wave and swell data* before July 1963 (exclusive):

a) If 51 ≤ direction ≤ 86, then direction = direction - 50 and height = height + 10.

b) If direction equals 99 and height < 10, then height = height + 10.

________________________

* Prior to the code change of 1 July 1963, only the higher of the (wind) wave and swell was reported. Standard practice at NCDC was to put this into the wave (not swell) fields.
________________________

2.5 Cloud, Wave, and Swell Fields

Sometimes / was keyed in place of - in fields where - was a legal value, and / or - were keyed in place of space (S) when the cloud or wave fields were missing. Specifically:

a) Cloud Fields. (& was also keyed in place of -.) These two steps were used to determine if the LMR cloud fields C, NH, CL, HI, H, CM, CH were all missing.

i) Any / or & changed into -.
ii) If all seven positions were then S or - in any combination, then all seven fields were considered missing and were changed into SSSSSSS.
Otherwise - was changed into S in fields where - was not legal (C, NH, HI).

b) Wave and swell fields were all missing if they fit one of these 5-character patterns:

i) SSSSS
ii) -----
iii) /////
and were all changed into pattern i). In addition, waves only (not swells) were all missing if they fit one of these 5-character patterns (which include i)-iii) as special cases):
iv) DDSSS
v) DD---
vi) DD///
where D is any character. That is, if the last three characters were SSS, ---, or /// this field was changed into pattern i).

2.6 Indicators Referring to Missing Data

Non-blank indicators referring to blank (missing) data were made blank:

a) Wind direction indicator if direction was blank.
b) Wind speed indicator if speed was blank.
c) Visibility indicator if visibility was blank.
d) Temperature indicator if all of the temperature fields were blank.
e) Cloud height indicator if cloud height was blank.
This rule does not apply to indicators that refer to erroneous data.

2.7 Time/Space Location Errors

Reports with errors or inconsistencies in time or geographical location were written out to a reject file for later work. For reports in which the Marsden Square disagrees with quadrant, latitude, and longitude (or the corresponding inconsistencies in Exchange format data), both the 10° box and 1° MSQ should be recomputed and the report relocated accordingly, when time permits. This was done for a few of the smaller data sets.

2.8 Card Deck Assignments

The following new card decks were assigned during this project:

*155 HSST Indian (Boulder conversion)
*156 HSST Atlantic (Boulder conversion)
897 Eltanin
*898 Japanese (change from 926)
899 S. African
900 Australian
926 IMMPC
Only those decks with an asterisk (*) required action at this conversion stage, the others had been assigned during NCDC's conversions. NCDC assigned 154 to its conversion of both the HSST Indian and Atlantic basins.

2.9 Monterey Telecom. Pre-processing

Owing to the questionable quality of this data set, checks were made for the following conditions:

a) Positions 70-77 (period and height of sea; direction, period, and height of swell) equal to -0031000.
b) Present weather missing (blank) when past weather was any nonblank character.
c) Calm wind direction when speed was greater than or equal to 7 knots.
Any report with one or more of a)-c) true was written to a reject file. This was expected to eliminate most hard duplicates (supp. K) internal to the Monterey Telecom.

2.10 Existing Ship Type

Only TD-1100 inputs had a field for ship type, to which these changes were made:

a) For any deck, a ship type of 2 with a negative overpunch was converted to 3 in order to help eliminate overpunches from the format. Subsequently, ship type was set to 2 if not 2 or 3 for source IDs 8, 9, and 20 (OSV data).
b) For source IDs 2 and 7 (HSST Pacific and Monterey Telecom.), ship type was set to missing.
c) For deck 891, a ship type of 6 was intended to indicate a research ship (or SD, meaning station data) but was inadvertently assigned to every report in this deck, including bathythermographs (XBT and MBT), during the translation into TD-11. Position 103 was expected to contain the type. Thus,
if type = 1 (MBT) then ship type = 7;
if type = 2 (XBT) then ship type = 7;
if type = 3 (SD) then ship type = 6.
If type was not one of these values, tests were made for the presence of the weather elements sea surface temperature, air temperature, pressure, and wind (speed and direction). If only sea surface temperature was extant, the ship type was changed to 7; otherwise ship type was left 6 to indicate a research ship.

2.11 Derived Ship Type

Inputs other than in TD-1100 did not have a field for ship type. Data in the Exchange format had no form of ship identification, so ship type became missing. For data in TD-1129M, ship type was set to 6 for Eltanin data, or else it was set to missing. For data in TD-1127 and TD-1129, ship type was set to missing with two exceptions for buoy data: a) '70s Decade or '70s Mislocated Data (source IDs 18 and 23) had ship type set to 5 if deck was 143 or 876-886; b) for source ID 24 ship type was automatically set to 5.

2.12 Past Weather Containing Overpunch

A negative overpunch with a numeric past weather was stripped off and the numeric retained in decks 151, 192, and 899.

2.13 Wind Speed Conversion from Knots to Meters Per Second

Decks 128, 150, 151, 152, 185, and 926 have been identified as cases in which some or all of the original wind speeds were translated from meters per second into whole knots to fit in TD-11. The international convention (1 m s-1 = 1.9438445 knot) was used to convert all decks back to meters per second, regardless of the fact that the U.S. convention (1 m s-1 = 1.94254 knot) was probably used for the reverse conversion in the six decks; this was done because of a lack of complete documentation -- the problem should be fixed later.

2.14 19th Century IMMPC

These were all changed to the corresponding year of the 20th century because manual inspection showed 19th century reports always to be adjacent, with a sharp break, to reports in the 20th century.

2.15 Japanese Wind Direction

These special characters were changed when encountered in the high-order position of the wind direction in the Japanese data:

! changed into 0,
w changed into 1,
u changed into 2,
v changed into 3.
In addition, when one of the three (lower-case only) letters was encountered, 100 was subtracted from the wind speed. (Original wind speeds less than 100 were considered erroneous.)

2.16 South African Minus Sign

Any field in the "regular" section that contained all minus signs (-), and for which "all minus signs" was not a legal value, was made blank.

2.17 Bucket Indicator

Only in TD-1129 was there a bucket indicator in the "regular" section, and only in the Australian set was there a value, 9, for a missing indicator. A blank in this field was interpreted as missing, except in the Australian set, where it was interpreted as intake.

2.18 Australian '70s

This set had WMO-defined quadrant numbers. These were translated into the quadrant system used by NCDC according to the following:

 WMO  NCDC
----------
  3    4  
  5    3  
  7    1  
  1    2  
----------

2.19 Wet Bulb With Ice in the German Exchange Format

At conversion time the sign character (E), which specified wet bulb with ice, was unknown, so that all wet bulb temperatures containing a character other than a blank, plus, or minus were considered erroneous.

2.20 Temperature Indicator

This indicator has a different meaning for source TD-11 data versus source Exchange data. In TD-11, the only legal values correspond to TI = 0, 1, or 2 (0.1°C, 1°C, or 0.5°C). Data converted from 0.1°F, 1°F, or 0.5°F were set at NCDC with TI = 0 to indicate that the tenths position of temperature, after conversion, might be any digit (e.g., not constrained to 0 or 5).

In contrast, the Exchange format has a flag with possible values for original measurement precision of 0.1°F, 0.1°C, 0.5°F, 0.5°C, and 1°F or 1°C. No allowance was made for mixed precision among the different variables or the state of the wet bulb as given by the Netherlands version of the Exchange format. When mixed precision was indicated, or precision of 1°F or 1°C, TI was set to missing.

3. Conversion from LMR into TD-1129(M)

For some recent data (TD-1129 or TD-1129M) this step reversed the conversion into LMR, except that corrections and modifications made at that stage were retained. However, most of the data required rearrangement of fields or other modifications to achieve a more uniform format. These transformations are covered in the background on TD-1129(M) in sec. 1.1. Additional details are given here.

3.1 Bucket Indicator

A missing indicator, and the values for unknown and implied bucket (BI codes 0 and 2) were all translated into 0 in TD-1129.

3.2 Uncertain Duplicates

Only reports with a dup status strictly less than 3 were converted into TD-1129(M), which eliminated all uncertain duplicates as defined in supp. K.

3.3 Undocumented Supplemental Data

Undocumented fields, such as the Atlas "area code" from original positions 137-140, were not blanked out, and as many such characters as would fit were included.

3.4 Erroneous Fields and their Indicators

A non-blank indicator associated with an erroneous field was blanked out, as was the field. However, quality control flags referring to erroneous data were retained.

3.5 Leading Zeros

Numeric values were prefixed by leading zeros where necessary to fill the entire field up.

3.6 Positive Temperatures

These have an explicit plus in the sign position.

3.7 Exchange Source Wind Directions

Because the value in degrees for decks 155 and 156 (or source IDs 3 and 4) may not coincide with any of the compass midpoints chosen for a given direction indicator, as discussed in supp. F, a deviation ± 2° around the values in Table F2-1 was allowed.

3.8 Source ID

Source ID was coded as a single character (A-X) corresponding to the numeric values in use in LMR (1-24), and placed in position 148 of TD-1129(M).

3.9 Special Transfers for 1970-1979 Data Exclusive of the '70s Decade

In order to make the '70s strictly TD-1129 (not TD-1129M), special modifications were required for some source TD-1100 data. Decks 128 and 891, exclusive of the '70s Decade (i.e., not source ID 18), were modified. Tables I3-1 and I3-2 show the respective transfers made for these two decks of data from TD-1100 positions 78-140. Any data not explicitly transferred from positions 78-140 were deleted, resulting in the loss of some supplemental data from TD-1129.


Table I3-1
Position Mapping for '70s Deck 128

   Field                                        TD-1100   TD-1129
-----------------------------------------------------------------
ocean weather station number or country code*   78-79     79-80  

when column 82=1                                                 
   type of ice accretion on ship                83        87     
   thickness of ice on ship                     84-85     88-89  
   rate of ice accretion                        86        90     

when column 82=6                                                 
   ship direction                               83        81     
   ship speed                                   84        82     
   barometric tendency                          85        83     
   amount of pressure change                    86-88     84-86  

when column 82=8                                                 
   significant cloud amount                     83        114    
   significant cloud type                       84        115    
   significant cloud height                     85-86     116-117

ship number                                     90-93     91-94  
original temperature units indicator            98        99     
sea temperature measurement method indicator    99        100    
wind wave period                                100-101   101-102
-----------------------------------------------------------------

* Not transferred if TD-1100 position 81 (ship type) was 2 or 3. 
_____________________                                            



Table I3-2
Position Mapping for '70s Deck 891

Field          TD-1100   TD-1129
--------------------------------
ship number    90-95     91-96  
--------------------------------



3.10 Overlaying of QC Flags in the '70s Decade

Because the '70s Decade data set had been previously quality controlled at NCDC, two sets of QC flags are available in LMR. In order to reconcile the two sets of flags, which are based on differing procedures, the more serious value from each pair of flags (see supp. J) was output. This will help catch those suspect or erroneous fields that NCDC failed to flag, and at the same time retain those flags received during the track check performed at NCDC.

Overlaying of flags was done only for the '70s Decade (source ID 18), not the '70s Mislocated Data (source ID 23). Except for flags R (correct) and S (missing), the flag with the higher alphabetic ranking A through Q was chosen. Flag R was always discarded in favor of any one of A through Q, and S should appear only with missing fields. In the event a new flag stated a field was missing, but the old did not, the new flag was chosen. In order to identify the quality control procedure(s) that produced the resulting flags, three different QC dates were output: a) old date if all old flags, or old and new were the same; b) June '84 if old and new mixture; or c) May '83 if all new.


MAIN | supp. A | supp. B | supp. C | supp. D | supp. E | supp. F | supp. G | supp. H | supp. I | supp. J | supp. K