=============================================================================== International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.1 Release 1a: 1980-92 (Earth System Monitor article) 27 February 2004 ==================================================================== This document is based on the article "Comprehensive Ocean-Atmosphere Data Set (COADS) Release 1a: 1980-92" appearing in the Earth System Monitor, Vol. 4, No. 1, September 1993, with minor changes to conform to the e-doc standard format (e.g., "sigma" replaces the corresponding Greek letter). A note has been added concerning the doubled wind speed problem in {Data sources and corrections}. Figures corresponding to the figure captions appearing in this text are found in a compressed postscript file. Document Revision Information (previous version: 9 September 2002): Updates for Release 2.1 and ICOADS. ------------------------------------------------------------------------------- Scott D. Woodruff and Sandra J. Lubker, Climate Monitoring and Diagnostics Laboratory, NOAA/ERL; Klaus Wolter, Cooperative Institute for Research in Environmental Sciences; Steven J. Worley, National Center for Atmospheric Research; and Joe D. Elms, National Climatic Data Center, NOAA/NESDIS NOAA/ERL R/E/CG2, 325 Broadway, Boulder, CO 80303; CIRES, Campus Box 449, University of Colorado, Boulder, CO 80309; NCAR, P.O. Box 3000, Boulder, CO 80307; NOAA/NESDIS/NCDC E/CC22, Federal Building, Asheville, NC 28801 Weather observations taken by merchant mariners form one of the longest continuous climate records in existence. Digitized data are currently available starting about 1854, the year following an international meeting held to discuss standardizing observing practices among the maritime nations of the world. These surface marine data, supplemented in recent years by in situ measurements from increasing numbers of data buoys and other automated platforms, remain critically important as a baseline data set for comparison with remotely sensed data, for input to climate models and global reanalysis efforts, and for a broad range of climate diagnostic studies. The Comprehensive Ocean-Atmosphere Data Set (COADS) is the most extensive collection of surface marine data available for the world ocean over the past century and a half. In COADS processing, the basic observational data have been edited, using a "trimming" procedure to identify outliers with respect to climatological 3.5 sigma limits derived from data for three periods (1854-1909, 1910-49, 1950-79). A variety of observed and derived variables has then been summarized for each month of each year and decade of the period of record, currently 1854-1992, using 2-degree latitude x 2-degree longitude boxes (and using the 1950-79 limits for trimming after 1979). Resulting observational or summary statistical products are now used by over 200 research groups worldwide, but vigorous efforts continue to update and improve the available products. Until recently, COADS Release 1 (Slutz et al., 1985; Woodruff et al., 1987), covering 1854-1979, had been extended for 1980-1991 by a set of "interim" products. However, the interim products were constructed using simplified procedures (e.g., for elimination of duplicate reports) that did not yield the full set of Release 1 statistics. Annual updates of the interim products (for 1985-91) had the additional limitation of relying primarily on the Global Telecommunication System (GTS) for foreign ship reports because receipt of international ship logbook data is often delayed by 2-5 years (Jenne, 1992). Moreover, concerns about the trimming performance during extreme climate anomalies such as the 1982-83 El Nino/Southern Oscillation event (ENSO) (Wolter, 1992) prompted us to reevaluate the quality control procedures employed in COADS. COADS Release 1a, offering a set of extensive and easily used products for 1980-92, has now been completed to address many of these shortcomings for that period. To provide consistency with Release 1, as well as the broadest possible coverage and improved quality control, we have created two alternative sets of 2-degree monthly summaries for 1980-92. These products and other aspects of Release 1a are discussed in the following sections. {Data sources and corrections} Overall, COADS Release 1a offers a substantial augmentation in weekly numbers of reports in comparison to the interim data (Fig. 1A). When the comparison is restricted to ship data, inclusion of delayed international logbook reports is shown to yield a large benefit with the passage of time (Fig. 1B). In addition, the logbook reports contain important information, such as the sea surface temperature (SST) measurement method indicator, not currently available in GTS reports. Besides the available international logbook data, records from the Russian Marine Meteorological Data Set have been included, as well as special fishing fleet data that helped fill important gaps in data coverage in the tropical Pacific. Automated platform data have been similarly improved and expanded by including three important sets of delayed, quality-controlled data: -- Global drifting buoy data prepared by Canada's Marine Environmental Data Service (MEDS). For 1992 it should be noted that the MEDS data set includes proprietary World Ocean Circulation Experiment (WOCE) buoys from which only the 2-degree monthly summaries are available to the general research community, not the quality controlled individual observations. -- Hourly moored buoy and Coastal-Marine Automated Network (C-MAN) data from the NOAA National Data Buoy Center (NDBC). -- Data from NOAA's Pacific Marine Environmental Laboratory (PMEL): (a) Tropical Ocean-Global Atmosphere (TOGA) Program TAO ATLAS moored buoys, and (b) daily averages from Equatorial Pacific Ocean Climate Studies (EPOCS) moored buoys and low-elevation island stations. As part of Release 1a, a number of important corrections and archival improvements has been implemented in the basic GTS data used for the interim products and also in the delayed ship logbook reports. For example, data from NOAA's National Meteorological Center (NMC), which form the bulk of the basic GTS input for Release 1a, were reprocessed at the National Climatic Data Center (NCDC) to correct a number of errors in conversion from the original NMC format associated with code changes and insufficient documentation, e.g., doubled wind speeds from ships reporting meters per second during the approximate period February-June 1984. [NOTE: The doubled wind problem applies only to prelim- inary sets of interim products obtained before October 1987; see .] Similarly, U.K. logbook reports containing erroneous SST measurement method indicators were replaced from update tapes. Another example of a problem in the logbook data, extending through 1988, is a displacement of French reports by 10 degrees of longitude within 90E-90W across the dateline, due to a WMO code ambiguity. Replacement data have been received from France; however, because of questions about the exact duration of the problem, it was possible for Release 1a only to delete the logbook reports under the assumption that they would be replaced in many cases by properly located GTS data. Several comparatively minor archival problems were also left partially corrected or uncorrected in other data sources, awaiting future updates. Electronically available documentation for COADS products (see box) includes additional information about data problems and corrections. {Improvements in processing and products} As a first processing step, individual marine reports for Release 1a were translated into an expanded version of the variable-length binary format for Long Marine Reports (LMRs) used for Release 1 (1854-1979). In contrast to 1980-91 interim processing, an expansion of the duplicate elimination algorithm for Release 1 was used to eliminate duplicate LMRs, including improved identification of exact matches of duplicate reports in time/space location and ID (e.g., ship call sign or buoy number). The resulting Release 1a output totals about 41 million LMRs for 1980-92 (3.8 gigabytes), of which hourly NDBC data make up 19%. For Release 1a, a fixed-length version of the LMR format is also available, which contains all regularly observed weather elements and ID information, and has a flag set whenever feasible to indicate platform type (ship, drifting buoy, moored buoy, etc.). This represents an improvement over the Compressed Marine Report (CMR) format used for the interim products, which lacked some basic meteorological elements (e.g., wave fields) and ID information, and contained much less information regarding platform type. The requirements of many COADS users are better met by 2-degree monthly summary products rather than individual observations. For Release 1, 14 statistics, such as the mean and the median of data falling within a given year-month- 2-degree box, were computed for each of 19 observed and derived variables, such as SST and heat-transport parameters. In contrast, although the interim products included all 19 variables, only the mean and the number of observations were recorded for each variable. For Release 1a, we have extended the full matrix of 19 variables x 14 statistics used for Release 1 through 1992. However, because the full matrix is voluminous even for the 13-year update period (~0.5 gigabyte), selected statistics are most commonly distributed in the form of "group" files (~50 megabytes/group) (Table 1). Two separate sets of 2-degree monthly summaries were calculated for Release 1a: The "standard" set provides closest compatibility with Release 1, by retaining the original (1950-79) 3.5 sigma limits, and by restriction to ship data as nearly as practical. In contrast, the alternative set of "enhanced" statistics includes ships plus other in situ platforms, and employs wider 4.5 sigma limits to accommodate more extreme climate events. However, all NDBC C-MAN (near shore) data and non-3-hourly (i.e., not 0000, 0300, ..., 2100 UTC) moored buoy data, plus selected variables from other platform types, such as experimental wind observations from drifting buoys, were omitted from the enhanced statistics because of concerns about systematic data differences and over-sampling. Table 2 gives a summary of the composition of the resulting standard and enhanced statistics, as compared with that of the interim statistics. {Discussion} When comparing Release 1a with interim data, the large amount of delayed ship reports (ref. Fig. 1B) provides the largest increase in global 2-degree box coverage (illustrated for SST in Fig. 2). Globally, the number of year-month-2-degree boxes during 1980-91 containing data in the standard statistics increases by about 4% in comparison to the interim data. However, the more heterogeneous platform mixture used for both the enhanced and interim statistics complicates comparison with the ship-based standard statistics (see Table 2). Variables commonly measured by drifting buoys, such as SST and sea level pressure, gain up to an additional 3% coverage in the enhanced statistics (e.g., Fig. 2). This increase combines the effects of inclusion of more delayed data from automated platforms and of wider trimming limits. By itself, relaxation of the trimming limits to 4.5 sigma appears to have had a relatively small effect on overall data density and coverage (on the order of 1%) in the enhanced set. However, testing of the 4.5 sigma limits on subsets of Release 1a data, which did not consider effects of changes in platform mixture, showed important smaller-scale impacts: -- For particularly large climate extremes such as the 1982-83 ENSO, local increases over 10% in the number of observations were possible for varying time intervals. -- The number of 2-degree boxes with any observation was increased in areas with low observational density. The net effect was to reduce large gaps and spotty coverage (polar ice-edges, extratropical southern hemisphere) while well-observed regions (northern midlatitudes) did not gain additional coverage. -- The size of recorded climate anomalies is less restricted by the 4.5 sigma versus the 3.5 sigma limits, thus overall climate variability is enlarged. For instance, during the 1982-83 ENSO, peak SST anomalies in the eastern equatorial Pacific are up to 1 degree C higher. Preliminary comparisons between mean fields from the standard and enhanced statistics show compelling evidence that the combination of ship and buoy data is subject to systematic differences in boxes dominated by observations from one platform type or another (see, e.g., Wilkerson and Earle, 1990; Woodruff et al., 1991). Figure 3 illustates this for wind speed, and also shows frequent increases in wind speed in high latitudes stemming from the relaxed trimming limits. {Future plans} Two major COADS updates beyond Release 1a are planned over the next few years. Release 1b, a partial update for the period since about 1947, is now planned for completion during 1994. The period for Release 1b was chosen primarily on the basis of requirements for data inputs to global reanalysis (Jenne, 1992). Release 2 will involve a consistently processed update for the total period of record, incorporating a variety of new data sources as a result of international cooperation with countries such as Canada, China, France, Germany, Norway, U.K., and Russia. Release 2 is planned to include the surprisingly small amount of U.S. merchant marine data that was apparently preserved for the data sparse periods spanning the two World Wars, which is being digitized by NCDC. The period of record might even be extended before 1854 if sufficient data can be digitized (Elms, 1992). For Release 2, substantial improvements are also planned in quality control and product availability to better meet the needs of researchers (Diaz et al., 1992, pp. 377-383). For example, research continues toward an improved trimming procedure, which may include removal of outliers with reference to a local year-month mean in addition to climatological information. Furthermore, we are considering separations of statistics by time-of-day and platform type for selected variables, to address possible diurnal biases and biases between buoy and ship data. Availability of Release 2 is anticipated in the mid-1990s. However, that schedule is dependent on the difficulty of resolving these processing and quality control issues, including availability of adequate resources to complete the task. As a side benefit of the COADS project, the integration and comparison of many different data sources have revealed numerous data and archival problems that strongly indicate the need for improved interactions among different groups within NOAA involved with marine data, as well as among other organizations both nationally and internationally. To help strengthen interaction and provide adequate attention to issues of data continuity, we have recommended that a working group of marine data experts be established to serve as a focal point for facilitating and improving coordination with the climate research community. {Acknowledgements} COADS is the result of a continuing cooperative project between the National Oceanic and Atmospheric Administration (NOAA)--specifically its Environmental Research Laboratories (ERL), National Climatic Data Center (NCDC), and Cooperative Institute for Research in Environmental Sciences (CIRES, conducted jointly with the University of Colorado)--and the National Science Foundation's National Center for Atmospheric Research (NCAR). The NOAA portion of COADS is currently supported by the NOAA Climate and Global Change Program and the NOAA Environmental Services Data and Information Management (ESDIM) Program. {References} Diaz, H.F., K. Wolter, and S.D. Woodruff (Eds.), 1992: Proceedings of the International COADS Workshop, Boulder, Colorado, 13-15 January 1992. NOAA Environmental Research Laboratories, Climate Research Division, Boulder, Colo., 390 pp. Elms, J.D., 1992: Status of NCDC keying of historical marine data. Proceedings of the International COADS Workshop, Boulder, Colorado, 13-15 January 1992. H.F. Diaz, K. Wolter, and S.D. Woodruff, Eds., NOAA Environmental Research Laboratories, Boulder, Colo., 37-45. Jenne, R.L., 1992: The importance of COADS for Global Reanalysis. Proceedings of the International COADS Workshop, Boulder, Colorado, 13-15 January 1992. H.F. Diaz, K. Wolter, and S.D. Woodruff, Eds., NOAA Environmental Research Laboratories, Boulder, Colo., 9-15. Slutz, R.J., S.J. Lubker, J.D. Hiscox, S.D. Woodruff, R.L. Jenne, D.H. Joseph, P.M. Steurer, and J.D. Elms, 1985: Comprehensive Ocean-Atmosphere Data Set; Release 1. NOAA Environmental Research Laboratories, Climate Research Program, Boulder, Colo., 268 pp. Wilkerson, J.C. and M.D. Earle, 1990: A study of differences between environmental reports by ships in the Voluntary Observing Program and measurements from NOAA buoys. J. Geophys. Res., 95, 3373-3385. Wolter, K., 1992: Sifting out erroneous observations in COADS--the trimming problem. Proceedings of the International COADS Workshop, Boulder, Colorado, 13-15 January 1992. H.F. Diaz, K. Wolter, and S.D. Woodruff, Eds., NOAA Environmental Research Laboratories, Boulder, Colo., 91-101. Woodruff, S.D., R.J. Slutz, R.L. Jenne, and P.M. Steurer, 1987: A comprehensive ocean-atmosphere data set. Bull. Amer. Meteor. Soc., 68, 1239-1250. Woodruff, S.D., S.J. Lubker, R.G. Quayle, U. Radok, and E.D. Doggett, 1991: Differences Within and Among Surface Marine Datasets. NOAA Environmental Research Laboratories, Climate Research Division, Boulder, Colo., 216 pp. Table 1. Structure of group files 3-8 (groups 1 and 2 are reserved for "untrimmed" Release 1 products). Each group contains four variables and eight statistics for each variable: the median, the mean, the number of observations, a standard deviation estimate, the mean day-of-month of observations, the fraction of observations in daylight, and the mean longitude and latitude of observations. Derived variables in groups 5-8 are computed as indicated from individual observations of other variables, e.g., the wind-stress parameter "X" is the product of W and U. In addition, QS denotes saturation Q at SST. ------------------------------------------------------------------------------- Group 3 4 5 6 7 8 =============================================================================== Sea surface temp. (S) Scalar wind (W) Total cloud. (C) D=S - A I=UA S Air temperature (A) Wind U-comp. R E=(S - A)W J=VA A Specific humidity (Q) Wind V-comp. X=WU F=QS - Q K=UQ X Relative humidity (R) Pressure (P) Y=WV G=FW L=VQ Y ------------------------------------------------------------------------------- Table 2. Composition of the enhanced (4.5 sigma trimming) and standard (3.5 sigma) Release 1a statistics, and of the 1980-91 interim (3.5 sigma) statistics by platform type and variable (for observed variables only with abbreviations as given in Table 1, except that here W and R refer to wind and psychrometric data in general). For a given platform type, "x" indicates that no data were used, and "-" indicates data not observed, with some experimental exceptions, e.g., NDBC moored buoy and Coastal-Marine Automated Network (C-MAN) humidity observations. ------------------------------------------------------------------------------- Enhanced Standard Interim Platform type S A W P R C S A W P R C S A W P R C =============================================================================== Ships (& Ocean Wea. Stat.) S A W P R C S A W P R C S A W P R C Drifting buoys* S A x P - - x x x x - - S A W P - - Moored buoys** S A W P - - x x x x - - S A W P - - EPOCS daily buoys/islands S A W P - - x x x x - - x x x x - - Fishing fleet data S - x - - C x - x - - x x - x - - x Rigs and platforms S A W P R C x x x x x x S A W P R C C-MAN x x x x - - x x x x - - S A W P - - ------------------------------------------------------------------------------- * Note that the enhanced statistics include some drifting buoy data that passed the MEDS quality control as acceptable but that occurred in 2-degree boxes for which 1950-1979 trimming limits could not be computed. ** Includes NDBC buoys (reduced from hourly to 3-hourly for 1980-92 in the enhanced statistics, and for 1985-91 in the interim statistics), PMEL TOGA/TAO ATLAS buoys, and foreign moored buoys. ---------- =============================================================================== | {COADS Release 1a ELECTRONIC METADATA} | | | | More detailed information about COADS can be obtained in electronic form | | over the Internet. The information is currently available on a computer | | at the National Center for Atmospheric Research (NCAR) that is publicly | | accessible using an anonymous FTP login. | | | | After connecting to the NCAR machine (ncardata.ucar.edu or IP address: | | 128.117.8.111) change directories to pub/COADS. This directory will | | contain introductory information in a README file, and other sub- | | directories containing documentation applicable to the different COADS | | releases or the interim products. It is anticipated that the electronic | | metadata contained in this directory will be periodically updated. | =============================================================================== =============================================================================== | {COADS Release 1a DATA AVAILABILITY} | | | | COADS products for Release 1 (1854-1979) and Release 1a (1980-92) are | | available from: | | Data Support Section | | National Center for Atmospheric Research | | P.O. Box 3000 | | Boulder, CO 80307 | | USA | | or contact Steve Worley at 303-497-1248 (worley@ncar.ucar.edu) or Dennis | | Joseph at 303-497-1216 (joseph@ncar.ucar.edu). | | | | Fortran software is available to help read packed binary products. In | | addition, Release 1 individual marine reports in an ASCII format can be | | obtained from NCDC. | =============================================================================== Figure 1. (A) Weekly numbers of reports for Release 1a (solid) versus the interim products (dashed). The dotted curve shows 1985-92 Release 1a counts adjusted for better comparability with the interim counts by reduction of NDBC data to 3-hourly values (hourly NDBC data were included in the interim products for 1980-84). Sharp downward spikes in November 1986, especially in the interim data, represent a gap in the NMC data archived at NCDC; it is hoped that it will be possible to eventually fill this and similar gaps with data from NCAR or other institutions. (B) As for Figure 1A, except weekly numbers of ship reports for Release 1a (solid) versus the interim. Prior to 1984 in the interim set, data from ships could not be fully differentiated from data from other platform types. Figure 2. Global 2-degree box coverage for SST observations in January 1983. Light gray shading denotes 2-degree boxes covered by the interim, standard, and enhanced sets, medium gray refers to the standard and enhanced sets only, and black stands for coverage unique to the enhanced set. Figure 3. Annual average of 1980-92 monthly average differences between the Release 1a enhanced minus standard mean of scalar wind (meters per second). In many cases, negative differences (> -2 m/s) in 2-degree boxes around the U.S. coastline and across the equatorial tropical Pacific correspond to NDBC and PMEL moored buoy locations. Positive differences (< 5 m/s, but rarely above 2 m/s) arise from relaxation of the trimming limits to 4.5 sigma.