Introduction

The program EXDEC takes care of the extraction and decoding of the weather reports needed for the construction of the Actual Meteorological Database (AMD). The input for the EXDEC program consists of a file named GTS:Iyymmdd.DAT (yy=year, mm=month, dd=day). This file contains the MAIN-hour and INTERMEDIATE-hour SYNOP (FM 12-VIII) and METAR (FM 15-VIII) reports of all countries of interest of one day, received through the GTS (Global Telecommunication System). In this chapter the tasks performed by the EXDEC program are described.

Bulletin-checking

Before extracting the weather observations from the file GTS:I yymmdd .DAT the EXDEC program checks the bulletins on specific errors and removes these errors. Note that this is not a check on the correctness of the observations (see chapter 3), but a check on the correctness of the bulletin construction.

First a bulletin (regardless whether it is a SYNOP or METAR) is checked on:

• the presence of empty lines , which are removed from the bulletin;
• the presence of the NNNN sequence at the end of the bulletin, which is removed from the bulletin;
• the presence of lines which only contain an =-sign . The =-sign is moved to the end of the previous line and the empty line thus created is removed from the bulletin;
• the presence of single or multiple spaces at the beginning of a line, which are removed from the bulletin.

After this step the program checks whether the bulletin is a SYNOP-report or a METAR-report, and SYNOP-checking or METAR-checking is performed.

SYNOP-checking

The SYNOP-report is checked consecutively on:

• the presence and correctness of the AAXX identification, which is added when missing or modified when wrong;
• the presence of the sequence 100EEE , which is removed;
• the presence of alphabetical characters (except AAXX and NIL), which are converted to their telex number equivalent;
• the presence of commas , which are replaced by spaces;
• the presence of strange characters in front or at the end of a group which does not consist of 5 digits. These characters are replaced by spaces;
• mutilation of the 333 identification group (e.g. '3 3 3' or ' 33 '), which is corrected;
• the presence of single characters , which are replaced by a single space;
• the presence of multiple spaces , which are reduced to a single space;
• the presence of groups of 8, 10 and 11 characters (i.e. an identification group attached to a normal group, two normal groups attached to each other or two normal groups not separated by a space), which are separated into two groups;
• the presence of a double =-sign at the end of a line, which is reduced to a single one.

After this step the corrected SYNOP-report is ready for the actual data extraction.

METAR-checking

Since mistakes in a METAR-report are very rare, the METAR-report is only checked on:

• the presence of multiple spaces, which are reduced to a single space.

After this step the corrected METAR-report is ready for the data extraction.

Data extraction

The objective of the data extraction process is to select only those observations from the bulletins which are of importance for the construction of the Actual Meteorological Database. In Appendix A a list of meteorological stations is given. Each station is marked with an F or an S which denotes its status concerning the construction of the database. Observations of stations marked with:

• F: are not decoded because these stations are not located in the area of interest;
• S: are decoded and will become part of the AMD if on the day under consideration: - at least the 6 and 18 UTC bulletins are provided;
  • at least values for air temperature, dew-point temperature, wind speed, total cloud cover, pressure and precipitation are reported.

The result of the data extraction process is that the coded observations of the stations marked with S extracted from the file GTS:Iyymmdd.DAT are added to an intermediate database named CODED. This database consists of three files: an index file (REALDATA:CODED.IDX), a data file (REALDATA:CODED.BIN) and a file with a pointer to the last written datarecord (REALDATA:CODED.PTR). In CODED the extracted MAIN-hour and INTERMEDIATE-hour observations of the past seven days are stored. The data extraction process follows the same steps for SYNOP-reports and METAR-reports, except for a date-time check on SYNOP-reports: reports older than three days are skipped. The date-time check is not possible for METAR-reports because these reports do not contain the date (only time is included). From the bulletin one line is read, which in principle contains the observations of one station. The program checks if the line contains a complete observation, which is assumed to be the case if one of the following criteria is met:

• the line is the last line in the bulletin
• the line ends with an =-sign
• the line contains the string NIL (only for SYNOP-reports)
• the next line contains the string NIL (only for SYNOP-reports)
• the line and the next line contain together more than 132 characters in a METAR-report and more than 172 characters in a SYNOP-report
• the next line starts with a station number which is legal in this bulletin and the fourth group (the temperature group) starts with '1' (only for SYNOP-reports)

If the observation is complete and also proves to be a real observation (i.e. the line does not contain NIL) the station identification is checked. The station identification must:

• have an integer value in a SYNOP-report or a code in a METAR-report
• be a legal station identification
• be permitted to occur in a bulletin from this country
• not be denoted with an F in the station list (Appendix A)
• not yet be decoded with the same coded observation, or in case of a METAR-report, no SYNOP-report of the same station is decoded already

Only if all foregoing criteria are met the observation is stored in the database CODED and submitted to the decoding procedure.

Data decoding

The observations which are stored in the database CODED are also submitted to the decoding process. After this process the decoded observations are stored in the intermediate database DECODED. This database has the same structure as the database CODED (index file: REALDATA:DECODED.IDX, data file: REALDATA:DECODED.BIN, pointer file: DECODED.PTR) and contains the extracted decoded observations of the past seven days.

SYNOP-decoding

Before decoding a SYNOP-observation the EXDEC program tries to find and correct more errors in the coded observation. The correct structure of an SYNOP-observation is given in Fig. 2.1.

The observation is checked consecutively on:

• the presence of identical groups after the Nddff group. If two consecutive groups are equal one group is deleted;
• a missing I_rI_xhVV or Nddff group. If 1s_nTTT is the third group, 2s_nT_dT_dT_d the fourth group and 3P_oP_oP_oP_o or 4PPPP the fifth group a dummy group '///// ' or two dummy groups '///// ///// ' are inserted;
• the presence of no more than three groups before the 1s_nTTT group. If 1s_nTTT is the fifth group, 2s_nT_dT_dT_d the sixth group and 3P_oP_oP_oP_o or 4PPPP the seventh group there is one group to many. The double group is removed, and if no double group is found the fourth group is removed;
• the presence of a '1' as first character of the fourth group (1s_nTTT). If no '1' is present and the fifth group is 2s_nT_dT_dT_d a '1' is added in front of the fourth group (when it consists of four characters) or its first character is modified into '1' (when it consists of five characters);
• the presence of a '2' as first character of the fifth group (2s_nT_dT_dT_d ). If no '2' is present and the fourth group is 1s_nTTT and the sixth group is 3P_oP_oP_oP_o or 4PPPP a '2' added in front of the fifth group or its first character is modified into '2';
• the presence of a '3' as first character of the sixth group (3P_oP_oP_oP_o ). If no '3' is present and the fourth group is 1s_nTTT, the fifth group is 2s_nT_dT_dT_d and the seventh group is 4PPPP a '3' is added in front of the fifth group or its first character is modified into '3';
• the correctness of the sequence of the groups in section 1 . A wrong sequence is modified whenever there is no doubt about the mistake, otherwise the groups at wrong positions are ignored;
• the presence and position of the 333 identification group . If this group is missing it is inserted, and if it is at a wrong position it is moved to the correct position;
• the presence of a 1s_nT_xT_xT_x group or a 2s_nT_nT_nT_n group in section 3 of the 6 UTC and 18 UTC bulletin and the correctness of the identifiers of these groups. A '1' as identifier in the 6 UTC bulletin is modified into a '2' when the next group does not start with a '2' (some countries provide both 1s_nT_xT_xT_x and 2s_nT_nT_nT_n in the 6 UTC bulletin). A '2' as identifier in the 18 UTC bulletin is always modified into a '1'.

After this enhancement procedure the observation is decoded. Decoded values are obtained for the elements listed in table II.1. Derived values are computed from observed values using the equations given in Appendix B. All values are stored in the database DECODED.

METAR-decoding

The structure of a METAR-observation is given in Fig. 2.2. The observation is directly subjected to the decoding procedure, in which decoded values are obtained for the elements listed in table II.2. Derived values are computed from observed values as described in Appendix B. All values are stored in the database DECODED.

Messages

Messages generated by the EXDEC program are written to a log file. In table II.4 the possible messages are described. Also, the module which performs the OBSERVATION-checking writes the original observation line and the changed line to the log file if changes are made.