3.1 Checking the pipeline

3.1.1 Interactive checking

The check_arch command gives a rather comprehensive overview of the current state of the archive pipeline. It is a good idea to run it from time to time to make sure everything is all right. A typical output of this command is shown in figure 3.1.

[IMAGE (figure)]

The Current active archive processes list is the output of the Unix command ps -ax | grep archive and stands as an attempt to gather all the processes related to the archive pipeline. When running, the daemons may spawn child processes that are likely to appear in this list

The Current active archive directories stats the working directory of each daemon.

• #FITS is the number of FITS files currently in the directory (waiting for processing). There should be at most one in /archive/copy, and only a few in /archive/distributor and in the miscellaneous processes directories because the corresponding daemons don't do any blocking on the file numbers. There should be less than the limit number set in the corresponding parfile in the archiver daemons directories.
• du(kB) is the number of kilobytes taken up by the corresponding directory.
• status is the status listed in the .status file (NONE is shown if no .status file is found).
• pid is the pid listed in the .pid file (NONE is shown if no .pid file is found).
• in ps? is yes or no depending on whether the corresponding pid appears in the list of the current running process.
• to be processed shows the name of the first and the last of the #FITS files waiting to be processed.
• directory is the actual path to the daemon's directory.

The Current active list files list stats the list files of the daemons when applicable. The frame number in the last line (ie the last file processed), the time of the last update and the number of lines (ie the number of files processed) are listed. It is a good idea from time to time to check that for each archiver, the sum of #lines and #FITS from the previous list is a constant.

The Current active log files stats the logfile of each daemon. The time of the last update and the number of lines are listed. It is a good idea to check that the logfiles have been recently updated.

Then the df of each disk partition on the archive machine is listed. It is a good idea to check that the partitions don't become overfull.

Finally, the number of files in the summit directory and the corresponding disk use are listed.

3.1.2 Crontab job checking

Every hour the archive_supervisor command is spawned by the archive user crontab daemon of the archive machine. Currently, this command does the following:

• For each daemon:
  • Check the working directory for .status and .pid files;
  • Check if the pid number in .pid is indeed in the ps list;
  • Check if the logfile has been updated during the last hour to make sure that the daemon is not hung.
• For each disk partition on the archive machine, check if it is less than 95%full.

The archive_supervisor command may be run interactively but its output is much less verbose than the one of check_arch.