Next: 4.2 distd daemon Up: 4 Archive Pipeline: detailed Previous: 4 Archive Pipeline: detailed

4.1 copyd daemon

The copy daemon copyd makes sure that every file in the remote shadow directory makani: /users/observer/shadow is transferred to the distributor directory (currently /archive/distributor). The current parfile copyd.par listed below is self-explanatory.

#
# copyd.par - copy daemon parfile
#

# copyd handle several remote host/dir. They are given on one line,
# separated by spaces. The corresponding dirs and hosts must be given
# in the same order.

# Remote directories holding the files to be copied
remote_dirs = "/users/observer/shadow /data/uh8k/shadow"

# Remote hosts holding the remote directories
remote_hosts = "neptune makani"

# Local temporary directory for holding the files while being copied down
daemon_dir = "$ARCHDIR/copy"

# Distributor directory: once copied down, data are moved there
# WARNING: dist_dir and copy_dir MUST be on the same file system
dist_dir = "$ARCHDIR/distributor"

# INSTRUME card of a UH-8K fits file (file to be compressed)
uh8k_card = "UH8K Mosaic Camera"

# Name of each archiver's working directory
arch_dir = "$ARCHDIR/tarexa2 $ARCHDIR/tarexa"

# Name of the file to contain the list of the saved files
saved_list = "$ARCHDIR/sw/list/saved_list"

# "none" = no compression at all
# "compfits" = compression of UH-8K files using compfits
compress_mode = "compfits"

# Minimum free disk space (kB) required for the copy to proceed
# Must be high enough so as all the archivers may operate safely at the same time
# NB : each archiver copies on tape when it has 100 files or 200 Mb of data. It re-reads
# the tape after writing => at least 200MB (original files : hard links) + 2 x 200 Mb
# (the 2 .verify directories, full in the same time in the worst case) = 600 Mb
# The limit 200 Mb can be a little bit over taken (by a file size <= 20 Mb) => safer to
# take disk_limit = 220 Mb x 3 = 660000 Kb
disk_limit = 660000

# Name of asteroids files COPY directory
ast_dir = "/data/redn/veillet/uh8k/compress"

# Minimum free disk space (Kb) required on $ast_dir for the copy to proceed
# 2 Gb (This disk is not yet (May 97) dedicated to this program : shared with others)
ast_limit = "2000000"

# Mail adress of people concerned by the asteroids directory
ast_mail = "veillet,link"

# Mail adress of the archive managers
arch_mail = "archive"

# Number of seconds copyd is allowed to sleep if no files are coming
sleep_time = 400

Files are listed and copied over the network through the TCP/IP protocol (rsh and rcp commands) which allows a reliable transfer without the need of a remote NFS mount. copyd working directory is used as a temporary space where the files are fully copied before being transfered to distd working directory. Once copied, every file is moved from the shadow directory to /users/observer/shadow/.copied and once archived on the Exabyte tapes the files are removed from this directory. Any file whose name doesn't match a regular FITS name (such as sequence file for a combined exposure) is deleted without being copied.

Before copying, copyd checks the disk space available on Waimea disk against a limit defined in the parfile. How this limit is set is explained in 5.2.
N.B.: Sometimes, the df command fails for unknown reasons. The code has been modified to attempt a second df before giving up.

archive@cfht.hawaii.edu