CFHT archive manual - processes 

Archive Processes

The archive system generally consists of a pipeline of daemons which pass data to each other, performing a process at each step. Each daemon employs handlers which make the various software and hardware interactions generic. The daemons are modular and the pipeline extendable so new processes and pipelines can be added or removed rapidly and without interfering with the existing data path.

The current daemons are:

  • Copyd
  • Distd
  • Tarexad
  • Stripheadd
  • Aiqed
  • Dbd

    • The Copy Daemon

    Copyd is responsible for intelligently moving files from computers at the summit to the archive host. It can check for files on any number of hosts of any platform and copy files using a number of methods. When copying cfh12k mosaics it concatenates the individual files into a singe MEF (Multi Extension Fits) format as it copies the data. When copying Megacam data it splices the per amplifier mosaics into per chip mosaics.

    Parfile: /h/archive/sw/par/copy2d.par
    Working directory: /data/kapu/copy2d
    Associated utilities: mef runon

    The Distributor Daemon

    Distd receives data from copyd and distributes it to the tape daemons. It also handles the communication to and from the media-writing daemons when one reaches the end of it's media. The distributor empties the archiving daemon's working directories and gives a signal for the daemon to finish up once it is done writing the current set. Once both daemons have changed their media, distd repopulates their working directories with the files that it cleaned up thus ensuring that each piece of archive media begins with the same file.

    Parfile: /h/archive/sw/par/dist2d
    Working directory: /data/kapu/dist2d
    Associated utilites: change_auto_media

    The Header Daemon

    Stripheadd also gets data from the copy daemon and extracts information about each file from its fits header. It catalogs a complete copy of the primary header internally as well as on the ftp site. It also manages various lists that are used by other processes and people. It maintains a sybase database which contains basic information about each file and it manages the copy of each file to the distributed storage system. It takes a list of the available filesystems and evaluates which one has the most free space. It then copies the file to this storage area and updates the database with the current location and creates a symbolic link to the file in the virtual filesystem of archive data in /h/archive/current.

    Parfile: /h/archive/sw/par/stripheadd.par
    Working directory: /data/kapu/stripheadd
    Associated utilities: sql_update extracth log_sheet_line.cfh12k.tcl log_sheet_line 

    The Image Quality Daemon

    Aiqued scans files for various features and maintains lists which are its output.

    Parfile: /h/archive/se/par/aiqued.par
    Working directory: /data/kapu/aiqued
    Associated utilities: IRAF

    The Tar Daemon

    Tarexad's name is actually a little confusing. Tarexa initially referred to "tar exabyte" though it no longer writes exabyte tapes and it is no longer necessary that it write data in tar format. At any rate, this daemon is responsible for writing the archive data to media in a sensible format and for re-reading the data and checking it for errors. It also updates the sybase database with information about the location of each file on tape for retrieval. Tarexa makes extensive use of handlers to keep its function as administrative and device independent as possible. In order to produce two copies of archive media, two copies of tarexad are concurrently run, they communicate to each other via distd to keep their tapes identical.

    parfile: /h/archive/sw/par/tarexa3d /h/archive/sw/par/tarexa4d
    Working directory: /data/kapu/tarexa3 /data/kapu/tarexa4
    Associated utilities: tarexa3h tarexa4h tarexa_init sql_update

    The Database Daemon

    This daemon is still part of the pipeline though it has been replaced by calls to the database handler in both stripheadd and tarexad.
    Parfile: /h/archive/sw/par/dbd.par
    Working directory: /data/kapu/dbd
    Kanoa

    Last modified: Tue Jul 17 14:36:21 HST 2001