CFHT archive manual - processes

Archive Processes

The archive system generally consists of a pipeline of daemons that pass data to each other performing some process at each step. Each daemon employs hadlers which make the various software and hardware interactions generic. The daemons are modualar and the pipeline extensible so new processes and pipelines can be added or removed rapidly and without interfering with the existing data path.

The current daemons are:
  • Copyd
  • Distd
  • Tarexad
  • Stripheadd
  • Aiqed
  • dbd

    • The Copy Daemon

    Copyd is responsible for intelligently moving files from computers at the summit to the archive host. It can check for files on any number of hosts of any platform and cpoy files using a number of methods. When copying cfh12k mosaics it concatenates the individual files into a singe MEF (Multi Extension Fits) format as it copies the data.

    Parfile: /h/archive/sw/par/copy2d.par
    Working directory: /data/kapu/copy2d
    Associated utilites: mef runon

    The Distributor Daemon

    Distd recieves data from copyd and distributes it to the remaining daemons by linking the files in each daemon's working directory. It also handles the communication to and from the media writing daemons when one reached the end of it's media. The distributor empties the archiving daemon's working directories and gives a signal for the daemon to finish up once it is done writing the current set. Once both daemons have changed their media distd repopultes their working directories with the files that it cleaned up thereby ensuring that each piece of archive media begins with the same file.

    Parfile: /h/archive/sw/par/dist2d
    Working directory: /data/kapu/dist2d
    Associated utilites: change_auto_media

    The Header Daemon

    Stripheadd extracts information about each file from it's fits header. It catalogs a complete copy of the primary header internally and on the ftp site. It also manages various lists that are used by other processes and people. It maintains a sybase database which contains basic information about each file and it manages the copy of each file to the distributed storage system. It takes a list of the available filesystems and evaluates which one has the most free space. It then copies the file to this storage area and updates the database with the current location and creates a symbolic link to the file in the virtual filesystem of archive data in /h/archive/current.

    Parfile: /h/archive/sw/par/stripheadd.par
    Working directory: /data/kapu/stripheadd
    Assosciated utilites: sql_update extracth log_sheet_line.cfh12k.tcl log_sheet_line

    The Image Quality Daemon

    Aiqued scans files for various features and maintains lists which are its output.

    Parfile: /h/archive/se/par/aiqued.par
    Working directory: /data/kapu/aiqued
    Assosciated utilites: IRAF

    The Tar Daemon

    Tarexad's name is actually a little confusing. Tarexa initially referred to "tar exabyte" though it no longer writes exabyte tapes and it is no longer necessary that it write data in tar format. At any rate, this daemon is responsible for writing the archive data to media in a sensible format and for re-reading the data and checking it for errors. It also updates the sybase database with information about the location of each file on tape for retrieval. Tarexa makes extenisive use of handlers to keep it's function as administrative and device independant as possible. In order to produce two copies of archive media, two copies of tarexad are concurrently run, they communicate to each other via distd to keep their tapes identical.

    parfile: /h/archive/sw/par/tarexa3d /h/archive/sw/par/tarexa4d
    Working directory: /data/kapu/tarexa3 /data/kapu/tarexa4
    Assosciated utilites: tarexa3h tarexa4h tarexa_init sql_update

    The Database Daemon

    This daemon is still part of the pipeline though it has been replaced by calls to the database handler in both stripheadd and tarexad.
    Parfile: /h/archive/sw/par/dbd.par
    Working directory: /data/kapu/dbd
    Kanoa Withington
    Last modified: Tue Jul 17 14:36:21 HST 2001