2.1 Data flow

[IMAGE (figure)]

Figure 2.1 shows the general data flow diagram of the new CFHT archive pipeline.

The entry point into the pipeline is the summit shadow directory (currently /users/observer/shadow). Every single frame taken by the telescope should be hard linked there by the data acquisition software but this is not the concern of the archive system. Each file has to be taken down to Waimea to undergo different processes.

The archive pipeline can be broken down into three elementary independant steps:

• Transfer of the files from the summit disk to the Waimea disk over the T1 line.
• Distribution of the files within Waimea disk to the entry point (directory) of the different processes to be carried out.
• Processing of each file, which can be of two types:
• Archiving the file to a media (magnetic tape...);
• Miscellaneous processing of the file (Ex: to get some statistics from it, to extract its header...).

The transfer and the distribution steps are each handled by a single program. The processing step involves several programs (one per task to be performed) working independantly and in a parallel geometry. At any rate, when a file has been successfully processed, it is removed from the process directory. When it is removed from all the process directories, it diseapears from the archive system. Hence the global archive system is a pipeline (FIFO process).

The files from the /users/observer/shadow shadow directory are copied down to a Waimea disk over the T1 line by the copy daemon copyd. Currently the Waimea disk is the /archive partition on machine alohi and the target directory for the transfert is /archive/distributor. The distributor daemon distd distributes the files from /archive/distributor to the working directory of each archiver and miscellaneous process. The currently active archivers and miscellaneous processes are the following:

dumpexad, operating on /archive/dumpexa, that writes to Exabyte tape in DUMP format is available but not currently used.

Each miscellaneous process is an independant program whereas all the archivers share a same main code archd but are called with different parameters and invoke a specific handler to perform the device I/Os.