In the early history of CFHT, the images taken at the telescope were archived more or less manually on round magnetic tapes. In 1989, it was decided that every single frame taken, regardless of its content, should be automatically archived by a transparent process. To achieve this goal, optical disk drives were purchased from Delta Microsystem. Each optical disk (Maxtor disk) had the capacity of 800 megabytes (400 Mb/side). Rick McGonegal and Bob Link from the software group implemented the first archiving software and since then, it has been part of the work of the French cooperant to maintain and improve what became known as the ``CFHT archiving system'' under the constant supervision of Bob Link.
With the spectacular increase in the size of the CCDs, the Maxtor optical disks soon proved to be unable to face the huge quantity of data taken every night (up to 800Mb with 2Kx2K CCDs). In the middle of 1992, a new generation optical disk drive, featuring a capacity 3Gb per disk side, was purchased from Sony and was accommodated in the archive pipeline. It was also decided that the CADC (Canadian Astronomy Data Centre) in Victoria would archive the CFHT data and that the optical disks were to be shipped to them when full. To be able to keep a back-up copy as a permanent record for CFHT, the archive pipeline was also modified to mirror the archived data onto an Exabyte tape. Simultaneously, some miscellaneous features were added, such as an automatic image quality evalution and an automatic header extraction process.
Since its last important modification in early 1993, the CFHT archiving system has run quite smoothly (with the only exception of a crash in March 1993). The supporting software however, still based on the original implementation with a lot of successive changes, was becoming weak and unstructured. Moreover, the Sony optical disk drive is a very expensive hardware and soon it was decided that a maintenance contract couldn't be purchased. Concerns then raised about what was going to happen in case of a hardware failure as the current software was assuming the Sony optical disk as the master archive media and the whole implementation was very hardware dependent. Similarly, C-Shell inherent weaknesses such as the poor interruption control made the administration of the whole pipeline delicate at best.
Therefore it was decided in late fall 1993 to launch a project to reorganize the archiving system. After studies, it became obvious that a completely new architecture was required to solve the previous problems. The present manual describes this new architecture and the new programs that had to be written. It was decided to stick to C-Shell for simplicity, although the process control has been strongly tightened.
The new archiving system was put into operation on February 2nd 1994.
The main features of the new pipeline are:
- Separation of the copying process (transfer from summit to Waimea) from
the
different archiving processes (transfer to OD or exabyte),
allowing to continue transfering files while writing to media and even
if
one of the media is full;
- Better modularity that allows to very easily add any new archive media
in the
pipeline (;
- Better general reliability, data flow and interruption control.
This changes should be transparent to the external users. Specifically:
- A back-up copy is still made on exabyte to be stored as a permanent record
at CFHT;
- FITS headers are still extracted on the fly and put in
uwila:/opt/uwila_data/fits_hdrs;
- AIQE results files are still updated every morning in
uwila:/opt/uwila_data/aiqe.