Detrend database

Perhaps the most imporatant type of Elixir data product is the collection of master detrend images. By working with the collection of images from an entire CFH12K run (the period the camera is mounted on the telescope), we have a unique opportunity to create the best possible set of master detrend images. A crucial element of this process is a database of the resultant detrend images which can be used to make the associations necessary between master detrend images and raw science images.

Detrend Images and the Detrend Database

Elixir generates master detrend images for each type of detrend data. So far, the types which we are supporting consist of: darks, biases, flats and fringe frames. To give an idea of the number of master detrend images, here are some typical numbers. We produce flat field images for each of the broad-band filters (5) and narrow-band filters as needed. We also produce bias images, as well as dark images for several pre-selected exposure times: 60, 300, 600, and 1200 seconds. The dark images may be scaled from one of these special exposure times to whatever is appropriate for the specific image. We also produce fringe images for each of R, I, and Z. Assume for now that we don't produce more than one set for the run, and that we don't produce detrend images in the narrow-band filters. In that case, a single 12k run will produce 11 distinct detrend images (full-mosaic). This represents 2.5 GB. Over the course of 2 years, we can expect to have roughly 20 12k runs, which means about 50 GB worth of detrend images, or about 2600 different CCD frames, for two years of operations.

It is not a significant drain on resources to keep the 50GB of detrend images on spinning disk. However, it is crucial to have a method to determine the appropriate detrend image for a given situation, or a given image. This is the role of the Detrend database.

The detrend database is used to organize the detrend images and make the association between the specific detrend image and those images for which it is valid, and vice versa. The database follows the format of the other databases used in Elixir: a file written with binary data defined by a C header structure, along with a FITS header giving summary information. File locking is used to maintain the database integrity.

 /* Detrend Database structure */
 typedef struct {
   unsigned long int tstart;
   unsigned long int tstop;
   unsigned long int treg;
   float sigma, clipsigma;
   int type;
   int filter;
   int ccd;
   char filename[254];
   char flag1, flag2;
 } DetReg;    /* 272 bytes */
 

Above we show the C structure used to define the detrend database data. The data included in this structure includes: the image name, image type (flat, etc), filters as appropriate, complete path to the images, and so forth. In addition, to basic information about the image, three important time references are kept as well: The time the detrend image was registered in the database, and the start and end times for which this detrend image is valid. These dates are assigned during the detrend creation process (see ....).

Interaction with the detrend database takes place through two command-line tools, one for input, one for output. The first tool, detregister, is used to add, and if needed delete, detrend images from the detrend database. Ideally, the header of the detrend image supplies all of the information needed by this program to fill in the database fields. However, command line options allow database fields to be provided by the user if those in the header are missing or inaccurate.

The second tool, detsearch, lets the user peruse the detrend database and grab specific detrend images or sets of images as needed. One use of this tool is to select the appropriate detrend image of a given type for a specific image:

 detsearch -image example.fits 00 split -type flat
 
Lines of this type are used in the 'flatten' routine to provide the appropriate flats, biases, etc. In this example, we request the most appropriate flat field image for the image example.fits. The program looks in the header of example.fits to decide the appropriate filter, time, etc, and selects the image from the database. Other uses of 'detsearch' could include listing all detrend images of a given type, ccd, filter combination, etc. The program 'detsearch' therefore provides all necessary queries of the detrend database and provides the association between science images and detrend images. By incorporating the function in the detrending program, the association is automatic and transparent to the user.

A crucial element in associating a given image with a given detrend image is the time of the image. A given detrend image has a period of time over which it is valid. This may be short, as in the case of fringe frames which are only valid for a small number of days, or it may be very long, as in the case of mask images, which are valid as long as the CCD arrangement is not changed. During the detrend creation process, the valid time range for the images is defined. If an image is added to the detrend database which overlaps an existing image, the most recent image take precedence by default. A search for a detrend image may specify if the resulting images exactly overlap the specified time, or the search may optionally return the most recent image, even if the end of the valid time range is before the time in question. This option is useful to perform a rough analysis on image from an ongoing run before the latest detrend images have been produced. In such a case, the detrend images of the previous run would most likely be chosen as acceptable, if not ideal.