System Overview

In this document, we will discuss the overall structure of the Elixir system of programs and their implementation at CFHT. Additional documents will discuss in more detail specific components of Elixir.

Introduction

With the creation of the large mosaic cameras, data handling tasks which were once trivial have become onerous for most end users. Simple steps such as median filtering 10 or 20 images to produce, for example, high-quality flatfield images has become a difficult job for users without significant investments in computer hardware (memory and disk space). Just the bookkeeping required to keep track of the large number of objects which can be detected on a single set of images has become a significant database problem. The resulting intimidation many users feel inhibits some from requesting time with the large cameras, or acts as a barrier to the data reduction task if they actually acquire the data. The groups that have been most successful in producing results from the large mosaic cameras at CFHT and elsewhere have usually spent large amounts of time writing dedicated software for the job, and frequently bring their own hardware to the telescope to analyze the data as it comes off the telescope to avoid the large overhead of writing to and reading from tapes. If these large mosaic cameras are to be used by the majority of observers, support must be provided to help them overcome these hurdles.

At CFHT, the Elixir project has the goal of enabling the optimal use of data by our observers, in particular from the large mosaic cameras. We call the project `Elixir' after the fabled goal of the ancient alchemists: the Elixir which could restore youth or turn lead into gold. The Elixir project seeks to restore youth to archived data and turn weighty mass of unmined data collected at CFHT into nuggets of gold.

There are several ways in which a dedicated project, having access to all acquired data, can smooth the data reduction process.

$\bullet$ All available detrend¹ data can be used to produce optimal `master' detrend frames for a given set of science data. Having access to all detrend images during a particular camera `run' (the period the camera is mounted on the telescope) avoids the limitation of using only the few images one can obtain in a single night or two at the telescope.

$\bullet$ Constant monitoring of the photometric, astrometric, and other system-wide characteristics of the combined camera-filter-telescope system over all relevant time scales can aide in the identification of systemic problems and can allow for improved determination of calibration terms such as photometric zero-points. We can determine on-the-fly if a particular set of standard star images are taken in photometric conditions, and we can watch for slow trends if, for example, components acquire deposits or the detector quality fades.

$\bullet$ All data can be passed through a standard reduction system, providing observers a `quick-look' mechanism in real-time and sanity checks when reducing the data. Dedicating computers and software to this task allows for optimization of the analysis routines to provide high-quality reductions on short time scales. This allows observers to decide in real time if their images are deep enough, cover the correct part of the sky, or if they have contamination from, ie, bright stars. For some projects, the standard reduction results produced by Elixir may be sufficient for the intended science.

Elixir System Overview

The Elixir project consists of a number of independent software components which are connected together as needed to perform a wide variety of analyses. Figure 1 shows the connections between these components at the highest level. There are several types of analyses, each with different time scales and differences in how they interact with images. These differences are ignored in this diagram, which simply shows the conceptual connections between different elements. In this diagram, the arrows trace the motion of information about images, groups of images, or derived quantities as it moves through the Elixir system.

Figure 1: Conceptual diagram of data flow in Elixir. The arrows show the path that data about an image follows through the components of Elixir. Rounded rectangles represent programs, circles represent data products.

Figure 1 also shows the general relationship between Elixir and the other components of the CFHT environment, NEO (which controls the telescope and detectors and creates the images), QSO (which controls scheduling and image validation during Queue runs), and DADS (which controls data distribution). The connection to NEO is made via a script invoked as part of the exposure process. The bulk of the Elixir system runs autonomously, waiting for triggers to perform analyses on different images. Elixir, QSO, and DADS exchang information as needed by calling programs which interact with any of the various databases, inserting or retrieving data. Elixir also provides tools to DADS which invoke a specific version of the Elixir analysis system.

In the diagram, there are several blocks which represent fairly complex collections of programs. These consist of: 'imstats', a system which provides a very minimal, fast set of analyses on all images, 'ptolemy I', which provides a modest amount of analysis on all science images, balancing the needs of speed and detail to provide semi-real-time information, 'ptolemy II', a more detailed science image analysis which is run at the end of a CFH12K run, and 'mkdetrend', a system to produce the master detrend frames from the collection of detrend images taken during a CFH12K run. The details of these blocks are described in separate documents for each component.

Each of these blocks represents a collection of programs which perform a series of operations on images or groups of images. We have developed a program called `elixir' to organize and coordinate a complex set of operations on a number of images or image groups, using an arbitrary number of clustered computers to perform the operations in parallel. Elixir takes a set of distinct programs and provides a mechanism to pass information from one program to the next to manage the analysis process. Depending on the configuration, elixir can define a wide variety of analysis systems by grouping together different programs as needed. Each implementation is defined by a simple text-based configuration script. We call a particular implementation an `elixir' and give the different elixirs names relevant to their tasks, such as `imstats'. The program `elixir' is described in a separate document ('Elixir - the program-organization program').

Elixir Components

The `realtime' block consists of several independent processes which provide immediate feedback of several types to the observers. Unlike the analysis elixirs, this component is synchronous in the sense that the processes are immediately launched for each image, and therefore are completed in a generally predictable time from the acquisition of an image. The analyses performed include: 1) a minimal seeing measurement, 2) creation of a binned, grey-scale jpeg image, and 3) analysis of a focus frame, and 4) saving a copy of the image for the other Elixir systems, and 5) passing the name of the image to the rest of the Elixir system. The results from these realtime processes, along with other asynchronous results, can be viewed by the observer within a single display tool. The details of the realtime components are described in 'Realtime Elixir Components'.

The quick-statistics elixir, 'imstats', performs a few basic measurements on each CCD image and places the results in a database of registered images. One element measures the bias and sky level while a second element uses `sextractor' to measure the FWHM of the brightest stars. To speed up this analysis, only a small segment of each image is used. This operation is meant to happen reasonably quickly after the image has been taken, so the observers can have near-real time feedback. The user display tools (indicated on the bottom of the figure and described elsewhere: `user displays') include plots of the FWHM and sky brightness as a function of time for a recent time-period.

The `medium-depth' analysis (ptolemy I) and the detailed analysis (ptolemy II) perform very similar operations, with some small distinctions. Both perform a complete photometric and astrometric analysis of each CCD image: detrending, object detection and flux measurement, astrometric calibration, and incorporation into a photometry database. Each of these steps are performed by a separate program, with a modular nature which can allow substitution of different components as needed. For example, the current system employs the program `gophot', a variant on 'dophot', to perform the object detection / flux measurement step, but if another program is developed which is superior, it may be easily substituted for `dophot'. In some tests, we have used `sextractor' for this step with complete success. There are two major distinctions between `ptolemy I' and `ptolemy II': 1) the accuracy of the detrending step and 2) the depth of the analysis. `Ptolemy I' is performed immediately after the image is obtained, and is intended to finish within 10 - 30 minutes. The simplest bias-subtraction and flat-field division recipe is applied using a flatfield image which is probably not the best possible for the night, but simply the closest appropriate one available. In some cases, this may even be an image from a previous run with the camera. To increase the speed, the analysis is performed only to a moderate depth, without pushing for the detection of the faintest stars in the image. `Ptolemy II', on the other hand, is performed later, generally at the end of a CFH12K run, giving time for the Elixir system to generate a `best set' of detrend images. The analysis is pushed to a significant depth, tentatively a 5$\sigma$ limit instead of a 15$\sigma$ limit.

The 'mkdetrend' component is used to create the master detrend images. This component actually consists of several elixirs, glued together with a perl / html interface. At the end of a CFH12K run, and during the run as needed, this component is used to generate the master detrend images. A web-driven tool makes it easy to evaluate the selection of input images used to create the master detrend frames and alter the input as needed. The mkdetrend system organizes this process and makes some rudimentary image selections, but this component requires some human interaction to refine the correct input image selection.

The elements on the bottom of the figure illustrate the data visualization tools and the interaction with the rest of the telescope observing environment. Several tools exist for querying the Elixir databases. There are command line programs that can be used to select subsets of the data in the detrend and image registration databases. The photometry database is a bit more complex. The program, `status', can be used to query the various database tables created by the Elixir system, and for displaying data of interest in various forms. Status is a rather complex program with a command line interface similar to a UNIX shell complete with a sophisticated data-analysis oriented macro language. Status is described in detail elsewhere. It may be used to perform specific queries as desired, or it may be set to run a continuously looping monitor program. An implementation of status generates reports during an observing run, providing a variety of interesting plots to the user display tool, which can be seen by the observer in semi-realtime. Queries via status can also be used to extract subsets of the various database tables, ie for export to the observer after the observing run is completed. Another document describes the different datasets available to the observer from Elixir.

External Packages

The Elixir system is designed to be flexible. The specific operations which are performed on the images may use whatever tools are available, with little difficulty. There are several steps which invoke programs developed outside of the Elixir system.

The Flips collection of programs by Jean-Charles Cuillandre are designed to perform the basic image reduction operations related to the detrending process. In this collection, there are tools to optimally merge, for example, several input flats to create a master flat, or a master bias frame. There are also tools to apply the resulting detrend images to science images. The Flips tools are designed to make these steps fast and efficient, and to make the manipulation of CCD mosaics transparent. We use Flips programs to create the master bias, dark, flat, and fringe frames, and to apply these frames to the science images.

The Sextractor package by Emanual Bertin is an effecient and easy-to-use tool for performing stellar photometry on an image. The output is very flexible, and can easily include as many or as few measureable quantities for each object detected. Sextractor can be run with very little initial information, which makes it particularly useful in an automatic processing environment.

Gophot is our adaptation of the program Dophot by Paul Schechter. The Dophot algorithm measures stellar photometry of objects in an image by performing analytical fits to the object profiles. Unlike Sextractor, Gophot / Dophot uses a simple 2-D Gaussian profile to measure the photometry. Gophot / Dophot has a somewhat more physical classification scheme for the detected objects than Sextractor, but it is somewhat slower.

Elixir Philosophy

The Elixir system is intended to be very flexible. Abstraction of keywords, files. Extensive configuration system.