The CFH12k Queued Observing Program:
Operational Model Overview

Pierre Martin, Dennis Crabtree
January 11, 1999

 

Table of Contents



1. INTRODUCTION

SAC has recommended that CFH12k be operated in a queue scheduled/ service observing (QS/SO) mode beginning in Semester 99II. This document presents an overview of the proposed operational model for the queue scheduling/service observing (QS/SO) mode to be implemented for CFH12k. Another relevant document has been published by T. Abbott and D. Crabtree on the proposed queue-scheduling rules and will only be summarized here. At this point, details for the implementation of specific operational issues have been developed and actions and resources needed to carry them out have been identified. For other issues (e.g., the calibration plan and the reduction pipeline), details will be better defined during the next semester after commissioning of the camera. Experience gathered with CFH12k during that period will be extremely valuable to evaluate the model proposed and test the diverse algorithms.

Queue scheduling/service observing (QS/SO) mode was primarily designed to increase the scientific productivity of telescope time and not to reduce the cost of the telescope operations. This document includes some rough estimates on the resources and manpower needed to fully implement an affordable and highly efficient model for QS/SO with CFH12k. Readers should keep in mind, however, that even for an imager QS/SO mode is a complicated process. Changes to the model described here are to be expected during its implementation.


2. OUTLINE OF THE QS/SO MODEL

The operational model proposed can be divided in a few specific steps which are more extensively discussed in the following sections:

i) Proposal Submission: The submission of proposals is done in two phases. Phase I is based on the current submission procedure and is used by the TAC to evaluate the science merit of the proposals and recommend the integration time (I-time) that should be allowed for a given program and for the QS/SO run. CFHT will provide an estimated efficiency for QS/SO, which allows for both weather and observing efficiency,  to convert hours of telescope time into hours of I-time. Phase II addresses directly the information required for the queue execution (observing blocks). For Phase I and  Phase II, the investigators enter the details of their program through a set of Web forms and/or observing tools and this information is sent to a database at CFHT.

ii) Queue Execution: Prior and during each observing night, a list of observing blocks is assembled to optimize science productivity according to the weather conditions and requested specifications. This list is derived from the database using  sophisticated software, the scheduler, evaluating the priority of each observing block using a weighting system. Dedicated and trained observers perform the observations, as well as any required calibrations. The management of the queue and all decisions related to the QS/SO run are made by one of the resident astronomers assigned as queue coordinator.

iii) Data Reduction and Evaluation: The astronomical observations are automatically pre-reduced to remove instrumental signatures using a database of relevant calibration frames. A thorough data control quality evaluation is carried out on the data and the pre-reduced data by the queue coordinator.

iv) Data Distribution and Archiving: The astronomical data (raw and/or pre-reduced) and relevant calibration frames are distributed to the investigators. The actual medium use and the manpower needed for data distribution has not been identified yet. All data will be archived at CADC.


3. THE DATABASE

An essential tool for the QS/SO mode is the database. This tool will be used for different purposes at CFHT and its development is already in progress. For QS/SO, the database will contain essentially the programs (i.e. the observing blocks) prepared by the investigators. Database management software, an electronic logbook and the scheduler will give access to the database for the control of the queue (i.e. blocks remaining for a given program, programs completed, comments on data evaluation, etc).

Basically, the observing blocks should contain the following information:

These fields will be accessible to the queue scheduler for sorting out the observing blocks into a prioritized list. The database will exploit Sybase technology and will be managed on an NT machine. We are hiring a University of Victoria co-op student for four months starting in January to develop the necessary tools (from Perl or Java) to access and manage the database.

4. PROPOSAL SUBMISSION

The proposal submission will take place in two tiers. Phase I applies to all the instruments at CFHT and is a conventional proposal oriented on the scientific justification. Phase II is specific to CFH12k  (at least for the moment) and provides all the information required to execute the observing queue.

4.1 Phase I

The first tier proposal is based on the current observing proposals used at CFHT with some refinements required for QS/SO with CFH12k. Its purpose is to allow the TAC to establish a scientific priority for each observing program and to provide the amount of time to be allocated for each accepted proposal. As described in Abbott & Crabtree document, the easiest way to manage the time allocated for QS/SO is to consider only the integration time (I-time) for a given program, that is, the time at the telescope under a clear dark sky with the camera shutter open. This does not include overheads associated with telescope pointing and camera readout time. We estimate that 25% of time will be lost to weather and that CFH12k will be operated at 70% efficiency, i.e. from twilight to twilight the shutter will be open 70% of the time. If the time from twilight to twilight is 10 hours, this translates to an average of 5.35 hours of open shutter time for each night. A program requiring 16 hours of I-time will need 3 nights to be completed on average.

After its evaluation, the TAC must provide some information that will be used for Phase II preparation and during the queue execution:

  1. The maximum total I-time allowed for each program
  2. The relative rank of each accepted proposal. Proposals would also be classed as either A (must get done), B (high priority), or C (time available)
  3. The number of nights to be allocated to the QS/SO program
The Phase I proposal form is Web-based and is being developed at CADC (Poopsy). The astronomers at CFHT will be testing this prototype and the final version should be available for the general users at the end of January. The deadlines for submission of Phase I proposal remain the usual dates (March 1st, September 1st). Figure 1 presents a flow diagram of Phase I.
 
 
psfig {figure=phase1.ps,height=3in}
 
Figure 1: Flow diagram of Phase I proposal submission
 

4.2 Phase II

The second tier proposal addresses directly the information required for each observing program. From an observing proposal form especially designed for Phase II and provided by CFHT, the investigators can specify individual or group of exposures (observing blocks). The set of pre-defined keywords/value pairs assembled from this form (see section 3 for the list) is then sent to the QS/SO database and constitutes the specifications examined by the scheduling software for the prioritized list of programs.

The Phase II proposal form will have the following characteristics:

Among the specification fields offered, the investigators will be able to select these options:
  1. Exposure time
  2. Number of exposures
  3. Seeing (S) bands: S £  0.55''; 0.55"< S £ 0.65''; 0.65'' < S £   0.8''; 0.8" < S£ 1.0"; S > 1.0''
  4. Sky Brightness [photons sec-1 arcsec-2] (e.g. MK dark sky = 50 photons sec-1 arcsec-2 at V)
  5. Time constraints
  6. Observation type: fixed; dithering plus dither pattern
  These constraints are then be used to automatically build a set of templates (see Abbott & Crabtree). These templates are then brought together to form an observing block which is treated as a single unit by the queue scheduler. At the moment, there are two options for the Phase II proposal form: No definitive decision has been made although the possibility for a modified version of the Gemini OT is being thoroughly examined at the moment (discussion with the UKIRT group has been also initiated). According to the actual time schedule of Phase I and TAC evaluation, the final submission deadline of the observing blocks has been fixed to one month before the beginning of a semester, that is, June 30th and December 31st. Figure 2 summarizes the sequence of events of Phase II.
psfig {figure=phase2.ps,height=3.2in}
Figure 2: Flow diagram of Phase II proposal submission
 

5. QUEUE EXECUTION

5.1) Queue execution rules

The QS/SO observations will be executed according to specific rules. The algorithms sorting out the observing blocks from the queue database (see below) will automatically apply these rules. The decision trees are explicitly described in Abbott & Crabtree document and are only summarized here:

Program Viability: To be considered as viable, a program should meet some specific criteria:
Program Completion:
5.2) Scheduler

Optimizing science productivity being the ultimate goal of the QS/SO mode, it is necessary to introduce a sophisticated algorithm to prioritize the diverse programs available in the database. The development of this software, the scheduler, is one of the most challenging tasks of all QS/SO programs. Basically the scheduler assembles a list of observing blocks to be executed according to a set of constraints evolving with time. The scheduler will also be able to produce a suggested list of observation blocks given the LST, seeing, sky brightness and transparency. The priority of each program is constantly evaluated during a QS/SO run (and for during a given night) from a weight system according to the specifications requested by the investigators, TAC evaluation and the actual sky conditions. The following constraints will be included in the scheduler algorithm:

Defining the importance of the weight and its time-dependent function for each constraint is the key to QS/SO. An obvious advantage of CFH12k, however, is the possibility to use the observing programs done under real sky conditions during the 1999I semester to test the algorithm. The scheduler will be developed and tested by the CFHT astronomers. The approved version should be available before the Phase II submission deadline and will be extensively used before the first QS/SO run with the 1999II programs submitted in the database for final testing.

5.3) Calibration Plan

An important element of QS/SO mode is the calibration plan, that is, a set of procedures applied regularly through a queued observing run to gather the necessary calibration data. For CFH12k, the plan is divided in two parts:

1) Instrument and CCD Maintenance Plan:

In QS/SO, it is crucial to monitor the status of the instrument and the detectors on a regular basis. For instance, calibration ``master frames'' for the data reduction can only become usable when the stability of the CCDs is known. At the beginning of each QS/SO run, a set of CCD calibration frames (biases, darks, flat-fields) will be obtained and included in the database for the upcoming observations. All these frames will also be processed automatically to analyze the characteristics of the CCDs (readout noise, cosmetics, etc).

2) Astronomical Calibrations:

In order that the data obtained with CFH12k in QS/SO mode useful for archival research, we plan on observing every field, in each filter requested, during at least one photometric night. If a night is photometric we will take short exposures of as many uncalibrated fields as possible, as well as photometric standards. These exposures can then be used to calibrate the data obtained for the scientific program. This astronomical calibration plan will be executed as an independent observing program in the queue. It will have the highest rank among the other scheduled observations but will not consumed any I-time allocated to these programs by the TAC. A set of photometric standards will be observed each night the sky is photometric. The general plan will be published before the deadline for the Phase II proposal submission. If specific calibrations are required for certain programs, the investigators will have to include them in their program and I-time will then be accounted. The main strategy for the all the astronomical programs requiring photometry will be to obtain at least one exposure in the desired filter for each field of the program during a photometric night so that subsequent exposures not done under good conditions can be calibrated.

J.C. Cuillandre and P. Martin will develop the full calibration plan after the commissioning of CH12k and with the experience acquired with the camera during the 1999I. A detector specialist supervised by the resident astronomer responsible for CFH12k will carry out the maintenance plan. The astronomical calibration will be performed by the designed queue observers and reviewed by the queue coordinator (see below).

5.4) Observations

Operations in a QS/SO run extend beyond the normal hours associated with observing. The database and the queue need to be prepared prior to each observing night, the data quality from the previous night must be evaluated and data from the observations have to be gathered. A team of individuals is needed to carry out these tasks:

  1. The QS/SO coordinator, stationed in Waimea, is responsible for sorting and updating the database, evaluating the reduced data and controlling the archiving. He is the only direct contact between the investigators and their data. His work his done during the day but is available on a 24-hour basis when decisions affecting the QS/SO run must be made. He has full authority involving feasibility of programs already started and returning observations to the queue if found not to meet the specifications. This role is assigned to one of the resident astronomers.
  2. 1a) The QS/SO coordinator is stationed in Waimea and is responsible for maintaining the database, evaluating the data, overseeing the archiving and any contact with the investigators. He has the final authority involving the feasibility of programs already started and returning observing blocks to the queue if they do not meet the specifications. This role is assigned to one of the resident astronomers.

  3. The Waimea Observer is present at the beginning of each night to review the program prepared by the coordinator and anticipate the needs for carrying out the calibration plan. His main role during the night is to evaluate the data obtained, run the queue and data reduction pipeline. He is the authority in the decision process affecting which observing block should be executed or not. He is in constant contact with the summit observer via audio/video links and keeps a close look at the ongoing process. This role is filled by one of the resident astronomer and/or an internal observer specifically trained for QS/SO (post-doc, graduate student, hired QS/SO observer).
  4. 2a) The Observer, based in either Waimea or at the summit, is the one primarily responsible for operating the instrument and data acquisition system. He is also responsible for running the queue and the quick-look reduction pipeline. He is the authority in the decision process affecting which observing block should be executed or not. If Waimea-based, he is in constant contact with the summit observer via audio/video. This role is filled by one of the resident astronomers and/or an internal observer specifically trained for QS/SO (post-doc, graduate student, hired QS/SO observer).

  5. The summit observer operates the telescope, the instrument and the data acquisition system. He is responsible for the first ``snapshot'' evaluation of the data (seeing, sky brightness, transparency) as well as the main source of information for the weather conditions. This role is carried out by one of the CFHT Observing Assistants.
3a) The summit observer operates the telescope and is responsible for the first ``snapshot'' evaluation of the data (seeing, limiting magnitude) as well as the main source of information for the weather conditions. This role is carried out by one of the CFHT Observing Assistants.

4) The CFHT director represents the final authority in the case of conflicts between the queue coordinator and the investigators.

Although this model seems to be the most efficient alternative to carry out observations in the QS/SO mode with CFH12k, it introduces a few logistic difficulties. First, there is a clear manpower problem for the coordinator and Waimea observer positions. It's unrealistic to assume that on the long-term, both positions should be assigned to two CFHT resident astronomers. For instance, in 1999I, 80 nights have been assigned for CFH12k observations. To cover their other duties and assuming the role of coordinator and/or observer at the same time is not possible. CFHT is actually considering the possibility to hire post-docs for which an important fraction of their time could be spent as a QS/SO. Another solution is to hire a specifically trained observer for the task. Second, there is a policy at Mauna Kea requiring that two persons should be present in the dome during an observing night at all time for security reasons. With the model above, this condition is not filled. We are discussing this issue actively with other observatories at the summit to find a solution. A security program with well-defined actions in the context that the summit observer is in constant communication by video/audio link with the Waimea summit could certainly be developed. Finally, an observing room has to be installed at the Waimea headquarters, the actual computer room being very inadequate. We have already identified the location and the resources necessary to make this room available during the 1999I semester.


6. DATA EVALUATION AND REDUCTION

A thorough control of the data quality must be performed during a QS/SO run to optimize the science productivity of the telescope time. The data quality will be evaluated through many channels with the queue coordinator having the final responsibility for deciding if an observation meets specification or should be repeated. For a given observation, the following process is proposed:

CFHT will also provide a pipeline of pre-reduction of the QS/SO data. A software prototype has been developed for UH8k (FLIPS) by J-C. Cuillandre and will be extended to cover peculiarities of the CFH12k camera. The software will probably only cover the pre-reduction steps to remove instrumental signatures (e.g. bias, flat-fields, cosmic rays) and will be operated during the observing night by the Waimea observer. At the moment, there is no plan to calibrate astronomically the images (photometry, astrometry) but the necessary calibration frames will be available for the investigators.

7. DATA HANDLING AND ARCHIVING

The amount of data generated with CFH12k ($sim$ 200 Mbytes/image) will be very large. Copies of the data will only be kept at CADC, as CFH does not have the appropriate resources. However, CFHT will be responsible for distributing the data to the investigators. This is a process for which the complexity cannot be overestimated. At the moment, data distribution introduces many unanswered questions for the QS/SO mode with CFH12k at CFHT. It is still unclear how the data will be distributed due to the lack of manpower required to take care of this task if physical devices (e.g. DLTs) are used. Compressed data could also be transferred by fast Internet access but this is not yet available.

Some other issues have been addressed but comments and suggestions from the SAC will be particularly useful at this point:


8. OBSERVING TIME ACCOUNTING

It is important that the accounting of observing time between the three Agencies, and Discretionary time, be properly handled in QS/SO mode. There are two points of concern. First, the number of hours of observation carried out for each Agency must be close to that expected. The expected number of hours is the number of nights allocated to QS/SO by the Agency multiplied by average length of a night and the expected overall efficiency, including time lost to weather. For example, if NRC allocates 30 nights to QS/SO, and with an average night length of 10 hours and overall efficiency of 53.5%, then they expect approximately 157.5 hours of exposure time on Canadian programs. No one Agency should benefit to a large degree at the expense of another Agency.

The second important point is the amount time actually charged to each Agency. The number of nights allocated to each Agency during any one Semester no longer needs to conform to the nominal 42.5/42.5/15 distribution. This flexibility will also extend to the QS/SO time. The total exposure time for each Agency will be calculated at the end of each Semester. Each Agency will be charged the number of nights that their fraction of the total exposure time represents. For example, if Canada's total exposure time was 44.3% of the total for the semester, and there were 66 nights allocated for QS/SO, then Canada would be charged for 29.24 nights. This number would be added to the number of nights scheduled traditionally to determine Canada's allocation for the Semester. Canada's initial allocation for the next Semester would be adjusted to approach the long-term average of 42.5%.


9. SCHEDULE

It is expected that most of the observations performed with CFH12k in the 1999II semester will be performed in this mode. The following Table gives a rough estimate of the schedule expected for the implementation of the QS/SO mode with CFH12k.
 
 

Item
Completion Time
Manpower
Database Development
01/31/99
DC/GH/PM
Database software
03/31/99
GH/DC/PM/SoftW
Phase I Proposal Web-form
01/31/99
CADC/Astros/
Phase II Proposal Tool
04/30/99
DC/PM/SoftW/Astros
Queue scheduler
05/31/99
PM/GH/DC/Astros
Calibration Plan
03/31/99
JCC/PM
Data evaluation tools
04/30/99
DC/JCC/PM/Astros
Data Reduction Pipeline
06/30/99
JCC/PM
Waimea observing room
07/15/99
SoftW/Astros/OAs
Simulation/Testing
07/31/99
Astros/OAs
Legend: DC: D. Crabtree; JCC: J.-C. Cuillandre; GH: G. Higginson; PM: P. Martin; Astros: CFHT resident astronomers; SoftW: software group; OAs: observing assistants