Queued Service Observations with CFH12K: Semester 2001A Report 08/02/01

A - Introduction

The Queued Service Observing (QSO) Project is part of a larger ensemble of software components defining the New Observing Process (NOP) which includes NEO (acquisition software), Elixir (data analysis) and DADS (data archiving and distribution). About 56 QSO nights (or more generally, NOP nights) with CFH12K were scheduled for the semester 2001A. This report presents some statistics and comments on our first experience with the queued service observing mode.

B - General Comments

Despite a few technical problems and some considerable time lost to weather, our first experience with the QSO mode has been quite positive during 2001A. The following comments address the main issues related to the QSO/NOP system for this semester:

1. The QSO concept is sound. With the possibility of preparing several queues covering a wide range of possible sky conditions in advance of an observing night, a very large fraction of the observations were done within the specifications (see below). The ensemble of QSO tools allows also the quick preparation of queues during an observing night for adaptation to variable conditions, or in case of unexpected overheads. The observing tool offers the possibility to load a pending queue and prepare it (for instance, by including the focus sequences) while another one is being executed so that the transition between queues is done rapidly.

2. Technically, the entire chain of operation, QSO --> NEO --> TCS, is very efficient and robust. This is a complex system and some glitches have been found and fixed during this semester. In general, the entire operational software chain has proved very satisfying. More than 4000 exposures have been taken through the QSO/NEO system in 2001A... Some areas can still be improved to increase efficiency (i.e. moving filters during readout) and we are actually working on these modifications for 2001B.

3. Our initial estimate of the general integration-time available for a given night (6.5 hr) proved to be correct (see below). However, this value depends strongly on the complexity of the queued programs to be performed. During the last three runs, we have been generally able to do more than 6.5 hours per night, even during the shorter summer nights. The main operational overheads are: focus sequences, pointing checks and dome rotation. As much as possible, the queues are always prepared to minimize the overheads. Due to the implementation of a new prime focus bonnette controller during March and April, severe overheads were introduced in the pointing operation. Fortunately, these have now been reduced to almost none by a new bonnette calibration procedure, done at the beginning of the each run.

4. The main difficulty related to the QSO operational concept is the adaptation to very variable seeing and non-photometric conditions. During 2001A, we did not have enough programs requesting bad seeing or non-photometric conditions. Also, not enough programs requesting bright time were available. We hope that more diversity will be found in programs for 2001B... Unfortunately, there is no seeing monitor available for Mauna Kea. We only get the seeing value after the readout of an exposure. This lack of reference for the image quality resulted in some observations being done in worse conditions than requested and those were not validated.

5. A severe constraint for the QSO model is the limitation on the number of filters that can be mounted in the wheel of CFH12K at the same time. A filter must be mounted for at least three nights to make sure that the necessary detrend calibrations can be obtained. Thus the number of filter changes during a run is limited. This introduces an additional, restricting parameter in the selection of the programs to be carried out. Due to this constraint, a target might not be observed with all the filters required during the Phase 2 during the same run. We always try to achieve this for minimizing pointing discrepancies and slight rotations between the frames but this is not always possible. This constraint is more severe even for "non-standard" filters. Since those are generally less requested than the BVRI set, observations cannot be always performed during sky conditions specified by the investigators. In fact, for 2001A, most of the observations with the non-standard set were done in better conditions than requested. This is good because it respects the queue concept. However, this can be frustrating for scheduling other programs using the standard set and requesting precisely these conditions since these observations could not be done because the filters needed were not mounted at the time. This constraint also has severe implications for the completeness of certain programs.

6. The quality of the calibrations gathered during a QSO run is very high. Calibrations are another area where the QSO mode is a clear improvement over the classical mode. Each night, twilight flats are taken with the filters mounted in the wheel so the best frames can be combined to create accurate flatfielding calibrations. The Elixir Team is producing frequent reports on calibrations during the run so the QSO Team always has clear guidelines for acquiring the most needed calibrations and optimizing the final product. About 3 standard fields were observed each night for photometric calibration without introducing large overheads. The number of fields will go down when the photometric calibration from skyprobe is performed. If the observations were done during non-photometric conditions but require photometry, short exposures were automatically generated on these fields and conducted during photometric nights. All the fields observed and requesting photometry have been calibrated in 2001A.

7. The proposal evaluation of the TAC is a fundamental component of a successful queue scheduling model. Since a large majority of the programs requested targets between 10 -14 hr and due to scheduling constraints of other instruments, this resulted in several programs not being started or carried out only partially at the end of the semester (see statisticsbelow). As a global result, the completeness of the programs is lower than anticipated. New guidelines have been issued to the TAC regarding proposals for 2001B.

8. Precise Agency time accounting is possible but not realistic on a short timescale basis. Several constraints dictate what programs should be executed. We have implemented a dynamic statistical tool so we can follow the progress of the programs for each agency and adjust the program selection for the queues. But, in reality, since not all the possibilities in sky conditions are equally represented by the programs of the different agencies, trying to adjust the time distribution on a short timescale (for instance, within a short QSO run) does not make sense. Nevertheless, as showed below, we were able to reach the level of telescope time allocated for each agency by the end of this semester.

9. Monitoring programs are well-suited for the QSO mode. With the system implemented in the Phase 2 tool, repeating specific observations N times within a given period is relatively easy. It is important to understand, however, that time-constrained programs have an impact on all the programs available in the database. Time constraints have the highest weight in the preparation of the queues to be executed. Thus some regular programs might not be included in those queues. More monitoring programs will be available for 2001B and we should acquire more experience in dealing with these requests.

10. Long observation groups (OGs) are more difficult to schedule within a queue than short ones. The 2001A database did not contain enough short OGs even if this was clearly stated in the PH2 tutorial. The end of nights was sometimes particularly difficult due to lack of appropriate, short OGs. Most of the time, the queues for a given night include observations from diverse programs in order to fill the available time. To achieve this, it is generally much easier to build a queue containing short OGs.

11. The operational model used during the QSO run is appropriate. During 2001A, two resident astronomers assumed the coordinator role while several other members of the staff filled the observing task. This model worked quite well because it would be very difficult for the observers to assume all the management of the queue database and the creation of the queues. However, the coordinating load remains quite heavy because the work starts way before the beginning of a run and does not end with it. Fortunately, management of the data itself is made easier by the design of the database and the electronic logbook.

12. Queue observations from the Waimea observing room is much easier than from the summit. Most of the observations for 2001A were carried out from the summit, in order to meet the "two person rule" at the summit. However, service observing is not easy and some nights can be quite busy: execution of the queues, preparation of the pending queues, filling up data logbooks, switching queues if necessary, applying the calibration plan, etc. For several nights, we have instead observed from the Waime control room while another member of CFHT assumed the role of the second person at the summit. It is much more efficient and easy to run in a queue mode from this environment. The only difficulty is to evaluate the weather but this is a task normally carried out by the observing assistant, and this has become also easier with the availability of skyprobe. Full implementation of Waimea observing remains a high priority for the QSO project.

13. Sharing nights between the QSO mode and classical observers is not very efficient. We have tried this twice during 2001A and it was not easy for both parties. In one case, there was also a conflict on the filters necessary for both programs. The transition between both types of observing was also time consuming (slewing, focusing, etc). This practice is strongly discouraged.

14. Semester boundaries are restrictive to the QSO mode. In a queue mode, targets can generally be observed on several runs, sometimes during the entire semester and more. The completion level of certain programs could be higher if the semester boundaries were removed, or at least , if some recycling of the programs completed at a certain level could take place. Balancing the Agency telescope time would be also much easier and would not have such a strong impact on the completion level of programs.

C - Global and Program Statistics

The following table presents some general numbers regarding the queue observations for 2001A (C, F, H, and D-time):

Remarks:

• The fraction of time lost during QSO nights in 2001A due to weather is almost equal to the global fraction usually lost for Mauna Kea.

• The fraction of photometric nights is much smaller than what it is supposed to be for Mauna Kea (~55%). This has resulted in difficulties in making progress for some programs requesting to be done in photometric nights only.

• The time lost due to technical problems is relatively small considered that it was the first semester for the entire NOP system. In fact, an important fraction of the time lost was due to severe problems with both CFH12K shutters prior to the April run. Both shutters have been upgraded and performed basically flawlessly for the rest of the semester.

• A lot of programs were started, except 5 of them. Unfortunately, not many were finished (see figure below). The main reasons behind this are: 1) Not enough time could be scheduled with CFH12K in March/April due to other instruments while a large fraction ~(60%) of QSO programs requested targets between 10-14hr. 2) Six nights were entirely lost to weather in March and April; 3) Our observing efficiency during the March run and the beginning of the April run was lower than expected due to a problem with the new prime focus bonnette controller. Pointing was often (randomly) off by as much as 45" which forced us to look at all the fields requested, a very time consuming operation. ; 4) Importance given to balancing the time for the different Agencies.

• The number of hours of integration time per night can be calculated as follows: [I- Time Observed]/[Night Allocated - Nights lost to Weather/Technique]. For 2001A, we roughly reached 7 hours of observed I-time per night. The number of hours of I-time validated is smaller, however, because about 15% of the observations did not meet the specifications: about ~6 hours per night. During the last three runs, as much as 7.25 hours of I-time were obtained on science targets during the short summer nights.... However, 6.5 hours per night at the moment is still the best round number available. The semester 2001B will be more revealing at this subject because most of the glitches have been fixed and we have acquired more experience with the preparation and execution of the queues.

• About 85% of the observations conducted were found to meet the specifications. A fraction of observations were not validated because of the pointing problem mentioned above. However, most of these unvalidated observations was due to our lack of experience in selecting the programs to be done during rapidly changing conditions while operating without a seeing monitor. We are now a bit more conservative on this and we expect the fraction to go up quite a bit for 2001B. However, the fraction of validated exposures for 2001A is already quite impressive and is certainly higher than what can be hoped for on a time scale of a semester in a classical mode of observation.

D - Agency Time Accounting

Balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues. The figure below presents the Agency time accounting for 2001A. The top panel presents the relative fraction requested by the different agencies, according to the total I-time calculated from the Phase 2 database. The bottom panel represents the relative fraction for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the relative distribution of the total integration time of validated exposures between the Agencies was clearly balanced at the end of the 2001A, within a margin of about 2%. Since the total number of hours validated was about 270 hours, these differences represent about a differential of 5 hours of integration time... Almost a perfect balance of telescope time between the Agencies is surely possible in our QSO operational system!

Remarks:

• The relative fraction for discretionary time is slightly over the relative fraction of time requested; this is due to the availability of programs requesting bad seeing, non-photometric conditions and/or bright time for this Agency. Most of the D-time programs were done in fact during the first QSO run (bright time).

• It is important to notice that balancing the Agency time is somewhat incompatible with the idea of having the highest completion rate for the programs. This is especially true for the completion of some programs toward the end of a semester when time accounting generally becomes a bigger factor in the preparation and execution of the queues.

• Since there is no real correlation between the seeing conditions requested and the program grade, interpreting the relative fraction of time with respect to the grades must be done carefully. For instance, UH fraction of validated I-time for their A programs is ~ 5% to where is should be. However, their C programs are 10% higher than the fraction requested and their most completed C program has been done during excellent seeing conditions, to help balancing the telescope time between the Agencies.

E - Additional Remarks

Although our first semester with the queue mode was quite successful, there are several issues that need to be addressed for 2001B. We will particularly focus on these issues:

• Completion level of programs. This is the most dissapointing result on QSO for 2001A. Our strategy will change slightly for 2001B: we will try to stick with a given program for longer periods of time during a QSO run. It is not always possible to do so (i.e. due to the filter constraint) but we will try to do so. The distribution of A programs is more realistic for 2001B and scheduling is more optimized. On the other hand, we will have more programs with time constraints and those are somewhat difficult to finish at a very high level of completion if the weather conditions are not appropriate.

• Higher observing efficiency. The observing efficiency has been increasing during the last QSO runs of 2001A. It remains one of our highest priorities and we are still developing better ways to prepare and execute queues within the NOP. With the main glitches gone and now with more operational experience, the 2001B semester should be quite revealing on this issue.

• Higher validation rate. We are aiming at the rate of about 90% of validation for all the exposures. This should be easier to achieve following our experience for 2001A. However, the lack of a seeing monitor will always be problematic for quick decision making on the queue selection during an observing night.

• PI/QSO interaction. About two-thirds of the PIs have been contacted by the QSO Team during the semester regarding their programs. However, some mistakes have been done on our side because we should have contacted some PIs and requested more information and we did not. We are certainly planning to remedy to this situation for 2001B.

• Photometric calibrations. We are planning to decrease the number of standard fields observed during photometric nights, as soon as it is showed than accurate photometry can be achieved with the images of Skyprobe. This could result in several hours available for science at the end of a semester. For the time being, however, we will keep observing 3 fields per photometric night.