Queued Service Observing with MegaPrime: Semester 2004B Report 02/03/05

A - Introduction

The Queued Service Observing (QSO) Project is part of a larger ensemble of software components defining the New Observing Process (NOP) which includes NEO (acquisition software), Elixir (data analysis) and DADS (data archiving and distribution). The semester 2004B was the fourth semester for which MegaPrime was used within the NOP system and it was by far the best queue semester with MegaCam. Weather was good for most of the semester (except at the end) and technical problems were less severe as with the past semesters. The 2004B semester was extremely complex on the scheduling process since several programs requested time constraints. We worked really hard to make sure that everything fits as much as possible! A very good number of programs were completed, our global efficiency improved, and the balance for the different Agencies was satisfying. The overheads with MegaPrime are still longer than with CFH12K so we are not as efficient on the sky as we were in the past, although guiding acquisition is now much more reliable.  We are expecting overheads to go down early in 2005 since modifications to the guide probes will be applied.  In conclusion, a very good semester and hopeful for the next one!  In 2005B, we will also start operating WIRCam in a queue mode so it's going to be another challenge, with CFHT being operated in queue mode for ~300 nights per year....

B - General Comments

The 2004B semester went really well, although we would like to have less time lost to bad weather and technical problems, as always. The weather was generally great except for the last 1.5 run.  The run in September was exceptional with photometric nights and stable seeing.  After two runs only, we had already validated 80% of the total number of hours validated in the 2004A semester!  Unfortunately, the last run was much more difficult and we could not complete all the programs we wanted to finish.  The semester 2004B included a lot of very difficult scheduling issues related to the presence of several (13) PI programs with some time critical observations; in the context of CFHTLS which represented 50% of the total time in QSO, this added considerable complexity to the scheduling process. A positive aspect of the 2004B is that the CFHTLS Steering Group got together more to discuss priorities for a given run and transmitted those priorities to the QSO Team.  Also, of course, the  accidental discovery of the  improved image quality of MegaCam by flipping L3 in the Wide-field corrector is another very positive result coming out of 2004B!

Some general remarks on QSO in general for the semester 2004B:

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is very small. This is a complex system and we have worked real hard to reduce the overheads on this. Glitches appear from time to time, mostly on the guide star acquisition, but the system is quite reliable and efficient.

2. The QSO concept is sound. With the possibility of preparing several queues covering a wide range of possible sky conditions in advance of an observing night, a very large fraction of the observations were done within the specifications. The ensemble of QSO tools allows also the quick preparation of queues during an observing night for adaptation to variable conditions, or in case of unexpected overheads. The introduction of the CFHTLS and several other PI programs with time constrained observations on a large-scale adds significant complexity to queue scheduling and requires much more work on planning of the runs. For 2004B, the global validation rate (validated/observed) was very good, one of the highest ever since we started QSO. A discussion on this is included in section C.

3. QSO is well adapted for time constrained programs. The Phase 2 Tool allows the PIs to specify time constraints. Two of the components of the CFHTLS have very restrictive time constraints. We can handle those easily if the weather is cooperative (of course!) although the introduction of time constrained observations on a large-scale adds up definitive complexity in the scheduling process.

4. Very variable seeing and non-photometric nights represent the worse sky conditions for the QSO mode. In 2004B, we were still a bit short on "shapshot" programs or regular programs requesting mediocre conditions (1" to 1.2"). As a result, we were often forced to observe programs in conditions worse than requested, mostly by the end of the semester. Again, we were able to calibrate all the fields requesting photometry but originally done during non-photometric conditions. The availability of Skyprobe and real-time measurements of the transparency is extremely valuable and regularly used do decide what observations should be undertaken.

5. Observations of moving targets is feasible in a queue mode. During the 2003A semester, we implemented a way of preparing observations for moving targets in our Phase 2 Tool (ephemeris tables). The process is a bit laborious but works really well and several programs have used this option again for 2004B as well. Non-sidereal guiding is not yet offered.

C - Global Statistics, Program Completeness, and Overheads

1) Global Statistics

The following table presents some general numbers regarding the queue observations for 2004B (C, F, H, K, L, and T, D-time, excluding snapshot programs). Note: 1 night is 9.5 hours.

Remarks:

• The fraction of time lost during QSO nights in 2004B due to weather and technical problems is about 29% of the semester. This is larger than the global fraction expected (~20%). In fact, the number given above might be a bit underestimated since we had to observe for several nights with winds close to the acceptable limit which introduced bad wind shake and degraded the image quality. 

• The time lost due to engineering and technical problems is, of course, still important this semester. The number given is also approximate because not everything could be taken into account (for instance, it is often difficult to untangle the time used to engineering during bad weather and count that as time lost for science). It is important to note that most of the time lost to technical problems occurred at the beginning of the semester: we are getting better!

• The global validation rate (validated/observed) is now quite acceptable  ~88%.  It fact, some efficiency was lost during two nights when we had to operate with only half of the mosaic.  Up until the last run (very unstable weather), the efficiency was above 90%.  Considering how complex some of those programs were in 2004B, this is not bad at all! It might be very difficult to get higher than this but we'll always try.

• The total number of hours OBSERVED during the semester is 621 hrs.  We then find [ 621 hrs /(116) ] ~ 5.4 hours /nights of observations.  Considering that 29% of the time was lost, which is 9% higher than the nominal 20% assumed for 6 hours per night of queue observation, the assumption of 6 hrs/nights of time is still valid.  The overheads remain a bit higher than the 17% or so assumed; that's why it is a top priority to bring them down in 2005A.
  

2) Program Completeness

The figure below presents the completion level for all of the programs in 2004B, according to their grade:

 

Remarks:

• The global completion level of the programs is quite good, for both A (97%) and B (~67%) programs. The completion level would have been even higher than this with a couple of good nights at the end of the semester, since several programs have completion level in the range of 85 - 100%.

•  Note: The A program completed at 50% is one of the programs for the Very Wide Survey; this was the choice of the VW coordinator since the other (B) program was done at 100%.
  

3) Overheads

There is no doubt that the overheads with MegaPrime are more important than with CFH12K. The following table include the main operational overheads (that is, other than readout time of the mosaic) with MegaPrime during the semester 2004B. The numbers have not really changed since 2003B; they are expected to change for 2005A since modifications to the guiders (software and hardware) are being made. This is given as a reference; overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc.

• Overheads to filter changes are large and constitute the main difference with CFH12K. The total time for a filter change is about 127 seconds but this is done in parallel during readout or while the telescope is moving (so, we do not always have an overhead for a filter change). The global overheads also depends strongly on the number of standard stars observed for a given night and also if switching from a queue to another is necessary (since overheads due to filter change are minimized within a specific queue). Until we have another system, this overhead will remain with us....

• Focus sequences are more frequent with MegaPrime because the camera seems to be more sensitive to position and temperature changes. The auto-focus feature is almost available and should contribute to significantly increase the time we spend observing instead of focusing...

• Overheads due to dome rotation are again minimized as much as possible within a specific queue. Note that a lot of rotation is necessary to reach standards stars on the equator when we observe northern targets. Hopefully, the use of the Deep survey fields from CFHTLS as secondary standards will help on this. Rotation of the dome is now optimized and cannot be made faster.

• Guide star acquisition is now fully automated and in general, works really well. Acquisition tends to take longer when the seeing is bad or cirrus are present. Programs with frequent guide star acquisition with short exposure strategy (e.g. sequences for the Very Wide survey in i band) increase the global overheads). The main overhead related to the guide star acquisition is coming from the probe motion. We are working on making them maybe 3 or 4 times faster which should provide a spectacular gain in efficiency.

Note that overheads for calibrations (standard stars and Q98 short exposures for photometric purposes) are not included in this table. For 2004B, we observed about 2 standard star fields during a photometric night (12 minutes / fields due to filter changes). 
 
D - Agency Time Accounting

1) Global Accounting

Balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues. The figure below presents the Agency time accounting for 2004B. The top panel presents the relative fraction allocated by the different agencies (program A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the relative distribution of the total integration time of validated exposures between the different Agencies was balanced at the end of the 2004B.

Remarks:

• The global distribution between the Agencies is quite good. The Legacy Survey was a bit late due to the unfortunate weather in the last run. The UH Agency is slightly ahead: that was a conscientious effort from the QSO Team in order to compensate for the slightly late statistics for the 2004A semester.  The balance achieved at the end is quite remarkable since by far the scheduling issues were much more complex than usual in 2004B.

•  It is often difficult to reach the fraction requested for the small Agencies, because they have often a limited number of programs and they do not cover a wide range of conditions. KAO is an exception during this semester since most of their programs were requesting less than average conditions.

2 ) CFHTLS Accounting

The following figures show the time accounting for the different CFHTLS components:

Since each component of the survey is divided into two programs, the global fractions are given in the following table:

• The distribution of time validated still greatly favors the Deep Survey although this is not as bad as in the past. This is essentially due to the time critical observations for the Supernova search. When weather is worse than expected, the time constraints become even more weightful with respect to the other programs, in particular compared to the Wide Survey since targets for both of these surveys are close on the sky. The IQ constraint also for the first and last nights of the run is also relaxed for the SNe program so that helped obtaining data while other CFHTLS programs could not be done. With that in mind, it is probably not realistic to propose to increase the time sampling of the supernovae search without compromising the other surveys. Finding a way to achieve a good balance between the different components of the CFHTLS remains a fundamental goal for the upcoming semester.

• Although smaller in terms of time allocated, the Very Wide survey has some strong time critical constraints too. In particular, a sequence of observations necessitates more than 3 consecutive hours of clear, decent seeing weather. This introduces an additional risk for QSO, even worse when the weather is unstable. However, the fraction of time achieved for 2004B is satisfying.

• The Wide Survey is greatly at a disadvantage in the context of time constrained programs on a large scale. The situation would have been much better if the weather would have been good at the end of the semester (and considering that almost an entire run was done without any Wide observations at the request of the coordinators since they wanted to wait for the definitive flipping of L3). However, the 50 hours or so on W1 were done in less than 2 runs in 2004B so when weather is good, the Wide survey can get a lot of data too!
  

E - Conclusion

Our fourth semester with the queue mode with MegaPrime was quite a good one! The number of programs completed is quite high and the completion rate of 97% for A programs is quite remarkable. The Queue efficiency has also improved and balance of the Agencies was quite satisfying. Improving efficiency remains a high priority, in particular by increasing the validation rate and implementing the auto-focus feature, and we are hopeful that 2005A will be also quite productive on the science side.