QSO report 2005B

The Queued Service Observing (QSO) Project is part of a larger ensemble of software components defining the New Observing Process (NOP) which includes NEO (acquisition software), Elixir (data analysis) and DADS (data archiving and distribution). The semester 2005B was the first semester in the history of QSO that two instruments were offered in that mode: MegaPrime and WIRCam. The latter was offered as a shared-risk instrument and a large fraction of the time scheduled was for engineering and commissioning purpose. Bad weather was another important factor during the semester; fortunately we got an exceptional December run with MegaCam which allowed us to catch up on several programs. Again, the 2005B semester was extremely complex on the scheduling process since several programs requested time constraints. We worked really hard to make sure that everything fits as much as possible which was not an easy task with the uncooperative weather!

They are several very positive development that occurred during this semester: 1) The efficiency on the sky for MegaCam has really been optimized with better and fast guiding acquisition and the implementation of a very reliable automated focus model; 2) The image quality of MegaCam is now fully optimized, producing spectacular results across the entire field of view; 3) WIRCam was commissioned in QSO mode and produced right away an appreciable amount of high quality data requested by PIs through a new version of the Phase 2 Tool adapted for WIRCam. For 2006A, about 150 nights of QSO time is scheduled... quite a challenge!

The 2005B semester for MegaPrime was quite successful despite again significant time lost to bad weather and technical problems. The weather was very average but the run in December was so exceptional that we were able to catch up on the observations. Fortunately since the last run of 12 nights was 85% lost to weather! The middle of the semester was difficult with lots of cloudy periods and unstable bad seeing. The semester 2005B included very difficult scheduling issues with time critical observations from several PI programs and, of course, CFHTLS. A positive aspect of the 2005B was the continued improved observing efficiency due to reduction of the overheads by the guide probe motion and focus sequences. This observing efficiency for instance was clearly demonstrated in December when the weather was nice; 8.2 hours per night were validated and for most of the nights, operational overheads were reduced to less than 10%.

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is completely negligible. This is a complex system and we have worked real hard to reduce the overheads on this. The system is quite reliable and efficient.

2. The QSO concept is sound. With the possibility of preparing several queues covering a wide range of possible sky conditions in advance of an observing night, a very large fraction of the observations (>90%) were done within the specifications. The ensemble of QSO tools allows also the quick preparation of queues during an observing night for adaptation to variable conditions, or in case of unexpected overheads. The introduction of the CFHTLS and several other PI programs with time constrained observations on a large-scale adds significant complexity to queue scheduling and requires much more work on planning of the runs. For 2005B, the global validation rate (validated/observed) for MegaCam was excellent (section C)

3. QSO is well adapted for time constrained programs. The Phase 2 Tool allows the PIs to specify time constraints. Two of the components of the CFHTLS have very restrictive time constraints. We can handle those easily if the weather is cooperative (of course!) although the introduction of time constrained observations on a large-scale adds up definitive complexity in the scheduling process.

4. Very variable seeing and non-photometric nights represent the worse sky conditions for the QSO mode. In 2005B, we were still a bit short on "shapshot" programs or regular programs requesting mediocre conditions (1" to 1.2"). As a result, we were often forced to observe programs in conditions worse than requested. Again, we were able to calibrate all the fields requesting photometry but originally done during non-photometric conditions. The availability of Skyprobe and real-time measurements of the transparency is extremely valuable and regularly used do decide what observations should be undertaken.

We offered WIRCam as a shared-risk instrument for 2005B. A good fraction of the telescope time allocated to WIRCam was in fact used for engineering and commissioning but we were able to get some good quality data for several programs. Unfortunately, when the instrument became more stable during the last run, weather conditions degraded... The statistics given in the later section must then be taken with a grain of salt since the number of nights was small and that the instrument was not available all the time for QSO.

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is already quite small for WIRCam. In fact, there was a lot of optimization work done to minimize operational overheads. This is a complex system but reliable and efficient. At the moment, most of the overheads are related to guiding and the much longer readout overhead than we would like (10 seconds). Certain operational modes specific to WIRCam, like nodding (target-sky-target...) and chip-to-chip dithering, have longer overheads but some of them are charged during Phase 2; those modes have been tested and work very well.

2. The QSO concept is sound. As with MegaCam, the possibility of preparing several queues covering a wide range of possible sky conditions in advance of an observing night result in a very large fraction of the observations done within the specifications. For WIRCam, the sky background is more of a factor although its global variation through the night in Mauna Kea is fairly well known. Seeing is of course another important parameter but variations during the night in the near-IR are generally not as brutal as in the visible.

3. QSO is well adapted for time constrained programs. The Phase 2 Tool allows the PIs to specify time constraints. We can handle those easily if the weather is cooperative (of course!) although the introduction of time constrained observations on a large-scale adds up definitive complexity in the scheduling process.

4. Non-photometric nights represent the worse sky conditions for the QSO mode with WIRCam. An important difficulty on near-IR astronomy is the removal of the sky background. Non-photometric conditionsmake that operation a more difficult one. Nodding for instance cannot be done. The availability of Skyprobe and real-time measurements of the transparency is extremely valuable and regularly used do decide what observations should be undertaken. Also, the real-time analysis through Elixir provides a direct estimate of the extinction tthroughthe 2MASS catalog, helping even more the observing process.

The following table presents some general numbers regarding the queue observations for 2005B (C, F, H, K, L, and T, D-time, excluding snapshot programs). Note: 1 night is 9.5 hours.

The fraction of time lost during QSO nights in 2005B due to weather and technical problems is about 30% of the semester. This is significantly larger than the global fraction expected (~20%). In fact, the number given above might even be a bit underestimated since we again had to observe for several nights with winds close to the acceptable limit which introduced bad wind shake and degraded the image quality.

The time lost due to engineering and technical problems is still important this semester for is getting better and better. The number given is also approximate because not everything could be taken into account (for instance, it is often difficult to untangle the time used to engineering during bad weather and count that as time lost for science).

The global validation rate (validated/observed) is excellent ~90%. This is the best we have done with MegaCam. During some of the good runs the efficiency was way above 90%. Part of this comes from the now excellent focus model with automatically adjust the focus between exposures and keep the image quality optimized. The most difficult conditions come from very rapidly changing seeing; we were faced with several nights like that during 2005B.

It is difficult to establish the correct statistics for WIRCam for 2005B since science time was mixed with enegineering and commissioning. The validation rate was around 80% and most of that, of course, was due to failures happening during the observations with diverse sub-systems. The completion A+B was 48%, not too surprising again. The coming semester will be more helpful and descriptive for getting real statistics.

The figure below presents the completion level for all of the programs in 2005B, according to their grade:

The global completion level with MegaCam for A programs is ~95% while B programs were done at ~77% . These values are excellent, the best we have ever done with MegaCam. The much improved observing efficiency helped in achieving this as well as an exceptional run in December. With a better run in January (85% lost), A+B programs would have been completed close to 95%. Note that one low ranked B program which requested 50 hours at < 0.8" got a completion of 65%. The B program at 0% is a Target-of-opportunity program but no observations were requested during the entire semester.

Remarks:

The global completion level for A+B programs is not too good (~48%) but this is not at all surprising since a very large fraction of the QSO time scheduled in 2005B was in fact used for engineering and commissioning of the camera. Completion of C programs is not bad and reflect the fact that the seeing in the last run was very bad.

3) Overheads

The following table include the main operational overheads (that is, other than readout time of the mosaic) with MegaPrime during the semester 2005B. This is given as a reference; overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc. globally, the operational overheads constitute now about 10-15% of an observing night, the number originally expected before MegaPrime observations started.

Overheads to filter changes are large and constitute the main difference with CFH12K. The total time for a filter change is about 127 seconds but this is done in parallel during readout or while the telescope is moving (so, we do not always have an overhead for a filter change). The global overheads also depends strongly on the number of standard stars observed for a given night and also if switching from a queue to another is necessary (since overheads due to filter change are minimized within a specific queue). Until we have another system, this overhead will remain with us....

Focus sequences have been almost completely removed from our operations. The auto-focus model is available and contributes to significantly increase the time we spend observing instead of focusing. We take a few sequences during the first nights of a run to confirm the zero points of the model; other than that we just operate with the focus model.

Overheads due to dome rotation are again minimized as much as possible within a specific queue. Note that a lot of rotation is necessary to reach standards stars on the equator when we observe northern targets. Hopefully, the use of the Deep survey fields from CFHTLS as secondary standards will help on this. Rotation of the dome is now optimized and cannot be made faster.

Guide star acquisition is fully automated and except from some rare problematic acquisitions, it works really well. Acquisition tends to take longer when the seeing is bad or cirrus are present. Programs with frequent guide star acquisition with short exposure strategy (e.g. sequences for the Very Wide survey) increase the global overheads). The main overhead related to the guide star acquisition has been reduced dramatically in 2005A by accelerating the probe motion. Dithering patterns offsets for instance are now completely hidden in the readout time, which was not the case in the past.

Note that overheads for calibrations (standard stars and Q98 short exposures for photometric purposes) are not included in this table. For 2005B, we observed about 2 standard star fields during a photometric night (12 minutes / fields due to filter changes).

WIRCam

Gigantic efforts have been made to reduce the overheads for WIRCam during the runs of semester 2005B. It is too early to give definitive numbers yet because things are still moving. The overheads are still too large but we can expect several improvements in 2006A.

Balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues. The figure below presents the Agency time accounting for 2005B. The top panel presents the relative fraction allocated by the different agencies (program A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the relative distribution of the total integration time of validated exposures between the different Agencies was balanced at the end of the 2005B.

The global distribution between the Agencies is really excellent. CFHTLS is a bit ahead but it compensates somewhat for some lower stats during previous semesters. The Taiwanese agency is a bit later despite finishing all of their programs. this is because they had a B program on GRB that was not sued at all for lack of interesting targets during the semester.

WIRCam

As with MegaCam, balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues for WIRCam. The figure below presents the Agency time accounting for 2005B. The top panel presents the relative fraction allocated by the different agencies (program A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the relative distribution of the total integration time of validated exposures between the different Agencies was not very balanced for WIRCam at the end of the 2005B.

Remark:

As showed in the plots, the relative distribution of the total I-time between the different Agencies was not very balanced for WIRCam at the end of the 2005B. This is not at all surprising: we had a small number of nights, small number of programs and a large fraction of the time could not be used by QSO for engineering/commisionning purposes. After the first two runs, the balance was much better but bad conditions and problems in the last run resulted in discrepancies at the end. Nothing to worry about; this is just the result of the particular circumstances of running QSO during the first semester of a shared-risk instrument.

CFHTLS occupies a large fraction of the I-time allocated for QSO for MegaCam. The following figures show the time accounting for the different CFHTLS components for 2005B:

Since each component of the survey is divided into two programs, the global fractions are given in the following table:

Survey	Programs	Fraction Requested	Fraction Validated for 2005A
Deep Synoptic	L01 + L04	42.6 % + 2.4% = 45.0%	44.3% + 2.3% = 46.6%
Wide Synoptic	L02 + L05	20.5% + 26.9% = 47.4%	23.7% + 21.4% = 45.1%
Very Wide	L03	7.5%	7.1% + 11.7% = 8.2%

Remark:

For the first time, the final time distribution of validated data within CFHTLS is very close to the respective allocation of each survey before the semester. A good reason for this is that the pressure coming from the PI programs on the Very Wide W1 field was lower than usual. Also, the Wide has relaxed some airmass constraints on some filters which greatly helped in scheduling more observations.

Our sixth semester with the queue mode with MegaPrime was excellent. The fraction lost to bad weather was again too high but gains in observing efficiency resulting from MegaCam improvements really helped achieving high completion and validation levels. During the exceptional December run, we validated 8.2 hours of data per night. Balance of the Agencies was quite satisfying. The balance between the LS surveys was much better this time around.

A very positive conclusion: QSO is ready to go with WIRCam! It took significant efforts to develop the phase 2 tool and apply other modifications to existing software components but already, we were able to gather good, high quality data during the engineering/commissioning period for WIRCam. Reducing overheads remain a major objective i the coming months but essentially everything is ready to go on the QSO side for the next semesters.

Parameter	Number
Total number of Nights	105
Nights lost to weather	~27 (~26%)
Nights lost to (engineering + technical) problems	~ 5 (~4%)
QSO Programs Requested	40 (+ 3 snapshots)
QSO Programs Started	35
QSO Programs Completed	21
Total I-time requested (hr.) (A+B+C)	684
Total I-time validated (hr.) (A+B+C)	538 (~79%
Completion A+B Programs	88%
Queue Validation Efficiency	~ 90 %

Event	Events/night	Overhead	Total overhead per night
Filter Change	15 - 25/ night	90s /change	1500 - 2200 seconds
Focus Sequence	~ 0 / night	200s / seq	0 seconds
Dome Rotation > 45 d	5 ?	120s	< 600 seconds
Guide Star Acquisition	20 - 30 ?	20 s / acq	< 600 seconds