Queued Service Observing with MegaPrime, WIRCam, and ESPaDOnS:

Semester 2008B Report

04/02/2009

TABLE OF CONTENT

A - Introduction
B - General Comments
C - Global Statistics and Overheads
D - Agency Time Accounting
E - Conclusions


A - Introduction

The Queued Service Observing (QSO) Project is part of a larger ensemble of software components defining the New Observing Process (NOP) which includes NEO (acquisition software), Elixir (data analysis for MegaCam), `I`iwi (data analysis for WIRCam), Upena (data analysis for ESPaDOnS) and DADS (data archiving and distribution). Semester 2008B was the second semester during which three instruments were offered in QSO mode: MegaPrime, WIRCam, and ESPaDOnS.

The main development for semester 2008B was related to the integration of non-sidereal tracking in the QSO mode, for all 3 instruments. After the required software modifications and on-sky tests on all 3 instruments, non-sidereal tracking is now offered for all 3 instruments. The tests confirmed that the telescope's tracking abilities starts to degrade after 120-180 seconds. There is no plan to offer non-sidereal guiding.

2008B was also the last semester for CFHTLS, and the first semester with Large Programs, one on MegaCam and two on ESPaDOnS.


B - General Comments

MegaPrime

The 2008B semester for MegaPrime started with excellent weather and science productivity, but the bad winter season started to affect observations in Oct/Nov, and overall, about half the time was lost to weather! The only major technical failure was due to an electronics board failure, which cost a little over one night of time lost.

  1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is negligible. The system is quite reliable and efficient.
  2. The QSO concept is sound. The possibility of preparing several queues covering a wide range of possible sky conditions (absorption and Image Quality) in advance of an observing night is essential for imaging in the visible, where seeing can change quite a lot, and very quickly. The ensemble of QSO tools allows also the quick preparation of queues during an observing night for adaptation to variable conditions, or in case of unexpected overheads. The QSO mode also makes possible time constrained programs such as the CFHTLS. In 2008B, due to the inclement weather, the validation rate of A+B+C programs (hours validated/hours observed) for MegaPrime is lower than usual, at 84% (usual rates are over 90%).
  3. QSO is well adapted for time constrained programs.The Phase 2 Tool allows the PIs to specify time constraints, even very restrictive ones. Time constrained programs complicate the planning and scheduling but CFHT provides tools for both PIs and Queue Coordinators.
  4. Very variable seeing and non-photometric nights represent the worse sky conditions for the QSO mode with MegaPrime. Snapshot programs and regular programs requesting mediocre conditions (1" to 1.2") are used to cover those conditions. Fields requesting photometric sky conditions but originally done during non-photometric conditions are calibrated on perfect nights. SkyProbe and real-time measurements of the transparency are used to decide what observations should be undertaken.

WIRCam

The 2008B semester for WIRCam went very well. There were no major loss of time due to technical problems; a couple of hours total were lost to small software glitches with WIRCam, and another couple of hours were lost after a fluid leak on the telescope. WIRCam also got lucky and was the instrument the less affected by the winter weather (36% of the time lost).

  1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is very small. Certain operational modes specific to WIRCam, like nodding (target-sky-target...) and chip-to-chip dithering, have higher, unavoidable, overheads but some of them are charged directly to PIs during Phase 2.
  2. The QSO concept is sound. As with MegaPrime, the possibility of preparing several queues covering a wide range of possible sky conditions (absorption and Image Quality) in advance of an observing night results in a very large fraction of the observations done within the specifications. For WIRCam, the sky background is more of a factor although its global variation through the night on Mauna Kea is fairly well known. Seeing is of course another important parameter but variations during the night in the near-IR are generally not as brutal as in the visible.
  3. QSO is well adapted for time constrained programs.The Phase 2 Tool allows the PIs to specify time constraints. We can handle those easily if the weather is cooperative although the introduction of time constrained observations on a large-scale adds up definitive complexity in the scheduling process.
  4. Non-photometric nights represent the worse sky conditions for the QSO mode with WIRCam. An important difficulty on near-IR astronomy is the removal of the sky background. Non-photometric conditions make that operation a more difficult one. Nodding for instance cannot be done. The availability of SkyProbe and real-time measurements of the transparency is extremely valuable and regularly used do decide what observations should be undertaken. Also, the real-time analysis through `I`iwi provides a direct estimate of the extinction through the 2MASS catalog, helping even more the observing process.

ESPaDOnS

The 2008B semester with ESPaDOnS was almost completely free of any technical issues; only the telescope caused a couple of hours of time lost. The last 2 runs were severely affected by excessively snowy weather. Severe winter conditions even delayed the installation of the instrument until the 9th day of the last run. Overall, about 43% of the time was lost to weather, which severely affected the success of one Large Program. Efficiency on the sky was also affected by telescope pointing issues.

  1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is small. Besides telescope pointing difficulties, most of the overheads come from acquiring (finding) the targets, performing full telescope focus sequences for the first 1-3 nights of a run, measuring the Image Quality, and initiating the guiding.
  2. The QSO concept is sound. For ESPaDOnS, and in contrast to our imagers, seeing and extinction are much less of an issue and do not factor much in the preparation of the queues. Queue Coordinators usually prepare one or at most 2 queues per night. The advantage of the QSO mode comes from the ability to schedule observations exactly when they are needed.
  3. QSO is well adapted for time constrained programs. ESPaDOnS observations are characterized by a high demand for time constrained observations and monitoring requirements: in 2008B, almost all programs had special requests for timing. The queues are usually prepared by taking first into account programs with strict time windows. It was not unusual to juggle a program with a target to observe every night, a second program with a target to observe 2-3 times a night with 1-hr gaps between Observing Groups, a third program with a target to observe every 2 or 3 or 4 nights, and 2 programs to execute within a certain time window from one another. There are also programs that necessitates continuous blocks of 4 to 8 hours during the same night. The PH2 tool allows PIs to specify all sorts of time constraints, and add any comment to help the QSO Team select appropriate programs.
  4. ESPaDOnS can deal with non-photometric nights and bad Image Quality Except for very faint targets (fainter than about 13) which can be difficult to find and center with cloudy conditions or highly degraded seeing, most observations can be carried under a very wide range of sky conditions. Extinction and bad seeing reduce the amount of flux getting into the instrument, but the Service Observers compensate by repeating Observing Groups (at no cost to the PIs) to recover some of the lost flux.


C - Global Statistics and Overheads

(1) Global Statistics

The following table presents some general numbers regarding the queue observations for 2008B (C, F, H, L agency for MegaPrime, P agency for the Large Programs , and T, excluding Discretionary time, engineering time, and snapshot programs unless noted otherwise).

Parameter MegaPrime WIRCam ESPaDOnS
Number of Nights
(CFHTLP; calculated from the hours allocated by TACs)
79.3 44.1 38.7
Hours per night 5.5 6.0 7.5
Hours lost to weather and % lost
(% from Number of nights X Hours per night)
239hrs
(49%)
108hrs
(36%)
148hrs
(43%)
Hours used for engineering
or lost to technical problems, and %
32hrs (6.6%) 29hrs (9.7%) 23hrs (6.7%)
QSO Programs Requested 30 (A/B/C) +
2 Snapshots
21 (A/B/C) +
3 Snapshots
14 (A/B/C) +
2 Snapshots
QSO Programs Started 29 (A/B/C) +
3 Snapshots
21 (A/B/C) +
3 Snapshots
14 (A/B/C) +
1 Snapshot
QSO Programs Completed 17 (A/B/C) +
0 Snapshot
14 (A/B/C) +
1 Snapshot
5 (A/B/C) +
1 Snapshot
Total I-time allocated by the TACs
(i.e., A and B programs)
431hrs 259hrs 306hrs
Total I-time validated
(for A+B+C programs)
393hrs 248hrs 256hrs
Completion rate
(A+B+C validated time / A+B allocated time)
91% 96% 84%
Queue Validation Efficiency
(A+B+C time validated / A+B+C time observed)
84% 94% 84%

Notes concerning MegaPrime

Notes concerning WIRCam

Notes concerning ESPaDOnS

This second semester with ESPaDOnS in QSO mode has confirmed the significant differences found in 2008A in terms of queue preparation, handling of priorities (grades/ranks), and particular scheduling difficulties.

(2) Overheads

MegaPrime

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with MegaPrime. These numbers have not changed, and are given as a reference. Overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc. The table below shows that overheads take a maximum of ~35min per night. A short summer night lasts about 9 hours with MegaPrime, so overheads take less than 10%. The number originally expected was 10-15%.

Event Events/night Overhead Total overhead per night
Filter Change ~12 / night 90s /change 1115 seconds
Focus Sequence ~ 0 / night 0 seconds
Dome Rotation > 45deg 5 ? 120s < 600 seconds
Guide Star Acquisition 20 - 30 ? 20s / acq < 600 seconds

Notes:

WIRCam

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with WIRCam. Overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc. The table below shows that overheads take a maximum of ~25min per night. A short summer night lasts about 9.5 hours with WIRCam, so overheads take around 5%, which is quite low.

Event Events/night Overhead Total overhead per night
Filter Change 15 / night 15s /change 225 seconds
Focus Sequence 2 / night 65s / seq 130 seconds
Dome Rotation > 45deg 5 ? 120s < 600 seconds
Acquisition 36 / night 12.5s / acq 455 seconds

ESPaDOnS

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with ESPaDOnS. Overheads during a given night depend mostly on the number of targets and the need to change Observing Mode or not. The table below shows that overheads can take up to 1hr per night, most of that coming from the acquisition stage (pointing the telescope, finding the target, centering the target, starting the guiding). A telescope pointing issue has been confirmed in 2008/2009, and means that more time is needed to acquire targets; the guide camera field of view has to be expanded (which takes more time to readout), and observers take extra care in identifying targets. A short summer night lasts about 9.5 hours with ESPaDOnS, so overheads take around 10%.

Event Events/night Overhead Total overhead per night
Readout Mode Change 3 0 0
Obs Mode Change 1 to 3
(< 1 on average)
7-8min / change 4-5min / night
on average
Telescope Focus Sequence ~ 1-1.5 / night
on average
120-180s / focus seq. 160 - 240 seconds
Image Quality Measurement ~ 6 - 12 / night
on average
30 - 60s / measurement 180 - 600 seconds
Dome Rotation > 45deg 5 ? 120s < 600 seconds
Star Acquisition
(pointing, finding)
12 / night
on average
1-5 min / acq 10 - 30 min

Notes:



D - Agency Time Accounting

(1) Global Accounting

Balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues. After 2 semesters with ESPaDOnS in QSO mode, it was found that this is much more difficult to do with ESPaDOnS than with the other 2 instruments, because a lot of ESPaDOnS programs request strict time constraints or have narrow windows of execution, which takes precedence over Agency Balancing. The queues are basically done according to the requested time constraints and try to follow the programs' ranks as much as possible. The agency balance is whatever it ends up being at the end of the semester. The bad weather for 2008B compounded this difficulty.

MegaPrime

The table below presents the hours allocated by the TACs, requested by the PIs, and validated by the QSO Team for each agency.
MegaPrime C F+O H T L P
Allocated (A+B)73.953695.515475.3
Requested (A+B)73.852.8681.413175.4
Validated (A+B+C)65.550.970.31.4129.375.3
Validated (C)054.6000

The CFHTLS was finished during 2008B, and despite the bad weather, almost all the time requested was observed and validated. The Large Program was also completed. Taiwan did not request all the time that had been allocated by its TAC because one program was a Target-of-Opportunity program with less targets to observe than anticipated. Canada is the agency which did not quite get its share of time with MegaPrime.

WIRCam

The table below presents the hours allocated by the TACs, requested by the PIs, and validated by the QSO Team for each agency.
WIRCam C F+O H T
Allocated (A+B)76.588.163.431.3
Requested (A+B)66.381.963.231.2
Validated (A+B+C)67.694.354.631.2
Validated (C)5.529.21.80

The 10 hours missing for the Canadian agency belong to a program which is actually executed outside of QSO. The French + Opticon agency is ahead, although a fair portion of the validates hours belong to C (overfill) programs.

ESPaDOnS

The table below presents the hours allocated by the TACs, requested by the PIs, and validated by the QSO Team for each agency.
ESPaDOnS C F+O H T P
Allocated (A+B)5262.507.5184.1
Requested (A+B)51.762.507.5181.8
Validated (A+B+C)50.847.607.5150.2
Validated (C)140000

In 2008B, the Canadian and Taiwanese agencies fared much better than the French/Opticon and the Large Programs ones. The poor weather affected timed-constrained programs tremendously: observations which were planned for a specific night and time, or within a certain number of nights from a previous observations, simply became impossible, and it was often impossible to re-schedule an additional observation to compensate.

(2) CFHTLS Accounting

Up to its last semester, the CFHTLS took a large fraction of the I-time allocated for QSO for MegaPrime.

The following table shows that despite the inferior weather, the QSO Team managed to complete the survey.

Survey Programs Hours requested Hours validated
Deep Synoptic L01 40.0 38.3
Wide Synoptic L02 + L05 85.7 85.7
Photometric Grid
for the Wide
L99 5.3 5.3


E - Conclusions

MegaPrime

The 12th semester with MegaPrime in QSO mode went well despite the incredible stormy winter weather. There were no major technical problems. Non-sidereal tracking has been integrated into QSO. Observations are running very smoothly. The CFHTLS has been completed, and one of the 2 MegaPrime Large Programs has been started without any issue.

WIRCam

The 7th semester with WIRCam in QSO mode also went well despite the bad weather. There were no major technical issues with the instrument. Non-sidereal tracking has been integrated into QSO. WIRCam has the highest completion rate and validation efficiency rate.

ESPaDOnS

The second semester with ESPaDOnS in QSO mode went very well, aside from the weather, which severely affected many time-constrained programs, in particular one of the Large Programs. There were no major technical issues with the instrument, but telescope pointing issues are lowering the efficiency on the sky. The completion and validation rates are the lowest of the 3 instruments, due to hard-to-meet time constraints given the impossible weather.