T0005 : no global reprocessing

From: Jean-Charles Cuillandre <jcc_at_cfht.hawaii.edu>
Date: Wed, 12 Mar 2008 16:13:44 -1000

 Hi,

This has been in the air for some time: integrating the latest findings from
SNLS on the photometry accuracy of the photometric flat to be applied for a
global reprocessing of the LS data since first light, while addressing other
issues such as zero points stability and global recipe consistency.

We are however facing issues that are forcing us to take some steps back on
this plan. I already consulted Yannick, Christian and Chris today about what's
best, and it appears wise to postpone to T0006 the global reprocessing. Please
feel free as a SG member to voice your view on this decision. Here are the
elements that dictated the choice:
        
        1) Bandwidth:
           The southern cross network access that was supposed to bring a
           30 to 40Mb/sec CADC-CFHT connection did not materialize: we're
           still getting only 3Mb/sec max. With 25000 frames of 700Mb each
           to reprocess and transfer both ways, that's a total of 120 days,
           not counting in there the overheads for running Elixir and the
           DADS distribution. So, we'd be looking at 5 or even 6 months to
           get all the data from/to CADC, that is not acceptable as we
           must stick to our commitment to our community and release T0005
           this spring (May, or June at the latest).

        2) Flat recipes:
           SNLS has strong requirements on the photometry and they have
           addressed some limitations caused by the Elixir native processing.
           But all the other programs can fully deliver with the 3% photometry
           precision we have right now. There is more work needed to validate
           the SNLS recipes as they get integrated in Elixir: preliminary
           results (by Terapix and S. Arnouts on photometric redshifts) with the current understanding of the flats strategy show no gain over
           the latest Elixir release.

        3) Data consistency:
           The current set of data at Terapix used for T0004 has various
           flavors of Elixir flats: B1, B2, and B3. The global reprocessing
           had the advantage of making all data processed the same way (B4).
           But then again, this is not (or yet anyway) a limitation for programs
           other than SNLS which has its own data set under control. B1 was
           however not the best, and I believe the B1 data set (which is
           pretty minimal, 10 or 12 runs) could be reprocessed with B3.
           B4 has been active since December, hence the past 3 runs will be
           reprocessed with B3 as well and pushed at CADC (Terapix has yet
           to retrieve any data from the second half of 06B, all of 07A, all
           of 07B and 08A - and they can go ahead right away with 06B/07A and
           the first four runs of 07B - this is urgent as the Wide observing
           strategy needs data quality assesment to define the Wide patching
           in 08A).

        4) Zero points:
           Clearly this was an issue and my recent message explained some
           of the gains on that one. It is still important to address this for
           T0005, hence I will re-apply that method to the ZPs of data
           processed with the B3 Elixir data. Terapix will be provided with
           an updated set of ZPs for all runs since first light, and they will
           have to integrate this somehow in their pipeline: the Elixir data
           will not reflect for now that change (i.e. nothing will change on
           the CADC files, or even the Elixir files already at Terapix).

In summary:
        o The global reprocessing will not happen, at most it will be the
           first year of LS data (03B/04A) that will be reprocessed with the
           best flat Elixir recipe to date (B3). The past three runs since B4
           was activated will be reprocessed with B3.
        o All runs zero points will be updated but not within the FITS files.

What we are facing for T0006:
        1) Address the bandwidth issue. 6 months simply isn't reasonable
           to consider for a reprocessing.
        2) Fully characterize and understand why the gain achieved by SNLS
           does not realize when integrated in Elixir. There is a lot of
           valuable work there and the data will eventually benefit from it.
        3) Fringing, fringing, fringing...
        4) Amplifier stability: that is a hard one, but it is becoming a
           limitation and it needs to be looked at.
        5) Use the Deep fields as the new photometric standards. The absolute
           calibration program is about to be analyzed by CFHT and SNLS.

By taking this decision now, not much time from any of us has been wasted yet.
Hopefully you will all agree this is the best path to take at this point to
serve our users.

Please let me know if you have any comments,

                                                Jean-Charles.
Received on Wed Mar 12 2008 - 16:13:44 HST