Re: T0005 : no global reprocessing

From: Jean-Charles Cuillandre <jcc_at_cfht.hawaii.edu>
Date: Thu, 13 Mar 2008 14:42:10 -1000

 Hi,

That CFHTLS news section is handy:

http://www.cfht.hawaii.edu/Science/CFHTLS-DATA/cfhtlsgeneralnews.html#0011

This was done under the demand from SNLS back in 2004 for the prep of the
first year data set paper. This means all B1 data were replaced by the B2
version that summer of 2004. I am checking with Terapix that all their data
on disk from spring 2003 to summer 2004 were indeed retrieved after that new
data set became available at CADC.

If confirmed, this means B1 data have never been part of any CFHTLS release.

                                                        Jean-Charles.

> Hi,
>
> This has been in the air for some time: integrating the latest findings from
> SNLS on the photometry accuracy of the photometric flat to be applied for a
> global reprocessing of the LS data since first light, while addressing other
> issues such as zero points stability and global recipe consistency.
>
> We are however facing issues that are forcing us to take some steps back on
> this plan. I already consulted Yannick, Christian and Chris today about what's
> best, and it appears wise to postpone to T0006 the global reprocessing. Please
> feel free as a SG member to voice your view on this decision. Here are the
> elements that dictated the choice:
>
> 1) Bandwidth:
> The southern cross network access that was supposed to bring a
> 30 to 40Mb/sec CADC-CFHT connection did not materialize: we're
> still getting only 3Mb/sec max. With 25000 frames of 700Mb each
> to reprocess and transfer both ways, that's a total of 120 days,
> not counting in there the overheads for running Elixir and the
> DADS distribution. So, we'd be looking at 5 or even 6 months to
> get all the data from/to CADC, that is not acceptable as we
> must stick to our commitment to our community and release T0005
> this spring (May, or June at the latest).
>
> 2) Flat recipes:
> SNLS has strong requirements on the photometry and they have
> addressed some limitations caused by the Elixir native processing.
> But all the other programs can fully deliver with the 3% photometry
> precision we have right now. There is more work needed to validate
> the SNLS recipes as they get integrated in Elixir: preliminary
> results (by Terapix and S. Arnouts on photometric redshifts) with the current understanding of the flats strategy show no gain over
> the latest Elixir release.
>
> 3) Data consistency:
> The current set of data at Terapix used for T0004 has various
> flavors of Elixir flats: B1, B2, and B3. The global reprocessing
> had the advantage of making all data processed the same way (B4).
> But then again, this is not (or yet anyway) a limitation for programs
> other than SNLS which has its own data set under control. B1 was
> however not the best, and I believe the B1 data set (which is
> pretty minimal, 10 or 12 runs) could be reprocessed with B3.
> B4 has been active since December, hence the past 3 runs will be
> reprocessed with B3 as well and pushed at CADC (Terapix has yet
> to retrieve any data from the second half of 06B, all of 07A, all
> of 07B and 08A - and they can go ahead right away with 06B/07A and
> the first four runs of 07B - this is urgent as the Wide observing
> strategy needs data quality assesment to define the Wide patching
> in 08A).
>
> 4) Zero points:
> Clearly this was an issue and my recent message explained some
> of the gains on that one. It is still important to address this for
> T0005, hence I will re-apply that method to the ZPs of data
> processed with the B3 Elixir data. Terapix will be provided with
> an updated set of ZPs for all runs since first light, and they will
> have to integrate this somehow in their pipeline: the Elixir data
> will not reflect for now that change (i.e. nothing will change on
> the CADC files, or even the Elixir files already at Terapix).
>
> In summary:
> o The global reprocessing will not happen, at most it will be the
> first year of LS data (03B/04A) that will be reprocessed with the
> best flat Elixir recipe to date (B3). The past three runs since B4
> was activated will be reprocessed with B3.
> o All runs zero points will be updated but not within the FITS files.
>
> What we are facing for T0006:
> 1) Address the bandwidth issue. 6 months simply isn't reasonable
> to consider for a reprocessing.
> 2) Fully characterize and understand why the gain achieved by SNLS
> does not realize when integrated in Elixir. There is a lot of
> valuable work there and the data will eventually benefit from it.
> 3) Fringing, fringing, fringing...
> 4) Amplifier stability: that is a hard one, but it is becoming a
> limitation and it needs to be looked at.
> 5) Use the Deep fields as the new photometric standards. The absolute
> calibration program is about to be analyzed by CFHT and SNLS.
>
> By taking this decision now, not much time from any of us has been wasted yet.
> Hopefully you will all agree this is the best path to take at this point to
> serve our users.
>
> Please let me know if you have any comments,
>
> Jean-Charles.
Received on Thu Mar 13 2008 - 14:42:10 HST

This archive was generated by hypermail 2.3.0 : Thu Jul 27 2017 - 17:52:27 HST