Reprocessing procedure and timing

From: Jean-Charles Cuillandre <jcc_at_cfht.hawaii.edu>
Date: Thu, 14 Feb 2008 11:08:32 -1000

 Hi,

Please read on, and then there's a question at the end.

The test data set (W4 ugriz in 06A/B and Deep and Wide for 04Bm03)
have been made available to Terapix at CFHT for evaluation.

This time, the set is final in what was intended to be accomplished for T0005:

                - New flats active (version B.4 based on SNLS collaboration)

                - New Zero Points remeasured (and optimized) based on these
                  latest flats (flats version B.4). The ZPs were derived
                  with new color terms injected in Elixir (consistent
                  analyses throughout).

                - Fringe patterns rebuilt with the new flats to avoid
                  highly variable sky background.

                - Color equation terms updated (SNLS origin) for FITS headers.

For the reprocessing, lists of essential data to proceed with the Deep,
Wide and Very Wide data, as well as i' and z' PI data have been built:
this should help the retrieval, minimizing the set to what we really need.

The reprocessing procedure with go through the following steps:

        1) DADS retrieves raw data from CADC (~3 runs at a time)
        2) Elixir rebuilds the fringe patterns for the retrieved run(s)
        3) DADS processes the Deep/Wide/VW data and push them to CADC

Some handshaking is needed between Elixir and DADS for all three steps
as I must wait for data to be on disk to proceed, and they must wait
for Elixir to be done to go to step 3. I'll build some simple page
that should allow all of us to see the progress on this.

What we have on disk at CFHT is all of 07A and 07B, one 04B run,
two 06A runs, one 06B run. The 5 remaining 06B runs are being
downloaded right now. Proceeding with 06B/07A/07B first is of
great importance for the Wide as global evaluation of the data
set has not yet taken place and observing strategy for 2008
depends on it.

Due to backloading and disk storage issues, we can not do the Wide
first then Deep&VW later on: the granularity of the reprocessing has
to be the observing runs (because of the re-creation of fringe patterns).
So, you'll be getting all three LS components at once for a given run
(PI data will be done when we're all done with the LS, it'll be simpler
as the fringe patterns will be existing then).

One question for CADC: we could start the reprocessing on 07A and
uploading to CADC tomorrow/next week, but do we want to replace old
files yet? Since time is of the essence and download time is the
limiting factor, I recommend we move on with the reprocessing of
the LS asap for 07A, then 06B, then 07B. My feel is that it'll
take more than the small test set to really validate this Elixir
B4 release and having 06B/07A/07B at Terapix through CADC would be a
good thing.

                        Comments?
                                                Jean-Charles.
Received on Thu Feb 14 2008 - 11:08:32 HST