Hi John,
> Ingestion of the reprocessed files will also replace the existing
> metadata for the old files. If you wish to preserve this metadata, it
> will require some work on JJ's part before we know if this is
> practical. Simply making a snapshot of the database tables will
> preserve the metadata, but won't provide a queriable interface. JJ is
> on vacation, and won't be back until the 24th: he is the only one who
> can determine the feasibility of this.
I suggest for now to take a snapshot of the db as of today (data could
start flowing tomorrow or the day after). Then we'll figure things out
at a later date if needed.
Yannick, I understand that in any case, Terapix preserves the headers
of all files processed at a given point in the pipeline throughout the
years?
Jean-Charles.
>
> J.
>
> P.S. Could someone please add me to the dog mailing-list? This is
> something I'd asked for at the last videocon.
>
>
>
> Jean-Charles Cuillandre wrote:
> >
> > Thus far, the Elixir code has not been changed (being conservative
> > at ensuring continuity in data delivered to our users), only the
> > master detrend files have changed throughout the years.
> >
> > To answer your questions:
> > (i) Yes, the header has the Elixir version in it
> > See
> > http://www.cfht.hawaii.edu/Instruments/Imaging/MegaPrime/dataprocessing.html#P1
> > for the comment fields:
> > COMMENT Elixir: processing performed _at_ CFHT, version 2.0
> > EL_SYS = '2.0 ' / Elixir System Version
> >
> > This is not changed for the current reprocessing. When we
> > do alter the fringing recipe (for the next release), then
> > this will change for example.
> >
> > (ii) Elixir is under CVS, we could go back on our tracks if
> > needed.
> >
> > The new reprocessing is limited to two main elements unrelated to the
> > Elixir code:
> >
> > - New flats (B4 version based on SNLS analysis)
> > - New zero points
> >
> > The new flat can be trace in the headers, in comparison to old
> > data, same applies for the ZPs (different value).
> >
> > Jean-Charles.
> >
> > > Yes, the fits headers would provide a path back to the original
> > > processing. The only other questions are: (i) does the fits
> > > header contain the version # of Elixir that was used? (I believe
> > > it does.); and (ii) can all versions of Elixir be recreated from
> > > source? If yes and yes, then I think this is sufficient.
> > >
> > > > Hi,
> > > >
> > > > It is lost, but can be recreated easily by feeding the old
> > > > flat-fielding and ZPs database to Elixir.
> > > >
> > > > Something we ought to do is preserve the FITS headers: this
> > > > is our only tracer. JJ, can you take a snapshot of these
> > > > headers and preserve them as the B1/2/3 Elixir set?
> > > >
> > > > Jean-Charles.
> > > >
> > > >> Does this mean that the old preprocessing data is lost forever,
> > > >> or is it backed up somewhere?
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I understand then that CADC is ready today to accept the
> > reprocessed
> > > >> > files and they'll replace automatically the old ones, and
> > Terapix will
> > > >> > be able to see/get them right away?
> > > >> >
> > > >> > The PI 07B DADS CFHT distribution is complete, and the machines are
> > > >> > available
> > > >> > now for the LS reprocessing - I have not heard about data transfer
> > > >> speed
> > > >> > improvement from Kanoa, that will be the key factor for getting all
> > > >> the
> > > >> > data
> > > >> > in time for a spring T0005 release.
> > > >> >
> > > >> > Jean-Charles.
> > > >> >
> > > >> >> Hi,
> > > >> >>
> > > >> >> I support the least effort option, after considering the
> > scientific
> > > >> >> issues. Let's just replace.
> > > >> >>
> > > >> >> David
> > > >> >>
> > > >> >>
> > > >> >> Séverin Gaudet wrote:
> > > >> >> > I support the least effort option, i.e., replacement. I do
> > > >> recognise
> > > >> >> > that my answer is based on technical reasons and not on
> > scientific
> > > >> >> > reasons.
> > > >> >> >
> > > >> >> > Séverin
> > > >> >> >
> > > >> >> > On Mar 4, 2008, at 5:40 PM, Jean-Charles Cuillandre wrote:
> > > >> >> >
> > > >> >> >>
> > > >> >> >> Hi JJ,
> > > >> >> >>
> > > >> >> >> I am afraid the long path is going to take some time to
> > realize,
> > > >> >> >> weeks at least, while the pressure to move on and start the
> > > >> >> >> reprocessing is rising quickly (specially with possible
> > bandwidth
> > > >> >> >> limitations).
> > > >> >> >>
> > > >> >> >> The initial word I have of Terapix on the first test set is
> > that
> > > >> >> >> it is not worse (nor better) than T0004/3/2 data (that's before
> > > >> >> >> applying a secondary color correction, something I attempted to
> > > >> >> >> explain to you last week). So, there you have it: at worst it
> > > >> >> >> will be the same, actually better since:
> > > >> >> >>
> > > >> >> >> - All data since first light follow the same photom flat
> > > >> recipe
> > > >> >> (B4)
> > > >> >> >> - The major ZPs issues has been smoothed out
> > > >> >> >> - (and I can't imagine all this improvement work on the B4
> > > >> flat
> > > >> >> >> by a bunch of us does not show up in the general
> > data set!)
> > > >> >> >>
> > > >> >> >> I want to say, let's just replace the files and I will document
> > > >> this
> > > >> >> on
> > > >> >> >> the MegaPrime pages asap - the user who pick them up ought
> > to find
> > > >> >> their
> > > >> >> >> way. We will make it clear that CFHT+CADC will do their best to
> > > >> >> support
> > > >> >> >> the old B1/B2/B3 versions if they turn out to be needed for
> > some
> > > >> >> reason.
> > > >> >> >>
> > > >> >> >> The advantage of replacing the files straight is that it
> > saves us
> > > >> >> from
> > > >> >> >> worrying about the replacement later on, no file confusion is
> > > >> >> possible.
> > > >> >> >>
> > > >> >> >> Shall we vote on this?
> > > >> >> >> Jean-Charles.
> > > >> >> >>
> > > >> >> >>> Hi All,
> > > >> >> >>>
> > > >> >> >>> I think the following is the flow of what will/must happen for
> > > >> the
> > > >> >> >>> switch for the current CFHT Elixir data to the CFHT Elixir-B4
> > > >> data.
> > > >> >> >>> Looking at what this will actually involve is starting to
> > make me
> > > >> >> >>> wonder if its worth the effort.
> > > >> >> >>>
> > > >> >> >>> The options available are listed below. Please comment.
> > > >> >> >>>
> > > >> >> >>> JJ
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> a) Overwrite the old files with the new ones as the
> > arrive... not
> > > >> >> get
> > > >> >> >>> worked up about this. 0 effort required.
> > > >> >> >>>
> > > >> >> >>> or
> > > >> >> >>>
> > > >> >> >>> b) the following list of tasks must be completed.
> > > >> >> >>>
> > > >> >> >>> - new archive will be created at CADC called CFHTB4 (JO)
> > > >> >> >>>
> > > >> >> >>> - new top level e-transfer directory will be created on a
> > CFHT
> > > >> >> >>> machine,
> > > >> >> >>> this will be the CFHTB4 archive e-transfer area. (CFHT who?
> > > >> HW?)
> > > >> >> >>>
> > > >> >> >>> - new detrended table at CADC will be created to accept the
> > > >> headers
> > > >> >> >>> of the new CFHTB4 archive. This is a duplicate of the current
> > > >> table
> > > >> >> >>> cfht..detrended and will be called cfht..detrended_b4 (IG)
> > > >> >> >>>
> > > >> >> >>> - The header ingestion script in the CFHT archive
> > > >> (CFHTHIngest.py)
> > > >> >> >>> will be modified to insert into the new 'detrended_b4'
> > tables and
> > > >> >> >>> called (CFHTHIngestB4.py) [JJK]
> > > >> >> >>>
> > > >> >> >>> - a cfhtFileIngest script called cfhtFileIngestB4 will be
> > created
> > > >> >> >>> which knows to call the CFHTIngestB4.py script instead [GM]
> > > >> >> >>>
> > > >> >> >>> - new instance of cfht etransfer will be configured at
> > CADC to
> > > >> add
> > > >> >> a
> > > >> >> >>> new archive called CFHTB4. This script will call
> > > >> cfhtFileIngestB4
> > > >> >> >>> instead of cfhtFileIngest (GM?)
> > > >> >> >>>
> > > >> >> >>> - etransfer will be turned on and accept files for CFHTB4
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> -- Once the entire transfer is finished:
> > > >> >> >>>
> > > >> >> >>> - date to switch to B4 will be agreed on.
> > > >> >> >>> - announcement of date on the CFHT www pages and via
> > CFHTLS
> > > >> list
> > > >> >> >>> will
> > > >> >> >>> occur. [JJK]
> > > >> >> >>> - some 'db magic' will be execute so that the CFHTB4 files
> > > >> >> replace
> > > >> >> >>> the CFHT files and the detrended_b4 entries replace the
> > detrended
> > > >> >> >>> entries. [who ???]
> > > >> >> >>>
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> Dr. David Schade
> > > >> >> Scientist and Group Leader
> > > >> >> Canadian Astronomy Data Centre
> > > >> >> Herzberg Institute of Astrophysics
> > > >> >> National Research Council Canada
> > > >> >
> > > >>
> > > >>
> > > >> -----------------------------------------------------------------
> > > >> Chris Pritchet pritchet at uvic.ca
> > > >> Dept of Physics & Astronomy http://www.astro.uvic.ca/~pritchet
> > <http://www.astro.uvic.ca/%7Epritchet>
> > > >> University of Victoria
> > > >> P.O. Box 3055 250-721-7744 (office)
> > > >> Victoria, BC V8W 3P6 250-721-7715 (FAX)
> > > >> CANADA
> > > >> -----------------------------------------------------------------
> > > >>
> > > >
> > >
> > >
> > > -----------------------------------------------------------------
> > > Chris Pritchet pritchet at uvic.ca
> > > Dept of Physics & Astronomy http://www.astro.uvic.ca/~pritchet
> > <http://www.astro.uvic.ca/%7Epritchet>
> > > University of Victoria
> > > P.O. Box 3055 250-721-7744 (office)
> > > Victoria, BC V8W 3P6 250-721-7715 (FAX)
> > > CANADA
> > > -----------------------------------------------------------------
> > >
> >
>
> --
> Dr. John Ouellette
> Operations Manager
> Canadian Astronomy Data Centre
> Herzberg Institute of Astrophysics
> National Research Council Canada
> 5071 West Saanich Road, Victoria BC V9E 2E7 Canada
> Phone: 250-363-3037
Received on Tue Mar 11 2008 - 13:57:18 HST