Minutes from Feb 06 Video Conf.

From: JJ Kavelaars <jjk_at_hia-iha.nrc-cnrc.gc.ca>
Date: Wed, 6 Feb 2008 12:41:21 -0800

Date: February 6, 2008 11:11 AM
Topic: Data flow issues

Attended: YP, HO, JCC, KW, JJK, JO

                - error in T0004 file handling at CADC

        JJK explained the complex collision of issues that resulted in files
from
        Terapix being replaced by old files in our storage system.

        Essentially the problem was caused by the extensive manual
interventions needed
        when the CADC e-transfer system fails and the lack of a procedure and
communication
        trail for clearing those errors.

        The resolution at the CADC end is to better control the flow of data at
        CADC end with John Ouellette taking the operations roll more.

        ACTION: JCC add JO to email list of dog

        To allow better communication with the users on the state of files
JJK has implemented
        an ‘ingest date’ column in the results for CFHSL-T000X queries.
Having some sort
        of ‘checksum’ available would really help users too.

        ACTION: JJK add md5 sum to TERPIX DOWNLOAD pages.

        The fact that new catalogues were created and the trouble with their
release along
        with an explanation of the available MD5 checksum and ingest date
info should
        be communicated to the Users ASAP. First we will contact the SAC and
let them
        know the situation has been corrected. Once the MD5 system is
working then the
        users can be contacted.

        ACTION: JCC [with YP and JJK] will draft a letter to SAC explaining
the situation.
                        after this letter goes to SAC a letter will go to LS community.
                        -

        NEXT two issues were not discussed. Are we really going to loose the
original T0004
        release data or should that be reserved someplace?

                - T0004.1 and verification

                - backlog of CFHT processing

        Backlog is being cleared. A system for giving the status of a file
in the e-transfer
        system is being worked on. This will help CFHT [and Terapix] track
the status of transfers
        and hopefully make problems easier to catch.

                - Elixir2 and the great reprocessing.
                                - what to do with the Elixir1 stuff

        JCC has some of the data off the CADC and will be providing
reprocessed data
        to the TERAPIX for T0005 testing. there are lots of things happening
in parallel.
        
        transfers will happen outside the data transfer area.

                                - flats from cfht and e-transfer

        they are versioned correctly so not a problem to put into CADC archive.

        ACTION: JCC to give KW the ‘new’ flats through e-transfer.
        Once the data is re-processed we will have the old processing system
held in archive.

        
                - T0005 update
        

        Getting the entire T0005 inputs set to TPix will take ~22 days (25000
files)
        really they would like to do the processing as fast a possible.
        Network is as tuned as can be.

        If the transfers are delayed at all then the T0005 release date
        [currently April 15 2008] will slip.

        The transfer from CFHT to CADC to IAP needs to be a flow: that is as
file is processed at CFHT and sent to CADC then IAP needs to start
getting it. Those files will then be staged at IAP until a large
enough ‘chunk’ arrives so the T0005 pipeline can start.

        ACTION: JCC will create a page the updates the status and flow of
reprocessing. This will help Fred keep track of what data is available
for transfer and keep up with the flow from CFHT.

                - QFits _at_ CFHT

        ACTION: holding until more time available for work needed for this.

        
Received on Wed Feb 06 2008 - 10:41:51 HST