Fwd: Minutes of March 28 Video Conf.

From: JJ Kavelaars <jjk_at_hia-iha.nrc-cnrc.gc.ca>
Date: Wed, 11 Apr 2007 09:33:40 -0700

Hi,

the minutes for the previous DOG meeting did not get to the DOG email
list. Here they are.

JJ


Begin forwarded message:

> From: JJ Kavelaars <jjk_at_hia.nrc.ca>
> Date: March 29, 2007 5:27:42 PM PDT
> To: Séverin Gaudet <Severin.Gaudet_at_nrc-cnrc.gc.ca>, Jean-Charles
> Cuillandre <jcc_at_cfht.hawaii.edu>, Frederic Magnard
> <magnard_at_iap.fr>, kanoa Withington <kanoa_at_cfht.hawaii.edu>
> Subject: Minutes of March 28 Video Conf.
> Reply-To: JJ Kavelaars <jjk_at_hia.nrc.ca>
>
> Hi, here is the first pass at the minutes. I have ACTIONs,
> OUTCOMES, DISCUSSION and NEW ACTION and NEW ITEMS as headers to be
> watched for. I think we're doing pretty good so far.... Please
> review and suggest revisions and additions.
>
>
> JJ
>
>
> DATE: March 28 2007.
> Present: JCC, KW, SG, JJK, FM, HO, CP
>
>
> Review of Action Items from the previous Meeting:
>
> ACTION - In order to keep the one-to-one interactions between Fred,
> JJ and
> Jean-Charles, JC will re-activate the Data Operation Group (DOG)
> mailing list
>
> OUTCOME: DOG email list was re-activated by Kanoa and is now being
> used for all discussion.
>
>
> ACTION - - JC will set up the system at CFHT to archive action
> items and decisions. It should be simple: minutes will be included
> with a set of actions open/closed. The CADC/CFHT/Terapix
> participants to videoconferences (in particular the the DOG), the
> OG. The SG will have access to the archive of the minutes.
>
> OUTCOME: JC has establish a www site and a link to DOG emails can
> be found there. Only DOG/OG members can access.
>
>
> ACTION: JJK to investigate why some files are missing from
> megaprime_proxy listing, even though they are known to exist.
>
> OUTCOME: Logic of the proxy was to only expose ‘complete’
> datasets. Sometimes CADC has incomplete QualityFTIS data and so
> the dataset is marked as unavailable. This rule has been
> changed. CADC will need to investigate who to correct the problem
> of incomplete datasets.
>
> ACTION: JJK to change to URL provided by megaprime_proxy so that
> ‘fits-compressed’ images are delivered instead of ‘gzip compressed’.
>
> OUTCOME: URL delivered by megaprime_proxy has been changed.
>
> ACTION: Kanoa will investigate why it not always FITS compliant and
> will make the FITS table stable.
>
> OUTCOME: The exact reason for the ‘occasional’ failures of
> skyprobe appears to be related to how ‘NULL’ values are handled.
> From now on MCMD tables [which contain the skyprobe values] will be
> sent via the ‘not-fits’ channel of e-transfer. Additionally, some
> of the missing skyprobe values are from the period of the
> Earthquake last year and those are missing because the don’t exist.
>
> NEW ACTION: Terapix to determine if any sky-probe values are
> missing from periods of time when actual data was acquired.
>
>
> ACTION: Fred will tell CADC the names of the machines, directory
> locations and PORTs where e-transfer will operate for terapix. CADC
> will start etransfer of weight images.
>
> OUTCOME: e-transfer are was established.
> cadcCRC checking of files stored at Terapix was performed and new
> and changed files placed into e-transfer by
> terapix.
> All good files were transfered but a large number of have some
> ‘FITSVERIFY’ errors.
>
> NEW ACTION: Fred to investigate the source of the FITSVERIFY
> errors. Could be coming from the original images. Will coordinate
> discussion with Kanoa on this issue.
>
>
> Actions: Terapix will provide a dump of the Terapix DB that
> includes the Grade and the Comment for each image in a CSV format
> when the release is frozen.
>
> OUTCOME: This was done. CADC has examined the file and is not
> sure if this is all the information that is needed.
>
>
> NEW ACTION: JJK to examine the requirements of the CADC retrieval/
> search system to ensure that no additional data is needed by CADC.
>
> ACTION: CFHT, CADC and Terapix will agree on a checksum format for
> all CFHTLS and PI data that will be created by CFHT and Terapix,
> and included in the FITS header.
>
> OUTCOME: A solution for CHECK-SUMS appears to exist in the FITS
> standard. This allows file-integrity checking but not ‘finger-
> printing’ which is needed for e-transfer. CADC is providing
> access to the cadcCRC values so they can be used as a ‘finger-
> printing’ for the files transfered to the CADC. This will be
> switch to MD5 when possible for the CADC.
>
> DISCUSSION:
>
> Existing tools, fpack and funpack can be used to add FITS CHECKSUM
> keywords and fitsverify will do the checking. this would be a
> great addition to data security for CFHT and CADC.
>
> NEW ACTION: Severin will investigate how long a moving CADC to MD5
> will/could take. Kanoa will investigate adding FITS checksums to
> headers.
>
> NEW ACTION: JJK will add the CRC values to the information
> returned by the megaprime_proxy table.
>
> NEW ACTION: JJK will send a instruction set for the use of
> megaprime_proxy and cfhtInfo CGIs.
>
> DISCUSSION:
> CHECKSUMS only work correctly on uncompressed-floats or integer
> data. The weight-maps are currently stored as floats and this
> makes using compressed file checksums [via the FITS standard]
> impossible [due to lossy compression].
>
> NEW ACTION: Fred will investigate switching the WEIGHTS to integer
> storage. This will allow FITS CHECKSUM to be used and also reduce
> the size of compressed weight-maps, lightening the load on e-
> transfer and data storage.
>
>
> NEW ITEM:
> Quality Fits at CFHT.
>
> CFHT continues to be interested in running QFITS as part of the
> QSO processes for all megaprime data. JCC would like to
> investigate the use of the QFITS system, with an aim to implement
> something for the summer of 2007. CADC will need to investigate
> how best to integrate the information/files produced by QFITS into
> the data search/delivery system.
>
> NEW ACTION: Fred will provide [by the end April] a new release of
> QFITS for installation at CADC and CFHT so that JJK and JCC can
> investigate details of using QFITS.
>
Received on Wed Apr 11 2007 - 06:34:05 HST