Re: CFHT, TERAPIX e-transfer report

From: Kanoa Withington <kanoa_at_cfht.hawaii.edu>
Date: Mon, 02 Mar 2009 15:29:05 -1000

Hi John,

Ah, I see. I do think we need to turn it back on. We don't have any
programmatic use of the no_check area other than for the metadata files
JJ already mentioned (which is to say: everything in there is put there
by a person).

 From time to time we end up with observations that are worth archiving
but not worth fixing if fitsverify decides to throw an error. Some
plausible circumstances include, aborted sequences, exposures taken
outside of the observing software, experimental observing techniques or
experimental instrument configuration, etc. In a perfect world we would
have enough staff to fix them all but as it is we get crunched just
fixing the ones that really need it. By "worth archiving" I mean that
they are not complete junk and in virtually every case whatever
fitsverify is complaining about has no impact on the science value of
the file. For each file someone verifies that before putting it in
"no_check".

My take on it is that if someone in the act of collecting a data set
decides to reject any CFHT file that does not pass fitsverify then they
can probably do so without missing anything important. I wouldn't see
the point in doing that but it's their call. Also, I can imagine a
future with a more useful and pragmatic tool than the fitsverify of
today which would render 99.9% of this irrelevant.

Aloha,

-K

John Ouellette wrote, On 03/02/2009 01:01 PM:
> Hi Kanoa -- the no_check pick-up was turned off deliberately several
> months ago in October due to a bit of a train-wreck of files at the
> time. The problem with no_check is that it does what it says: no file
> verification tests are run on the files and they are blindly put into
> the archive, potentially replacing good archive files with bad.
> Admittedly, we should have informed you of this in October and discussed
> the alternatives: mea culpa.
>
> Is there any reason we can't run these files through new/ and/or
> replace/? Before doing so, though, we should clean out the reject/
> states so that we can see where these particular files fail (or not).
> There are a large number of files in 'not_replace', some of which are
> due to a batch of retransferring that we did to clean up a header issue.
>
> J.
>
> Kanoa Withington wrote:
>>
>> Hi Ling,
>>
>> For some reason files in our "no_check" directory have not been picked
>> up in a while.
>>
>> Thanks,
>>
>> -Kanoa
>>
>> Ling Shao wrote, On 03/02/2009 11:48 AM:
>> >
>> > February 23, 2009 - March 1, 2009
>> >
>> > 14375 CFHT files (including 6716 raw files and 7659 detrended files)
>> > have been transferred and ingested into our archive.
>> > CFHT rejected files:
>> > corrupt: 21 files.
>> > empty: 1 file.
>> > fitsverify: 182 files (including 86 files in this
>> period).
>> > header: 2161 files (including 2160 files in this
>> period).
>> > name: 32 files.
>> > not_new: 836 files.
>> > not_repalce: 7366 files.
>> >
>> > No file from TERAPIX site has been transferred.
>> >
>> > CFHT User requests:
>> > 28 CFHT users submitted requests.
>> > Files have been down loaded: 5592 files. Size: 932.8 GB.
>> >
>> > Thanks.
>> >
>> > Ling
>>
Received on Mon Mar 02 2009 - 15:29:10 HST