Re: Missing data

From: kanoa <kanoa_at_cfht.hawaii.edu>
Date: Mon, 07 Apr 2008 15:41:42 -1000

O.K. here are some results on the missing files:

--> Summary (I'm putting this first for people who don't like to read)

We were missing two blocks of LS data in the CADC archive. A block of
raw data and a block of processed data. Each block was on a specific
date. Both have been updated to the CADC and we are now complete except
for a couple of files observed for the LS which cannot be detrended.

--> Missing detrended versions of LS files at the CADC

Sifting through the 1,700 or so files in the list JJ sent, 112 are LS
files (including L99) that do not have a detrended counterpart at the
CADC where there should be. The rest are not supposed to have
counterparts for a number of reasons.

Of these 112, six were sent but did not end up registered at the CADC,
the rest we skipped for some reason. Virtually all of them were from
three days: July 27, 28 and 29 2007. 2 could not be processed because
the raw files have an incompatible raster type. The remaining 110 were
detrended and sent to the CADC.

This is alarming so I ran my own search checking all LS data for all
time. I checked about 24,000 LS object files against the CADC and found
only one other missing counterpart. In summary, we are up-to-date except
for the two files that cannot be detrended:

714514
860178

Should we change the RUNID on these two files so they no longer appear
in queries?

--> Missing raw data at the CADC

I checked the same set of 24,000 files and found 69 raw files were also
missing. They were all from the short period in Jan/Feb '08 where there
was trouble with the e-transfer system. I re-sent those and confirmed
that they are now at the CADC.

--> Missing detrended versions of Q98 and Q97 data at the CADC

We have always had an incomplete set of q98 counterparts at the CADC. I
thought I had got them all but there were still 140 missing, including
the three that Yannick noticed. We are processing and transmitting them
now. There are also some missing q97 files which we will also process
can transmit.


Thanks everyone for noticing that something was amiss!

Aloha,

-Kanoa
Received on Mon Apr 07 2008 - 15:41:43 HST