Re: CFHT etransfer weekly report

From: Kanoa Withington <kanoa_at_cfht.hawaii.edu>
Date: Tue, 19 Nov 2013 16:47:54 -1000

Hi John,

Thanks for the tuning. The change in file size is typical of the change
in instrument so it does not worry me. The transfer rate in both
directions has always been much slower than it should be so that worries
but does not alarm me. It would appear that there are more files are in
the reject directory than are being ingested and that's what I find
alarming.

-Kanoa

John Ouellette wrote, On 11/19/2013 07:49 AM:
> Looking into this a bit, it seems that the transfers between CFHT and
> CADC are the problem: the rate of transfer hasn't really changed, but
> the median file size has gone from 16MB to 274MB between October and
> November. The median transfer rate isn't great (~1.5MB/s), either.
> None of the other etransfer steps (initial file checking, ingestion) has
> really changed much.
>
> I've increased the number of transfer threads, so that should help a
> bit. We can spread the load across more nodes if this is necessary to
> improve things further.
>
> John
>
> On 13-11-18 12:43 PM, Kanoa Withington wrote:
>> We're not making much progress, looks like most files are failing for
>> some reason or another...
>>
>> -Kanoa
>>
>> Ling Shao wrote, On 11/18/2013 10:21 AM:
>>> Novmeber 11, 2013 - November 17, 2013
>>>
>>> 1251 CFHT files (including 16 detrend files, 480 detrended files and 755
>>> raw files)
>>> have been transferred and ingested into our archive this week.
>>> CFHT rejected files:
>>> corrupt: 1 file.
>>> empty: 5 files.
>>> fitsverify: 194 files (including 62 files in this
>>> period).
>>> ingest error: 1556 files.
>>> name: 841 files.
>>> not replace: 1 file.
>>>
>>> No file from TERAPIX has been transferred and ingested into our archive
>>> this week.
>>>
>>> No file from terapixtemp has been processed this week.
>>>
>>> Processing:
>>> number of submitted jobs: 4355.
>>> number of jobs with successful completion: 679.
>>> number of jobs with errors: 3676.
>>> number of files archived: 1948 files.
>>>
>>> CFHT User requests:
>>> 46 CFHT users submitted requests.
>>> Files have been down loaded: 6669 files. Size: 237.1 GB.
>>>
>>> CAOM ingestion status:
>>> status "F": None.
>>> status "P": None.
>>> status "Y": 650.
>>>
>>> CAOM2 ingestion status:
>>> status "F": None.
>>> status "P": None.
>>> status "Y": None.
>>>
>>> Record of RUNIDs that should have pre-processed files available but some
>>> (or all) are missing
>>> num. missing runid instrument First
>>> Exposure Last Exposure ------------ --------------------
>>> -------------------- ------------------- -------------------
>>> 1 2013BC19 ESPADONS Aug 20 2013
>>> 5:53AM Aug 20 2013 5:53AM
>>> 13 2013BH03 ESPADONS Aug 20 2013
>>> 6:21AM Aug 20 2013 11:24AM
>>> 52 2013BP15 ESPADONS Aug 20 2013
>>> 11:46AM Sep 18 2013 1:18PM
>>> 189 2013BP16 ESPADONS Aug 20 2013
>>> 12:34PM Sep 27 2013 12:07PM
>>> 90 2013BC12 ESPADONS Aug 20 2013
>>> 2:28PM Sep 25 2013 3:37PM
>>> 50 2013BC09 ESPADONS Aug 20 2013
>>> 2:51PM Sep 27 2013 3:06PM
>>> 96 2013BC08 ESPADONS Aug 20 2013
>>> 3:23PM Sep 27 2013 10:07AM
>>> 128 2013BF96 ESPADONS Aug 21 2013
>>> 6:29AM Sep 25 2013 9:44AM
>>> 104 2013BC06 ESPADONS Aug 21 2013
>>> 10:38AM Sep 26 2013 5:07AM
>>> 44 2013BC24 ESPADONS Aug 21 2013
>>> 1:06PM Sep 26 2013 10:31AM
>>> 28 2013BC07 ESPADONS Aug 21 2013
>>> 1:49PM Sep 26 2013 9:51AM
>>> 12 2013BC33 ESPADONS Aug 21 2013
>>> 2:09PM Sep 14 2013 3:31PM
>>> 18 2013BT02 ESPADONS Aug 21 2013
>>> 2:41PM Sep 26 2013 1:08PM
>>> 92 2013BH29 ESPADONS Aug 22 2013
>>> 5:22AM Sep 15 2013 3:36PM
>>> 28 2013BC05 ESPADONS Aug 27 2013
>>> 2:27PM Sep 25 2013 2:43PM
>>> 106 2013BC32 MEGAPRIME Sep 1 2013
>>> 8:15AM Oct 8 2013 2:16PM
>>> 34 2013BO05 MEGAPRIME Sep 2 2013
>>> 6:50AM Sep 6 2013 6:44AM
>>> 18 2013AC13 MEGAPRIME Sep 6 2013
>>> 6:55AM Sep 7 2013 7:22AM
>>> 161 2013BP06 MEGAPRIME Sep 7 2013
>>> 9:56AM Oct 9 2013 1:51PM
>>> 378 2013BC99 MEGAPRIME Sep 7 2013
>>> 11:28AM Oct 6 2013 1:52PM
>>> 165 2013BC21 ESPADONS Sep 15 2013
>>> 9:10AM Sep 26 2013 12:30PM
>>> 31 2013BF15 ESPADONS Sep 17 2013
>>> 7:42AM Sep 26 2013 9:09AM
>>> 4 2013BC03 ESPADONS Sep 19 2013
>>> 5:26AM Sep 19 2013 7:16AM
>>> 3 2013BC10 ESPADONS Sep 26 2013
>>> 2:19PM Sep 26 2013 3:27PM
>>> 51 2013BP05 MEGAPRIME Sep 29 2013
>>> 7:48AM Oct 9 2013 12:52PM
>>> 17 2013AF05 MEGAPRIME Oct 4 2013
>>> 4:57AM Oct 6 2013 5:29AM
>>> 4 2013BH98 MEGAPRIME Oct 6 2013
>>> 8:25AM Oct 8 2013 7:04AM
>>> 29 2013BF05 MEGAPRIME Oct 8 2013
>>> 4:55AM Oct 9 2013 2:37PM
>>> 7 2013BO63 WIRCAM Oct 10 2013
>>> 4:53AM Oct 13 2013 9:49AM
>>> 298 2012BP23 WIRCAM Oct 11 2013
>>> 5:16AM Oct 19 2013 7:25AM
>>> 30 2013BH40 WIRCAM Oct 11 2013
>>> 2:19PM Oct 17 2013 9:24AM
>>> 359 2013BH99 WIRCAM Oct 13 2013
>>> 6:18AM Oct 19 2013 3:47PM
>>> 536 2013BH01 WIRCAM Oct 14 2013
>>> 4:31AM Oct 19 2013 3:09PM
>>> 161 2013BT04 WIRCAM Oct 14 2013
>>> 10:29AM Oct 19 2013 2:25PM
>>> 81 2013BF12 WIRCAM Oct 16 2013
>>> 10:53AM Oct 17 2013 2:46PM
>>> 19 2013BH22 WIRCAM Oct 16 2013
>>> 3:45PM Oct 19 2013 2:10PM
>>> 44 2013BT05 WIRCAM Oct 17 2013
>>> 8:30AM Oct 17 2013 12:41PM
>>> 30 2013BC17 WIRCAM Oct 17 2013
>>> 10:41AM Oct 19 2013 10:06AM
>>>
>>> Thanks.
>>>
>>> Ling
>>>
>
Received on Tue Nov 19 2013 - 16:47:59 HST