Looking into this a bit, it seems that the transfers between CFHT and
CADC are the problem: the rate of transfer hasn't really changed, but
the median file size has gone from 16MB to 274MB between October and
November. The median transfer rate isn't great (~1.5MB/s), either.
None of the other etransfer steps (initial file checking, ingestion) has
really changed much.
I've increased the number of transfer threads, so that should help a
bit. We can spread the load across more nodes if this is necessary to
improve things further.
John
On 13-11-18 12:43 PM, Kanoa Withington wrote:
> We're not making much progress, looks like most files are failing for
> some reason or another...
>
> -Kanoa
>
> Ling Shao wrote, On 11/18/2013 10:21 AM:
>> Novmeber 11, 2013 - November 17, 2013
>>
>> 1251 CFHT files (including 16 detrend files, 480 detrended files and 755
>> raw files)
>> have been transferred and ingested into our archive this week.
>> CFHT rejected files:
>> corrupt: 1 file.
>> empty: 5 files.
>> fitsverify: 194 files (including 62 files in this period).
>> ingest error: 1556 files.
>> name: 841 files.
>> not replace: 1 file.
>>
>> No file from TERAPIX has been transferred and ingested into our archive
>> this week.
>>
>> No file from terapixtemp has been processed this week.
>>
>> Processing:
>> number of submitted jobs: 4355.
>> number of jobs with successful completion: 679.
>> number of jobs with errors: 3676.
>> number of files archived: 1948 files.
>>
>> CFHT User requests:
>> 46 CFHT users submitted requests.
>> Files have been down loaded: 6669 files. Size: 237.1 GB.
>>
>> CAOM ingestion status:
>> status "F": None.
>> status "P": None.
>> status "Y": 650.
>>
>> CAOM2 ingestion status:
>> status "F": None.
>> status "P": None.
>> status "Y": None.
>>
>> Record of RUNIDs that should have pre-processed files available but some
>> (or all) are missing
>> num. missing runid instrument First
>> Exposure Last Exposure ------------ --------------------
>> -------------------- ------------------- -------------------
>> 1 2013BC19 ESPADONS Aug 20 2013
>> 5:53AM Aug 20 2013 5:53AM
>> 13 2013BH03 ESPADONS Aug 20 2013
>> 6:21AM Aug 20 2013 11:24AM
>> 52 2013BP15 ESPADONS Aug 20 2013
>> 11:46AM Sep 18 2013 1:18PM
>> 189 2013BP16 ESPADONS Aug 20 2013
>> 12:34PM Sep 27 2013 12:07PM
>> 90 2013BC12 ESPADONS Aug 20 2013
>> 2:28PM Sep 25 2013 3:37PM
>> 50 2013BC09 ESPADONS Aug 20 2013
>> 2:51PM Sep 27 2013 3:06PM
>> 96 2013BC08 ESPADONS Aug 20 2013
>> 3:23PM Sep 27 2013 10:07AM
>> 128 2013BF96 ESPADONS Aug 21 2013
>> 6:29AM Sep 25 2013 9:44AM
>> 104 2013BC06 ESPADONS Aug 21 2013
>> 10:38AM Sep 26 2013 5:07AM
>> 44 2013BC24 ESPADONS Aug 21 2013
>> 1:06PM Sep 26 2013 10:31AM
>> 28 2013BC07 ESPADONS Aug 21 2013
>> 1:49PM Sep 26 2013 9:51AM
>> 12 2013BC33 ESPADONS Aug 21 2013
>> 2:09PM Sep 14 2013 3:31PM
>> 18 2013BT02 ESPADONS Aug 21 2013
>> 2:41PM Sep 26 2013 1:08PM
>> 92 2013BH29 ESPADONS Aug 22 2013
>> 5:22AM Sep 15 2013 3:36PM
>> 28 2013BC05 ESPADONS Aug 27 2013
>> 2:27PM Sep 25 2013 2:43PM
>> 106 2013BC32 MEGAPRIME Sep 1 2013
>> 8:15AM Oct 8 2013 2:16PM
>> 34 2013BO05 MEGAPRIME Sep 2 2013
>> 6:50AM Sep 6 2013 6:44AM
>> 18 2013AC13 MEGAPRIME Sep 6 2013
>> 6:55AM Sep 7 2013 7:22AM
>> 161 2013BP06 MEGAPRIME Sep 7 2013
>> 9:56AM Oct 9 2013 1:51PM
>> 378 2013BC99 MEGAPRIME Sep 7 2013
>> 11:28AM Oct 6 2013 1:52PM
>> 165 2013BC21 ESPADONS Sep 15 2013
>> 9:10AM Sep 26 2013 12:30PM
>> 31 2013BF15 ESPADONS Sep 17 2013
>> 7:42AM Sep 26 2013 9:09AM
>> 4 2013BC03 ESPADONS Sep 19 2013
>> 5:26AM Sep 19 2013 7:16AM
>> 3 2013BC10 ESPADONS Sep 26 2013
>> 2:19PM Sep 26 2013 3:27PM
>> 51 2013BP05 MEGAPRIME Sep 29 2013
>> 7:48AM Oct 9 2013 12:52PM
>> 17 2013AF05 MEGAPRIME Oct 4 2013
>> 4:57AM Oct 6 2013 5:29AM
>> 4 2013BH98 MEGAPRIME Oct 6 2013
>> 8:25AM Oct 8 2013 7:04AM
>> 29 2013BF05 MEGAPRIME Oct 8 2013
>> 4:55AM Oct 9 2013 2:37PM
>> 7 2013BO63 WIRCAM Oct 10 2013
>> 4:53AM Oct 13 2013 9:49AM
>> 298 2012BP23 WIRCAM Oct 11 2013
>> 5:16AM Oct 19 2013 7:25AM
>> 30 2013BH40 WIRCAM Oct 11 2013
>> 2:19PM Oct 17 2013 9:24AM
>> 359 2013BH99 WIRCAM Oct 13 2013
>> 6:18AM Oct 19 2013 3:47PM
>> 536 2013BH01 WIRCAM Oct 14 2013
>> 4:31AM Oct 19 2013 3:09PM
>> 161 2013BT04 WIRCAM Oct 14 2013
>> 10:29AM Oct 19 2013 2:25PM
>> 81 2013BF12 WIRCAM Oct 16 2013
>> 10:53AM Oct 17 2013 2:46PM
>> 19 2013BH22 WIRCAM Oct 16 2013
>> 3:45PM Oct 19 2013 2:10PM
>> 44 2013BT05 WIRCAM Oct 17 2013
>> 8:30AM Oct 17 2013 12:41PM
>> 30 2013BC17 WIRCAM Oct 17 2013
>> 10:41AM Oct 19 2013 10:06AM
>>
>> Thanks.
>>
>> Ling
>>
--
Dr. John Ouellette
Operations Manager | Gestionnaire des opérations
Canadian Astronomy Data Centre | Centre canadien de données astronomiques
National Research Council Canada | Conseil national de recherches Canada
5071 West Saanich Road | 5071, chemin West Saanich
Victoria, BC V9E 2E7 | Victoria, (C.-B.) V9E 2E7
Telephone | Téléphone 250-363-3037
Government of Canada | Gouvernement du Canada
Received on Tue Nov 19 2013 - 07:50:03 HST