e-transfer from terapix

From: JJ Kavelaars <jjk_at_hia-iha.nrc-cnrc.gc.ca>
Date: Sat, 17 Mar 2007 08:34:50 -0700

Hi Fred,

To get e-transfer setup can you please provide a directory that will
be OWNED by cadc and will have the new/replace/... directories
included. Your job will be to put the sym links from the actual data
to the new or replace directory, as needed. Sometimes, to improve
transfer speeds, we create 'tar' files of multiple bits stuck
together. When we do that we need staging space, something on the
order of ~200Gbytes is usually pretty safe for this. We also request
that you 'batch' the placement of the sym-links so that we don't have
a process that is trying to get more then about 100Gbytes of data in
one step, just to be sure we aren't making mistakes. Also, the
staging space should not be on the root filesystem, to avoid disk-
full errors if we accidentally fill the staging space.

About the new/replace logic. We tried having just a single drop
area (regardless of new or replace) and we found that data-source
providers did not really pay much attention to what they were doing
and would place 1000s of files into the transfer system repeatedly
[mostly because they didn't know if they'd already sent the files].
Repeatedly transferring a few files now and then is OK but sometimes
we would be transferring many tera-bytes of data that we already
have. Although our network capacity is very good we don't really like
'wasting' that capacity on getting files we already have a copy of.

A work around for you is to place all sym-links into the new area.
If we already have a copy with that name we will put the sym link
into an area like rejected/not-new. Then you can move it to replace.
We will check the replace area file and if we have one with the same
CRC then we'll put the symlink into the directory rejected/not-replace.

This will generate lots of messages in our system with various bits
of logging associated with that, but no substantial problems will occur.

Alternatively, I can provide you with our 'CRC' of the file and you
can compare our 'CRC' to value you get by running our crc-generator
on the file. The binary for our crc generator is on your machine at

clix.iap.fr:/home/nis/cadc/bin/cadcCRC

I can add the CRC values we produce to the megaprime_proxy table.

?
JJ


Begin forwarded message:

> From: Geoffrey Melnychuk <geoffrey.melnychuk_at_nrc-cnrc.gc.ca>
> Date: March 16, 2007 11:21:19 AM PDT
> To: JJ Kavelaars <jjk_at_hia-iha.nrc-cnrc.gc.ca>
> Subject: Re: e-transfer
>
>
> Also, the etransfer machine at terapix (clix.iap.fr) does not have
> any directories for etransfer anymore. (the etransfer area was
> formerly /data/clix/fc3/cadc/) We need a cadc-owned (owned. not
> just writable) directory in which to build the etransfer state
> directories.
>
> --g.
>
Received on Sat Mar 17 2007 - 05:35:22 HST