CFHT, Instruments, Spectroscopy, ESPaDOnS

Upena, CFHT's reduction pipeline for
ESPaDOnS: an Echelle SpectroPolarimetric Device
for the Observation of Stars at CFHT

Table of Content

Libre-ESpRIT [Back to Top]

In Classical and QSO mode, data reduction is performed using Donati's data reduction software Libre-ESpRIT.

Upena [Back to Top]

In order to manage many nights' worth of data, with mutiple programs carried out during each single night, CFHT has built a pipeline called Upena.

Upena uses all of the routines used by Libre-ESpRIT, but:

In March 2008, Upena v0.1 was used. For information specific to that run, please see this page.

The current version of Upena, version 1.0, provides reduced data in FITS format and Libre-ESpRIT format. Each FITS processed data is associated with 2 *.s files, with the same odometer number. One file has a 'n' in its name, for the reduction with the continuum normalized to 1; the other file has a 'u' in its name for the reduction done without any continuum normalization.

For example, the file 987123i.fits corresponds to 987123in.s and 987123iu.s, and the file 912345p.fits corresponds to 912345pn.s and 912345pu.s.

Upena has a quick-analysis component and a final reduction component.

The quick-analysis component is run during the night as exposures are taken. The calibrations taken before sunset (biases, flat fields, comparison and Fabry-Perot exposures) are used to find the geometric and wavelength calibrations. Individual scientific exposures are reduced as they are taken. As each exposure is taken (calibration and scientific exposures), checks are performed to make sure the CCD readout noise is nominal, light falls on the CCD, exposures are not saturated, S/N is as indicated by the PIs, all devices are positioned correctly in the spectrograph and polarimeter, etc.

FITS format processed files [Back to Top]

All object files (*o.fits) are individually processed to extract the spectrum (polarimetric and "Star Only" modes) or spectra ("Star + Sky" mode), whether they were taken in spectroscopic or polarimetric mode. These spectra are saved in *i.fits files. The reduction is done once with the continuum normalized to 1, and again with the continuum not normalized. In the FITS processed files, the reduction is also done the standard way (without the 'w' option), and with the 'w' option (which cancels automatic wavelength correction from telluric lines, and is used mostly by spectroscopists).

Therefore, all *i.fits files contain 4 spectra (or 12 in the case of "Star + Sky" mode).

FITS keywords identify the type of reduction and content of the file:

REDUCTIO= 'Intensity'          / Type of reduction
FILENAME                                " Base filename at reduction"
DATE                                    " UTC Date of reduction"
UTIME                                   " UTC time of reduction"
HSTTIME                         " Local time in Hawaii of reduction"
FILENAMO                                " Base filename at acquisition"
DATE                                    " UTC Date of reduction"
UTIME                                   " UTC time of reduction"
HSTTIME                         " Local time in Hawaii of reduction"

For the "Star Only" and polarimetric data, the i.fits files have:

COL1    = 'Wavelength'         / Normalized
COL2    = 'Intensity'          / Normalized
COL3    = 'ErrorBar'           / Normalized
COL4    = 'Wavelength'         / UnNormalized
COL5    = 'Intensity'          / UnNormalized
COL6    = 'ErrorBar'           / UnNormalized
COL7    = 'Wavelength'         / Normalized, no autowave correction
COL8    = 'Intensity'          / Normalized, no autowave correction
COL9    = 'ErrorBar'           / Normalized, no autowave correction
COL10   = 'Wavelength'         / UnNormalized, no autowave correction
COL11   = 'Intensity'          / UnNormalized, no autowave correction
COL12   = 'ErrorBar'           / UnNormalized, no autowave correction

For the "Star + Sky" data, the i.fits files have:

COL1    = 'Wavelength'         / Normalized
COL2    = 'Star    '           / Normalized
COL3    = 'Star+sky'           / Normalized
COL4    = 'Sky     '           / Normalized
COL5    = 'ErrorBar1'          / Normalized
COL6    = 'ErrorBar2'          / Normalized
COL7    = 'ErrorBar3'          / Normalized
COL8    = 'Wavelength'         / UnNormalized
COL9    = 'Star    '           / UnNormalized
COL10   = 'Star+sky'           / UnNormalized
COL11   = 'Sky     '           / UnNormalized
COL12   = 'ErrorBar1'          / UnNormalized
COL13   = 'ErrorBar2'          / UnNormalized
COL14   = 'ErrorBar3'          / UnNormalized
COL15   = 'Wavelength'         / Normalized, no autowave correction
COL16   = 'Star    '           / Normalized, no autowave correction
COL17   = 'Star+sky'           / Normalized, no autowave correction
COL18   = 'Sky     '           / Normalized, no autowave correction
COL19   = 'ErrorBar1'          / Normalized, no autowave correction
COL20   = 'ErrorBar2'          / Normalized, no autowave correction
COL21   = 'ErrorBar3'          / Normalized, no autowave correction
COL22   = 'Wavelength'         / UnNormalized, no autowave correction
COL23   = 'Star    '           / UnNormalized, no autowave correction
COL23   = 'Star    '           / UnNormalized, no autowave correction
COL24   = 'Star+sky'           / UnNormalized, no autowave correction
COL25   = 'Sky     '           / UnNormalized, no autowave correction
COL26   = 'ErrorBar1'          / UnNormalized, no autowave correction
COL27   = 'ErrorBar2'          / UnNormalized, no autowave correction
COL28   = 'ErrorBar3'          / UnNormalized, no autowave correction

For the polarimetric case, sets of 4 exposures produce one *p.fits file, which again presents data reduced with the continuum normalized to 1 and not normalized, and also with and without the 'w' option. Whether the continuum is normalized or not (in intensity), the Stokes parameters are always un-normalized, in polarization (i.e., the column gives for example V/I * I where I is the intensity).

Un-normalized Stokes parameters go between -Intensity to +Intensity.

If the continuum is normalized (in intensity!) to 1 (to make the spectrum flat), the non-normalized Stokes parameters will go from -1 to +1 where Intensity<1 (i.e. in the continuum and absorption lines). In an an emission line, the non-normalized Stokes parameter can go above 1.

For example (taking real data):

A FITS keyword identifies the reduction:

REDUCTIO= 'Polar   '           / Type of reduction

In general, reduction is done with 4 exposures, as indicated by those keywords:

POLARSEQ=                    4 / Number of images in polar sequence
STARTSEQ=                    1 / Start image of polar sequence

If only 2 exposures were used, the above keywords will have different values.

The p.fits files have:

COL1    = 'Wavelength'         / Normalized
COL2    = 'Intensity'          / Normalized
COL3    = 'Stokes  '           / Normalized
COL4    = 'CheckN1 '           / Normalized
COL5    = 'CheckN2 '           / Normalized
COL6    = 'ErrorBar'           / Normalized
COL7    = 'Wavelength'         / UnNormalized
COL8    = 'Intensity'          / UnNormalized
COL9    = 'Stokes  '           / UnNormalized
COL10   = 'CheckN1 '           / UnNormalized
COL11   = 'CheckN2 '           / UnNormalized
COL12   = 'ErrorBar'           / UnNormalized
COL13   = 'Wavelength'         / Normalized, no autowave correction
COL14   = 'Intensity'          / Normalized, no autowave correction
COL15   = 'Stokes  '           / Normalized, no autowave correction
COL16   = 'CheckN1 '           / Normalized, no autowave correction
COL17   = 'CheckN2 '           / Normalized, no autowave correction
COL18   = 'ErrorBar'           / Normalized, no autowave correction
COL19   = 'Wavelength'         / UnNormalized, no autowave correction
COL20   = 'Intensity'          / UnNormalized, no autowave correction
COL21   = 'Stokes  '           / UnNormalized, no autowave correction
COL22   = 'CheckN1 '           / UnNormalized, no autowave correction
COL23   = 'CheckN2 '           / UnNormalized, no autowave correction
COL24   = 'ErrorBar'           / UnNormalized, no autowave correction

Please note the following comments:

COMMENT For Stokes Q, V, and W, keep the Stokes parameter sign as is
COMMENT For Stokes U, invert the sign of the Stokes parameter

FITS keywords from the 4 raw exposures are sometimes repeated 4 times (with new names) to indicate values that change (or should not change) from one exposure to the next. For example:

FILENAME= '977289p '           / Base filename at reduction
FILENAM1= '977289o '           / Base filename at acquisition
FILENAM2= '977290o '           / Base filename at acquisition
FILENAM3= '977291o '           / Base filename at acquisition
FILENAM4= '977292o '           / Base filename at acquisition
EXPTIME1= '356.000 '           / Integration time (seconds)
EXPTIME2= '356.000 '           / Integration time (seconds)
EXPTIME3= '356.000 '           / Integration time (seconds)
EXPTIME4= '356.000 '           / Integration time (seconds)
MJD-OBS1= '54544.362'          / Modified Julian Date at start of obs.
MJD-OBS2= '54544.367'          / Modified Julian Date at start of obs.
MJD-OBS3= '54544.371'          / Modified Julian Date at start of obs.
MJD-OBS4= '54544.376'          / Modified Julian Date at start of obs.
MOONALT1= '81.0000 '           / Moon altitude at start in degrees
MOONALT2= '80.9000 '           / Moon altitude at start in degrees
MOONALT3= '80.6000 '           / Moon altitude at start in degrees
MOONALT4= '80.0000 '           / Moon altitude at start in degrees

Libre-ESpRIT formatted processed data

The ASCII output from Libre-ESpRIT is also provided to PIs in *.s files, using the odometer number of the raw file (or the first raw file of a polarimetric sequence) as a base, and then adding one or more letters to identify the content:
iintensity spectra
ppolarimetric spectra
ncontinuum normalized to 1
ucontinuum not normalized

Each *.s file also has a *.out file which gives details of the reduction. In particular, the end of this file presents SNR values for each order, radial velocity correction from telluric lines (if applicable), and estimates of the V magnitude and effective temperature (in the normalized case only).

The fits2txt tool from the Opera project can be used to convert processed FITS files to ASCII files similar to the original output of Libre-ESpRIT. Click here to download fits2txt.

Ancillary data

In addition to the raw and processed data, ancillary information is also provided to PIs: details of the geometric and wavelength calibration (geom* and wcal* files), the average flat foeld (ff* file) and the comparison spectra (th* files).

Data reduction for PIs [Back to Top]

The final reduction component re-reduces all the data with additional flats taken in the morning (if available), and performs the polarimetric calculations if appropriate. All data are reduced, whether the exposures were validated or not (except for the first 5 nights of the first run where the 4th exposure of a polarimetric sequence is useless); if Libre-ESpRIT can reduce an exposure, regardless of Image Quality or extinction by clouds, the exposures are reduced. All data are reduced with the continuum normalized to 1, and also not normalized; PIs do not need to make a special request regarding continuum normalization. In QSO mode and for all PIs, the reduction is also done with and without the 'w' option, which uses or not the telluric lines to correct the wavelengths. The reduction is then done 4 different ways. For Stokes U measurements, the sign of the Stokes parameter (column 3) needs to be flipped (inverted). A comment in the FITS header indicates this.

The naming scheme for the reduced files is the following: regardless of the Observing Mode used, each single FITS raw file (o.fits) produces one reduced FITS file named i.fits using the same odometer number as the original raw file. Both spectra, with the continuum normalized and not normalized, are found within that FITS file. If the Observing Mode is the polarimetric one, an additional FITS file is produced, named p.fits, with the odometer number used the number of the first polarimetric file of the sequence of 4.

The w option [Back to Top]

For all PIs, the reduction is done with the 'w' option (i.e., without the automatic wavelength correction from telluric lines) and without that option. The result of those 2 reductions can be found in the FITS files.

Data distribution [Back to Top]

All data will be available to PIs via the network (not on tapes or drives) at a URL privately sent to each PI along with a key.

Data can be retrieved using wget. The following example shows how to retrieve only *o.fits files, and the command can be run once a day to retrieve only new files.

wget -r -nc -c -Ao.fits http://URL/[key]/

The "-r" sets directory recursion on.

"-nc" literally means no clobber, but tells wget to only get new files. Do not "clobber" previously downloaded files.

"-c" means to continue on any partially downloaded files.

"-Ao.fits" The files to download. In this case files that end with "o.fits"

There are also ways to download files listed in a master file. Many options are available. For more information, please see these URLs, or google wget:
http://en.wikipedia.org/wiki/Wget
http://www.gnu.org/software/wget/

The distributed data will consist of all the raw files (including the calibrations), all the reduced spectra in FITS format and Libre-ESpRIT format, all corresponding "log" file produced by Libre-ESpRIT (called *.out by Libre-ESpRIT), and other files used by Libre-ESpPRIT such as the average flat field, the geometric calibration data file, etc.

FilenameRaw/ProcessedFile typeNote
b.fitsRawBias
c.fitsRawComparison
a.fitsRawFabry-Perot
f.fitsRawFlast
x.fitsRawFocusNot needed by PIs
o.fitsRawObject
i.fitsProcessedIntensityContinuum normalised and not normalized, with and without the telluric line correction
in.sProcessedIntensityLibre-ESpRIT format, continuum normalised to 1
iu.sProcessedIntensityLibre-ESpRIT format, continuum not (un) normalised
p.fitsProcessedPolarizationContinuum normalised and not normalized, with and without the telluric line correction
pn.sProcessedPolarizationLibre-ESpRIT format, continuum normalised to 1
pu.sProcessedPolarizationLibre-ESpRIT format, continuum not (un) normalised
i/p.outoutput file from the Libre-ESpRIT reductionContains useful information on S/N ratios, radial velocity corrections, etc.
ffProcessedAverage flat field
th*.sProcessedComparison exposure (1D), not calibrated in wavelength
th*.wsProcessedComparison exposure (1D), calibrated in wavelength
geom*datgeometric calibration data
geom*outgeometric calibrations output file
wcal*datwavelength calibration data
wcal*outwavelength calibrations output file

Night Reports [Back to Top]

Night Reports are available on the web.

The Weather log section gives an idea of the sky conditions for any night. SkyProbe can also be used to see plots of the extinction; note that SkyProbe has an offset of about 0.1-0.2mag: when the sky is photometric, without any clouds, it sill shows extinction between 0.1mag and 0.2mag.

The Exposures log section show all validated exposures, ie, exposures that were graded by the Queue Coordinator and judged good for science. The column "Eval" indicates how good the exposure is. A grade of 1 indicates sky conditions from perfect (no clouds) to about 0.5mag extinction (with S/N ratios within 20% of the nominal value), and Image Quality (seeing) values within what PIs have requested. A grade of 2 indicates usually more clouds (up to about 1mag), and exposures are repeated once (as much as possible given te sky conditions) to recover some of the lost flux. A grade of 3 indicates even worse sky conditions; exposures are repeated twice.

After the March run, we plan on adding to the distribution other useful information such as a Summary table of the observations, the SkyProbe plots, weather information, etc.

(And Upena means fishnet...)


http://www.cfht.hawaii.edu/Instruments/Spectroscopy/Espadons/
This CFHT Web page is maintained by Nadine Manset (manset -=AT=- cfht.hawaii.edu)