Table of Content
In Classical and QSO mode, data reduction is performed using Donati's data reduction software Libre-ESpRIT.
In order to manage many nights' worth of data, with mutiple programs carried out during each single night, CFHT has built a pipeline called Upena.
Upena uses all of the routines used by Libre-ESpRIT, but:
In March 2008, Upena v0.1 was used. For information specific to that run, please see this page.
The current version of Upena, version 1.0, provides reduced data in FITS format and Libre-ESpRIT format. Each FITS processed data is associated with 2 *.s files, with the same odometer number. One file has a 'n' in its name, for the reduction with the continuum normalized to 1; the other file has a 'u' in its name for the reduction done without any continuum normalization.
For example, the file 987123i.fits corresponds to 987123in.s and 987123iu.s, and the file 912345p.fits corresponds to 912345pn.s and 912345pu.s.
Upena has a quick-analysis component and a final reduction component.
The quick-analysis component is run during the night as exposures are taken. The calibrations taken before sunset (biases, flat fields, comparison and Fabry-Perot exposures) are used to find the geometric and wavelength calibrations. Individual scientific exposures are reduced as they are taken. As each exposure is taken (calibration and scientific exposures), checks are performed to make sure the CCD readout noise is nominal, light falls on the CCD, exposures are not saturated, S/N is as indicated by the PIs, all devices are positioned correctly in the spectrograph and polarimeter, etc.
FITS format processed files [Back to Top]
All object files (*o.fits) are individually processed to extract the spectrum (polarimetric and "Star Only" modes) or spectra ("Star + Sky" mode), whether they were taken in spectroscopic or polarimetric mode. These spectra are saved in *i.fits files. The reduction is done once with the continuum normalized to 1, and again with the continuum not normalized. In the FITS processed files, the reduction is also done the standard way (without the 'w' option), and with the 'w' option (which cancels automatic wavelength correction from telluric lines, and is used mostly by spectroscopists).
Therefore, all *i.fits files contain 4 spectra (or 12 in the case of "Star + Sky" mode).
FITS keywords identify the type of reduction and content of the file:
REDUCTIO= 'Intensity' / Type of reduction FILENAME " Base filename at reduction" DATE " UTC Date of reduction" UTIME " UTC time of reduction" HSTTIME " Local time in Hawaii of reduction" FILENAMO " Base filename at acquisition" DATE " UTC Date of reduction" UTIME " UTC time of reduction" HSTTIME " Local time in Hawaii of reduction"
For the "Star Only" and polarimetric data, the i.fits files have:
COL1 = 'Wavelength' / Normalized COL2 = 'Intensity' / Normalized COL3 = 'ErrorBar' / Normalized COL4 = 'Wavelength' / UnNormalized COL5 = 'Intensity' / UnNormalized COL6 = 'ErrorBar' / UnNormalized COL7 = 'Wavelength' / Normalized, no autowave correction COL8 = 'Intensity' / Normalized, no autowave correction COL9 = 'ErrorBar' / Normalized, no autowave correction COL10 = 'Wavelength' / UnNormalized, no autowave correction COL11 = 'Intensity' / UnNormalized, no autowave correction COL12 = 'ErrorBar' / UnNormalized, no autowave correction
For the "Star + Sky" data, the i.fits files have:
COL1 = 'Wavelength' / Normalized COL2 = 'Star ' / Normalized COL3 = 'Star+sky' / Normalized COL4 = 'Sky ' / Normalized COL5 = 'ErrorBar1' / Normalized COL6 = 'ErrorBar2' / Normalized COL7 = 'ErrorBar3' / Normalized COL8 = 'Wavelength' / UnNormalized COL9 = 'Star ' / UnNormalized COL10 = 'Star+sky' / UnNormalized COL11 = 'Sky ' / UnNormalized COL12 = 'ErrorBar1' / UnNormalized COL13 = 'ErrorBar2' / UnNormalized COL14 = 'ErrorBar3' / UnNormalized COL15 = 'Wavelength' / Normalized, no autowave correction COL16 = 'Star ' / Normalized, no autowave correction COL17 = 'Star+sky' / Normalized, no autowave correction COL18 = 'Sky ' / Normalized, no autowave correction COL19 = 'ErrorBar1' / Normalized, no autowave correction COL20 = 'ErrorBar2' / Normalized, no autowave correction COL21 = 'ErrorBar3' / Normalized, no autowave correction COL22 = 'Wavelength' / UnNormalized, no autowave correction COL23 = 'Star ' / UnNormalized, no autowave correction COL23 = 'Star ' / UnNormalized, no autowave correction COL24 = 'Star+sky' / UnNormalized, no autowave correction COL25 = 'Sky ' / UnNormalized, no autowave correction COL26 = 'ErrorBar1' / UnNormalized, no autowave correction COL27 = 'ErrorBar2' / UnNormalized, no autowave correction COL28 = 'ErrorBar3' / UnNormalized, no autowave correction
For the polarimetric case, sets of 4 exposures produce one *p.fits file, which again presents data reduced with the continuum normalized to 1 and not normalized, and also with and without the 'w' option. Whether the continuum is normalized or not (in intensity), the Stokes parameters are always un-normalized, in polarization (i.e., the column gives for example V/I * I where I is the intensity).
Un-normalized Stokes parameters go between -Intensity to +Intensity.
If the continuum is normalized (in intensity!) to 1 (to make the spectrum flat), the non-normalized Stokes parameters will go from -1 to +1 where Intensity<1 (i.e. in the continuum and absorption lines). In an an emission line, the non-normalized Stokes parameter can go above 1.
For example (taking real data):
A FITS keyword identifies the reduction:
REDUCTIO= 'Polar ' / Type of reduction
In general, reduction is done with 4 exposures, as indicated by those keywords:
POLARSEQ= 4 / Number of images in polar sequence STARTSEQ= 1 / Start image of polar sequence
If only 2 exposures were used, the above keywords will have different values.
The p.fits files have:
COL1 = 'Wavelength' / Normalized COL2 = 'Intensity' / Normalized COL3 = 'Stokes ' / Normalized COL4 = 'CheckN1 ' / Normalized COL5 = 'CheckN2 ' / Normalized COL6 = 'ErrorBar' / Normalized COL7 = 'Wavelength' / UnNormalized COL8 = 'Intensity' / UnNormalized COL9 = 'Stokes ' / UnNormalized COL10 = 'CheckN1 ' / UnNormalized COL11 = 'CheckN2 ' / UnNormalized COL12 = 'ErrorBar' / UnNormalized COL13 = 'Wavelength' / Normalized, no autowave correction COL14 = 'Intensity' / Normalized, no autowave correction COL15 = 'Stokes ' / Normalized, no autowave correction COL16 = 'CheckN1 ' / Normalized, no autowave correction COL17 = 'CheckN2 ' / Normalized, no autowave correction COL18 = 'ErrorBar' / Normalized, no autowave correction COL19 = 'Wavelength' / UnNormalized, no autowave correction COL20 = 'Intensity' / UnNormalized, no autowave correction COL21 = 'Stokes ' / UnNormalized, no autowave correction COL22 = 'CheckN1 ' / UnNormalized, no autowave correction COL23 = 'CheckN2 ' / UnNormalized, no autowave correction COL24 = 'ErrorBar' / UnNormalized, no autowave correction
Please note the following comments:
COMMENT For Stokes Q, V, and W, keep the Stokes parameter sign as is COMMENT For Stokes U, invert the sign of the Stokes parameter
FITS keywords from the 4 raw exposures are sometimes repeated 4 times (with new names) to indicate values that change (or should not change) from one exposure to the next. For example:
FILENAME= '977289p ' / Base filename at reduction FILENAM1= '977289o ' / Base filename at acquisition FILENAM2= '977290o ' / Base filename at acquisition FILENAM3= '977291o ' / Base filename at acquisition FILENAM4= '977292o ' / Base filename at acquisition EXPTIME1= '356.000 ' / Integration time (seconds) EXPTIME2= '356.000 ' / Integration time (seconds) EXPTIME3= '356.000 ' / Integration time (seconds) EXPTIME4= '356.000 ' / Integration time (seconds) MJD-OBS1= '54544.362' / Modified Julian Date at start of obs. MJD-OBS2= '54544.367' / Modified Julian Date at start of obs. MJD-OBS3= '54544.371' / Modified Julian Date at start of obs. MJD-OBS4= '54544.376' / Modified Julian Date at start of obs. MOONALT1= '81.0000 ' / Moon altitude at start in degrees MOONALT2= '80.9000 ' / Moon altitude at start in degrees MOONALT3= '80.6000 ' / Moon altitude at start in degrees MOONALT4= '80.0000 ' / Moon altitude at start in degrees
Libre-ESpRIT formatted processed data
The ASCII output from Libre-ESpRIT is also provided to PIs in *.s files, using the odometer number of the raw file (or the first raw file of a polarimetric sequence) as a base, and then adding one or more letters to identify the content:
i | intensity spectra |
p | polarimetric spectra |
n | continuum normalized to 1 |
u | continuum not normalized |
Each *.s file also has a *.out file which gives details of the reduction. In particular, the end of this file presents SNR values for each order, radial velocity correction from telluric lines (if applicable), and estimates of the V magnitude and effective temperature (in the normalized case only).
The fits2txt tool from the Opera project can be used to convert processed FITS files to ASCII files similar to the original output of Libre-ESpRIT. Click here to download fits2txt.
Ancillary data
In addition to the raw and processed data, ancillary information is also provided to PIs: details of the geometric and wavelength calibration (geom* and wcal* files), the average flat foeld (ff* file) and the comparison spectra (th* files).
Data reduction for PIs [Back to Top]
The final reduction component re-reduces all the data with additional flats taken in the morning (if available), and performs the polarimetric calculations if appropriate. All data are reduced, whether the exposures were validated or not (except for the first 5 nights of the first run where the 4th exposure of a polarimetric sequence is useless); if Libre-ESpRIT can reduce an exposure, regardless of Image Quality or extinction by clouds, the exposures are reduced. All data are reduced with the continuum normalized to 1, and also not normalized; PIs do not need to make a special request regarding continuum normalization. In QSO mode and for all PIs, the reduction is also done with and without the 'w' option, which uses or not the telluric lines to correct the wavelengths. The reduction is then done 4 different ways. For Stokes U measurements, the sign of the Stokes parameter (column 3) needs to be flipped (inverted). A comment in the FITS header indicates this.
The naming scheme for the reduced files is the following: regardless
of the Observing Mode used, each single FITS raw file ( For all PIs, the reduction is done with the 'w' option (i.e., without the
automatic wavelength correction from telluric lines) and without that
option. The result of those 2 reductions can be found in the FITS files.
Data distribution
[Back to Top]
All data will be available to PIs via the network (not on tapes
or drives) at a URL privately sent to each PI along with a key.
Data can be retrieved using wget. The following example shows how to
retrieve only *o.fits files, and the command can be run once a day to
retrieve only new files.
wget -r -nc -c -Ao.fits http://URL/[key]/
The "-r" sets directory recursion on.
"-nc" literally means no clobber, but tells wget to only get new
files. Do not "clobber" previously downloaded files.
"-c" means to continue on any partially downloaded files.
"-Ao.fits" The files to download. In this case files that end with
"o.fits"
There are also ways to download files listed in a master file. Many
options are available. For more information, please see these URLs, or
google wget: The distributed data will consist of all the raw files (including
the calibrations), all the reduced spectra in FITS format and
Libre-ESpRIT format, all
corresponding "log" file produced by Libre-ESpRIT (called *.out by
Libre-ESpRIT), and other files used by Libre-ESpPRIT such as the average
flat field, the geometric calibration data file, etc. Night
Reports are available on the web.
The Weather log section gives an idea of the sky conditions for any
night.
SkyProbe
can also be used to see plots of the extinction; note
that SkyProbe has an offset of about 0.1-0.2mag: when the sky is
photometric, without any clouds, it sill shows extinction between 0.1mag
and 0.2mag.
The Exposures log section show all validated exposures, ie, exposures
that were graded by the Queue Coordinator and judged good for
science. The column "Eval" indicates how good the exposure is. A grade
of 1 indicates sky conditions from perfect (no clouds) to about 0.5mag
extinction (with S/N ratios within 20% of the nominal value), and Image
Quality (seeing) values within what PIs have requested. A grade of 2
indicates usually more clouds (up to about 1mag), and exposures are
repeated once (as much as possible given te sky conditions) to recover
some of the lost flux. A grade of 3 indicates even worse sky conditions;
exposures are repeated twice.
After the March run, we plan on adding to the distribution other
useful information such as a Summary table of the observations, the
SkyProbe plots, weather information, etc.
(And Upena means fishnet...)
http://en.wikipedia.org/wiki/Wget
http://www.gnu.org/software/wget/
Filename Raw/Processed File type Note
b.fits Raw Bias c.fits Raw Comparison a.fits Raw Fabry-Perot f.fits Raw Flast x.fits Raw Focus Not needed by PIs o.fits Raw Object i.fits Processed Intensity Continuum
normalised and not normalized, with and without the telluric line
correction in.s Processed Intensity Libre-ESpRIT
format, continuum normalised to 1 iu.s Processed Intensity Libre-ESpRIT
format, continuum not (un) normalised p.fits Processed Polarization Continuum
normalised and not normalized, with and without the telluric line
correction pn.s Processed Polarization Libre-ESpRIT
format, continuum normalised to 1 pu.s Processed Polarization Libre-ESpRIT
format, continuum not (un) normalised i/p.out output file from the Libre-ESpRIT
reduction Contains useful information on S/N ratios, radial
velocity corrections, etc. ff Processed Average flat
field th*.s Processed Comparison exposure (1D), not
calibrated in wavelength th*.ws Processed Comparison exposure (1D),
calibrated in wavelength geom*dat geometric calibration
data geom*out geometric calibrations output file
wcal*dat wavelength calibration
data wcal*out wavelength calibrations output file
http://www.cfht.hawaii.edu/Instruments/Spectroscopy/Espadons/
This CFHT Web page is maintained by Nadine Manset (manset
-=AT=-
cfht.hawaii.edu)