SkyProbe Support

Skyprobe Support

This page explains the components of the SkyProbe system, and discusses trouble shooting.

Overview

The SkyProbe camera is mounted on the Caisson Central and controlled by a iopener computer in the nearby cabinets. The iopener has no on-board storage, so all images are immediately sent from this machine via the ssh connection shell to a second computer at the Summit, wekiu. All stages of the skyprobe pipeline, including image analysis, storage of the resulting data products, and the generation of summary plots, is performed on wekiu. The disk on wekiu has 100GB, enough space for about 3 months of skyprobe data from a single camera. The entire data partition is mirrored to a location in Waimea every morning. On a monthly basis, this mirror is used to make a tape backup of the old images, which are then purged from the online system.
A daemon running on wekiu requests an image from the iopener once per minute. This same daemon is responsible for turning on the camera cooling system at the start of the night, turning it off at the end of the night, and checking that the data directory for each night is created. A separate daemon running on wekiu generates the skyprobe plot, and saves a copy of the plot in the archive at the end of each night.

Camera Control

The iopener which controls the skyprobe camera runs the program sp_ccdcontrol which communicates with the skyprobe camera directly. This program is in turn called by a program from the summit computers. The general command to run the camera is sp_command, which takes several options:
 [skyprobe@wekiu:~/] sp_command -h
 USAGE: sp_command init (path)
 USAGE: sp_command cool (temp)
 USAGE: sp_command warm
 USAGE: sp_command test
 USAGE: sp_command dark (exptime) (filename)
 USAGE: sp_command expose (exptime) (filename)
 
The init command checks for the data directory and creates it if needed. The cool command tells the camera to cool to the specified temperature, usually set to -20C. The warm command turns off the cooling system. The test command runs a detailed test suite, discussed below. The dark command takes a single dark exposure of the specified exposure time, writing the output to the given file. Finally, the expose command is used to take a single image of the specified exposure time, saving the output to the specified file. Any of these commands can take an optional flag -iopener name@address. This option tells sp_command to send the commands to a different host computer, with both the machine ip address and the camera account specified. In addition, the expose command can take an option -noprocess, which tells sp_command NOT to send the image to the analysis system. The sp_command program is usually run by the skyprobe daemon on wekiu at the summit.

The sp_command test suite performs a variety of useful checks on the skyprobe camera: it checks that the iopener is alive, it turns on the cooling system, waits 1 minute for the camera to cool somewhat, takes a single dark frame, checks that the temperature information is consistent with a cooling camera, checks the flux in the dark frame, takes a single light frame, and measures the flux in the light frame. It generates a postscript image with the current date and sends it to the printer 'sps'. The entire process runs for about two minutes. This should normally be run during the daytime, when skyprobe will not be controlled by the daemon. Although no harm will come by running the test when the daemon tries to take an image, the results will likely be mis-interpreted. If it is necessary to run the skyprobe test suite at night, it is best to halt the skyprobe daemon for the duration (see below). A typical output from the test suite is given here:
 [skyprobe@wekiu:~/] sp_command test
 
 starting skyprobe test suite
 this will take about 2 minutes
 
 iopener is OK
 
 cooling camera & waiting for 1 minute
 
 taking 1 sec dark frame test.dark.fits
 progress: Exposure complete                                  (100%)
 progress: Writing FITS data to stdout00%)
 logonly: Wrote 390150 pixels, min=84, max=234
 progress: sp_ccdcontrol done.
 
 temperatures: camera: -19.5, outside: 16.3
 camera temperature OK
 
 dark flux: 104
 
 taking 0.1 sec light frame test.light.fits
 logonly: ccd_status=2 controller                             (100%)
 logonly: ccd_status=2
 logonly: ccd_status=2
 warning: Exposure took an extra 0.15 seconds
 progress: Exposure complete                                  (100%)
 progress: Writing FITS data to stdout00%)
 logonly: Wrote 390150 pixels, min=2726, max=36145
 progress: sp_ccdcontrol done.
 
 light flux: 13996.2
 approx. expected counts: 
      100 for dark dome
    15000 for dome lights on
    30000 for dome open daytime
 
 

Skyprobe Daemon

The skyprobe daemon which runs the camera is called skyprobe and is run by the user 'skyprobe' on the computer 'wekiu' at the summit. This daemon is automatically started when this machine boots. The skyprobe daemon uses the Elixir nightd daemon system to run regular events during the night, as well as initial and final nighttime events. The skyprobe daemon (simply a link to the nightd daemon) is defined by the .skyproberc file in skyprobe home directory. This configuration file is given here:
 HOME      /h/skyprobe
 
 DATA_PATH $HOME/data/rawdir
 PID_FILE  $HOME/log/skyprobe.pid
 LOG_FILE  $HOME/log/sp_daemon.log
 
 CCD_TEMP -20
 EXPTIME   30
 
 INIT_COMMAND sp_command cool $CCD_TEMP
 INIT_COMMAND sp_command init $DATA_PATH/&DATE
 MAIN_COMMAND sp_command expose $EXPTIME $DATA_PATH/&DATE/sp_&DATE_&TIME.fits
 DONE_COMMAND sp_command warm
 
 NIGHT_START 18:00
 NIGHT_STOP  06:00
 
 PERIOD 60
 TIMEOUT 300
 
In this configuration file, there are a few keywords which are required to define the daemon, and the rest simply define values to be used elsewhere in the file. The required keywords are:
 INIT_COMMAND - command executed at the start of the night
 MAIN_COMMAND - command executed regularly during the night
 DONE_COMMAND - command executed at the end of the night
 NIGHT_START - local time of the start of a night
 NIGHT_STOP - local time of the end of a night
 PERIOD - MAIN commands are started on module PERIOD seconds
 TIMEOUT - processes which don't complete in this time are killed
 PID_FILE  - process ID file, used for locking and communication
 LOG_FILE  - process log file.
 
Multiple entries of the three _COMMAND keywords are all performed in sequence at the appropriate time. A single day runs for 24 hours from 12:00 noon local time. The special variables &DATE and &TIME are converted by the daemon to the current value of the date (the UT date for 09:00 of this 'day'; ie, if it is after 12:00 local time, &DATE will reflect the UT date at 09:00 the next morning). &DATE is provided in the format YYYYMMDD. &TIME is the current local time in the format HHhMMm. In the case of skyprobe, these two are used to generate unique names for each image.

The skyprobe daemon can be easily started and stopped using the following command-line options:
 skyprobe start : start the daemon
 skyprobe stop  : stop a running daemon
 skyprobe status : check the status of a running daemon
 skyprobe config : reload the configuration file
 

Skyprobe plotting daemon

The daemon which generates the skyprobe plots is called srt and is run on 'wekiu' as user 'skyprobe'. This daemon is based on the nightd system with the following configuration file .srtrc
 INIT_COMMAND sp_plots init
 
 MAIN_COMMAND sp_plots vnccheck 
 MAIN_COMMAND sp_plots plot 
 
 DONE_COMMAND sp_plots done &DATE
 
 PID_FILE /h/skyprobe/log/srt.pid
 LOG_FILE /h/skyprobe/log/srt.log
 
 NIGHT_START 18:00
 NIGHT_STOP  06:00
 
 PERIOD 60
 TIMEOUT 300
 

Skyprobe analysis

After sp_command has succesfully taken a picture and obtained TCS and logger information, the image is ready for processing. This is perfomed by the program sp_phot. This program performs a few major tasks:
  • the raw image is flattened by the program skyproberedccd (/h/skyprobe/data/rawdir/DATE/filename.fits to /h/skyprobe/data/smpdir/DATE/filename.flt)
  • photometry is performed on the flattened image by sp_sexphot, a script which calls sextractor.
  • astrometry is performed on the resulting list by gastro using the Tycho database. The output file is smpdir/DATE/filename.flt.
  • the stellar photometry is matched and compared to the Tycho photometry, and a zero-point is measured (addstar). These measurements are added to the photometry database (/h/skyprobe/data/catdir). These programs are part of the Elixir system and are discussed in other pages.
    The program sp_phot also has a test mode called with the command sp_phot test. In this mode, sp_phot uses a sample image (/h/skyprobe/data/Refs/sample.fits) to test each of the stages above. The image is sent through all of the processing stages, and the resulting entry is searched in the photometry database. If successful, it deletes the entry and reports the success.

    Odds and Ends

    TCS and data logger information are added to the skyprobe header using the /cfht/bin/tcsh and /cfht/bin/loggerh programs. These programs, part of the general CFHT observing software system, add specified information to the FITS header of an image defined by the environment variable FFTEMPLATE.

    Logs from most of the programs go to the skyprobe home directory. Log messages from the skyprobe daemon are written to ~skyprobe/log/sp_daemon.log. Output from sp_command expose is written to ~skyprobe/log/sp_expose.log, unless the -noprocess option has been chosen. Output from the image analysis (sp_phot) is written to ~skyprobe/log/sp_phot.log. The two daemons send their log entries to ~skyprobe/log/ert.log and ~skyprobe/log/srt.log.