WIRCam Hardware and Software Changes History

Table of contents:

The WIRCam instrument has evolved since its official first light using 2 engineering-grade chips on March 27th 2005. First science operation with 4 science-grade chips was in November 2005. The most important hardware changes occured before the September 2006 observing run (before semester 06B started) when the gain of the instrument was increased and before the July 2007 observing when cables were changed to eliminate negative crosstalk. The following table lists all hardware or software changes on the instrument or the telescope that may have affected the science images. We will slowly fill in the blanks of what happened before June 2007.

WIRCam Hardware and Software Changes

Date of change SystemWhat changed?
September 2017 Filter Wheel Failure

On Sunday, 03 September 2017, the filter wheel stopped working. It was possible to get the filter to J by increasing the motor torque limits, so observing proceeded for the night. The next day, the system was investigated and it was found that the lower wheel was moving roughly by hand indicating that the problem was internal. The instrument was removed from the telescope and allowed to warm up the next day; 5 September 2017.

After fully warmin the instrument, it was disassembled and it was found that the sapphire ball bearings in the lower filter wheel were all pretty much destroyed. It is likely that one ball failed and the rest were damaged during troubleshooting attempts. The race was also damaged by the failure. The inner bearings (stainless steel) were unaffected. In addition, the broken sapphire was completely confined to the lower wheel.

Everything was thoroughly cleaned and the lower, outer race and sapphire bearings were replaced. The upper sapphire bearings were replaced proactively. The lower, Teflon ball cages were not replaced as there were not sufficient spares available. They were thoroughly cleaned and put back into service.

Initial tests of the lower wheel were that it did not move smoothly. Measurements of the races showed that the gap on the inner circle was smaller than for the upper (original) races. This meant that the lower wheel was riding on the Teflon ball cages and not on the bearings. The cages were machined down from 1.8 mm thick to ~1.4 mm thick.

These changes worked. The system was reassembled on 12-13 September and tested warm on 15 September. The system was tested cold on 19 September.

September 2011 Chip #60 bias offset

Beginning with the WIRCam camera run 11Bw02 (Sept 6, 2011), Chip #60 (top, left chip in array) is showing a low level (< 1% in ADU counts) step offset running along the y-direction (dec axis). The location of the step changes from one image to the next, and the ADU value is marginally correlated with the exposure time; the offset is seen in all filters, so the origin is likely related to the bias voltage.

Various tests were carried out to trace the origin of this offset but the exact cause is still unclear at the time of writing (Feb 2012). The offset level is low and the effect on photometry in stacked images is well below the percent level. In addition, a correction using the median value effectively removes the offset in processed images. Given the low level of the offset, the effective correction method and the marginal effect of the photometry, the engineers have decided to monitor the chip's performance and not to attempt any correction at present.

All science data observed from 11Bw02 onward are corrected for this offset before being distributed to the PIs.

September 6, 2011 Swapped Master/Slave Detector Cables

On the first night of the WIRCam camera run 11Bw02 (Sept 6, 2011), it was noted that the Master and Slave cables had been swapped by mistake during the detector work carried out in August. This lead to significant noise features seen especially as horizontal bands in Chip #54 (bottom, left). The cables were reconnected properly the next morning and data taken on the night of 7 Sept no longer showed the banding.

Twilight flats are not available for the science images taken during the night of Sept 6/7. However, the full set of domeflats in all filters, taken at the time of the instrument exchange, are available. Therefore all the science data taken that night have been processed with the processed master domeflats only. On inspection, the processed images show only low level residual banding; the PIs of the affected programs are being informed of this at the time of data distribution.

August 2011 Hardware and Controller To make the system ITAR compliant, the DSP code was burned onto the EEPROM inside the SDSU controller. Booting of the code is now done from these PROMs. The master and slave code were accidentally reversed in this process but this was corrected on 1 September 2011. All spare cards were tested at the time of the change to EEPROM booting, so all cards in the SDSU were accessed.
July 2009 Software and Headers The Staring Mode was implemented. New header keywords such as DEFOCUS to indicate the defocus amount and WCSTARE to indicate if the Staring Mode is in operation or not, WCSTAREX,Y,E to indicate the target pixel position and extension number.
June 4 2008 Software and Headers The four on-chip guiding windows that are saved used to have problems: many frames were either skipped and lost or duplicated. Doug Teeple tracked the problems and made sure that neither case occur anymore. A new header keyword has been created to monitor if frames are skipped: WCSKIP. Please note however that guide cubes are generated to files, and no program writing to the filesystem can "absolutely" guarantee that slowdowns might not occur and cause skipped frames.
June 3 2008 Software A new command, gset, has been added to enable deliberately choosing specific guide stars so that guide cubes can be used for science projects. Thanks to Tom V.
Mid March to April 7 2008 Hardware The edge crosstalk has been removed in hardware and the noise level has been slightly reduced. More very soon on this.
February 29 2008 Image Headers An error in the values of WCGDDEC1,2,3,4 was fixed. Previously, the DEC values were divided by 15 before being converted to sexadecimal values.
October 26 2007 Image Headers The SLDATE?? keywords have been added. They represent the exact time when each image slice was obtained, starting with 01 and up to 28. Per Doug Teeple's email: "The DATE and HSTTIME keywords are now accurate to within a few milliseconds of the start of exposure of the first corner of a cube. New keywords SLDATExx (where xx is the slice number starting from 01) have been added to identify the start of each slice exposure time to within a few milliseconds. When the cube is built from the slices, the SLDATExx keywords appear in the primary header and all extension headers consecutively after the DATE keyword." They correspond to the time at the start of the reset.
October 2 2007 Image Headers Per Doug Teeple's email: "The DATE and HSTTIME headers now reflect the time at the beginning of exposure (start of the reset) of the first slice (fixing a bug where the time was the end of exposure)"
July 3 2007 Hardware One video board (the lower one y=1-512, on chip #54 - ext 3) has been swapped back to the one that was before. Noise level was higher and it experienced transient disfunctionning. This video board is special, it has a load resistor of 15k instead of 10k for all others.
June 29 2007 Hardware Cabling to the video board has been changed, i.e. the reference voltage bias (VREF) is no longer shared with the VSOURCE bias on the first video board of each array. This should totally remove the negative crosstalk that has plaggued WIRCam since day 1. The DSP code was adjusted (voltage changes) to minimize first horizontal line level offset of each amplifier. The system noise has been reduced by connecting all video boards ground wires. This should improve the noise level by a couple of ADUs. Five video boards have been replaced or swapped in the process.
June 19 2007 Hardware Video board #3 of extension 4 (third from the bottom - pixels y=1025-1536) was swapped for one having a load resistor of 10k-ohm like all other boards instead of previous 5k-ohm. It was shown that this board suffered from more positive crosstalk than all other boards. That should therefore reduce the crosstalk on that board. Also, many loose capacitors were resoldered on all boards but this should have no impact on images if not for lower noise levels. All the video boards were removed and reinserted in the process as well as one timing board.
June 6 2007 Image headers The default World Coordinate System (WCS) solution has been changed and now uses the values determined by Terapix. The 6 keywords to have changed are: CRPIX1, CRPIX2, CD1_1, CD1_2, CD2_1, CD2_2 for all 4 extensions.
Big gap to be filled here
January 19 2007 DSP/QSO All science exposures in the Low OH1/2 filters are now taken by default with Fowler=4. This is to obtain low readout noise for long exposures. But even short exposures are with fowler=4 for simplicity.
Big gap to be filled here
July 14 2006 DSP code/Detcom Marc writes that minor DSP code changes have been made to support the new version of dtecom and the new Lotus PCI driver. DSP version 2.3. Detcom now introduces bumping to position the guide stars in the center of the field within range of the ISU travel.
June 28 2006 DSP code Marc made a last minute change to "provide a patch to deal with a recurrent, intermittent problem with the guide window not working properly on only one chip after power-on."
May 3 2006 DSP code Marc sends a long email to wircam@cfht about a new DSP code. Results are 1adu lower readout noise )by slowing down the pixel rate), shorter minimum etime of 2.25se rather than 2.84s. He mentions a vertical band at the position of the guide window on extension #4 that is introduced in his code that was not there in the April run. He discusses a problem on booting the arrays with a 1 pixel shift that is not reproducible. This affects the positionning of the guide frame (star is then off center by one pixel) but does not affect the science image.
March 30 2006 DSP code Marc Baril sends an email to wircam@cfht stating that he finally has a code he likes using the enhanced mode. He describes how the high noise in some amplifiers is coming from digital signal of the other amps being sent out while reading the affected amps. This is between two observing runs. "Currently, we are reading the array twice on every read and discrading the first read to ensure absolutely pristine images, the minimum exposure time is therefore 2.8s. The exposure time requested is the exposure time obtained within 1ms."
March 15 2006 DSP code A "fast readout code" for which the first set of darks was taken March 21 and shows that the background gradient in extension 4 is now gone (as seen from images with the guider turned on).
January 18 2006 DSP code Marc writes about the exposure time not being calculated properly. The requested and actual exposure times seem to have been different up until now. Fortunately, both times are stored in the headers so we know what the actual etime was.
December 14 2005 DSP code The IMULT flash is noticed on the guide frames. Incsequential for science images except in the quality of guiding perhaps.
December 12 2005 DSP code Marc mentions synchronization problems. He found a bug in his DSP code and fixed it. It would have impacted long etimes. The loss of sync amounts to less than 100usec per exposure.
December 9 2005 DSP code Marc mentions that he and Jeff have found and fixed numerous bugs in the DSP clocks. The guide window should be a lo cleaner. He mentions problems with the guider the previous night where there were occasional flashes in the guide frames.
December 6 2005 DSP code Marc runs a new DSP code to fix a problem with the guider not being readable during science readout. Now it is readable. This introduces issues with exposure times not being exaclty what we request.
November 28 2005 DSP code Marc mentions that the two controllers may not have been synchronized in the past which means that exposure times on the two sets of detecttors may have been slightly different (msec, not sec).

Cooling Cycles and Filter Changes

Date ActionComment
June 23, 2022 Cool Down Cooled after emergency shutdown. Filter LowOH1 put in place of W, too.
May 20, 2022 Warm Up Warm up to swap W filter for LowOH1 and to remove WIRCam from cage for emergency shutdown.
January 25, 2022 Cool Down Cooldown after swapping BrGamma in for the CO filter.
January 13, 2022 Warm Up Warm up to swap filters.
October 26, 2021 Cool Down Cool down after swapping out cold head with spare.
October 7, 2021 Warm Up Warm up to swap cold head. Cold head had loss of cooling power.
July 1, 2021 Cool Down Cool down after filter change from Kcont to CO
June 2, 2021 Warm Up Warm up to change filters
January 8, 2021 Cool Down Cool down after shutdown
October 8, 2020 Warm Up Warm up in preparation for shutdown
September 22, 2020 Cool Down Cool down after filter change from CO to Kcont; shutdown delayed.
August 25, 2020 Warm Up Warm up in preparation for PM mirror coating shutdown.
August 13, 2020 Cool Down Cool down after Covid-19 warming.
April 7, 2020 Warm Up Warm up to save power during Covid-19 induced shutdown of many operations.
August 12, 2019 Cool Down Cool down after access to summit restored.
July 31, 2019 Warm Up Warm up as precaution since summit inaccessible due to protests.
April 17, 2019 Cool Down Cool down after leak check and gauge replacement.
March 29, 2019 Warm Up Warm up to leak check (seemed like a small leak was possible) and to replace vacuum gauge 1. Further troubleshooting after vacuum gauge was replaced was needed since both were still not working properly.
December 17, 2018 Cool Down Cool down after replacing the cold head. Pumped for ~1 week with heat on the getter.
November 29, 2018 Warm Up Warm up to replace cold head. Preventative maintenance.
September 19, 2017 Cool Down Cool down after repair of filter wheel. Cryovessel only pumped for about 2 days after being open for 1 week.
September 5, 2017 Warm Up Warm up to repair the filter wheel. Instrument completely disassembled.
August 16, 2017 Cool Down Cool down after shutdown. Cryovessel never open.
July 14, 2017 Warm Up Warm up and remove from cage in preparation for shutdown. Not opened and pumping in the compressor room.
March 7, 2017 Cool Down Cool down, but dewar never fully warmed. Dewar never opened, only pumped while partially warming.
March 2, 2017 Warm Up (partial) Partial warmup to around 208K in response to a prolonged power outage and loss of monitoring associated with the outage.
April 19, 2016 Cool Down Cool down after less than 24 hours of pumping in order to get on the telescope.
April 15, 2016 Warm Up Emergency warm up to fix a small vacuum leak that prevented the cryostat from staying at a reasonable pressure. Found the leak to be in the emergency blow-off valve. The valve was rebuilt and leak checks showed no leaks. The cryo-cooler was also losing power (though still functional), so it was also replaced since it was due to be replaced in August 2016 anyway.
August 10, 2015 Cool Down Replaced one vacuum gauge and replaced the narrow CO and CH4Off filters with the new CO filter and the new W filter. Checked and regreased accessible o-rings. Found one o-ring badly degraded.
July 8, 2015 Warm Up Replace vacuum gauges and filter change.
April 24, 2015 Cool Down Replaced both vacuum gauges, put more thermal grease on the cryo-cooler, and replaced the Kcont and LowOH2 filters with the CH4On and new CO filter that is too narrow (the proper filter is still being fabricated, but is very late).
April 9, 2015 Warm Up Replace vacuum gauges and filter change.
September 2, 2014 Cool Down Cool down after replacing cryo-cooler and fixing leak in slave feedthrough.
August 4, 2014 Warm Up Replace cryo-cooler (preventative), leak check.
January 23, 2014 Cool Down Cool down after filter change. >1 week pumping.
December 27, 2013 Warm Up CH4Off in for BrG and pumping.
January 28, 2013 Cool Down Cool down after power restored. Short pumping time.
January 20, 2013 Warm Up Warm up due to loss of building power.
September 28, 2012 Cool Down Filter LowOH2 in place of LowOH1 and cryo-cooler replaced.
September 24, 2012 Partial Cool Down Replaced cryo-cooler. Tested system to make sure contaminated lines not the issue. Detectors never below 273 K.
September 23, 2012 Partial Cool Down Cryo-cooler failed after ~30 minutes. Detectors never below 273 K.
September 7, 2012 Warm Up In preparation for filter change.
February 11, 2012 Cool Down Filter LowOH1 is in, LowOH2 is removed.
January 30, 2012 Warm Up In preparation for filter change.
January 12, 2012 Cool Down Cooled after connector repaired.
January 5, 2012 Warm Up A precaution in case the hermetic connector for the Master SDSU was breached during repair attempt.
August 31, 2011 Cool Down Filter LowOH2 is in, LowOH1 is removed.
August 16, 2011 Warm Up In preparation for filter change, and for alignment after shutdown.
April 6, 2011 Cool Down Filter LowOH1 is back in. CH4On is removed.
March 24, 2011 Warm Up In preparation for filter change.
September 9 2010 Cool Down Vacuum good again. Cryo-cooler replaced.
September 1 2010 Warm Up Cryo-cooler head contamination/failure.
December 13 2009 Cool Down Vaccum good again.
November 17 2009 Warm Up To improve the vaccum by pumping.
December 28 2008 Cool Down Filter Kcont is back in. CH4Off is removed.
November 17 2008 Warm Up In prevision of filter change as well as need to open the cage for secondary alignment after aluminizing.
August 23 2007 Cool Down After filters exchange and observatory shutdown.
August 15 2007 Filter Change CH4On and CH4Off in, LowOH1 and Kcont out.
August 10 2007 Warm Up In prevision of filter change and observatory shutdown for aluminizing the mirrors.
January 17 2007 Cool Down
January 9-10 2007 Warm Up To remove filters CH4On and CH4Off and put filters Kcont and Y in.
October 19 2006 Cool Down
October 15 2006 Warm Up 6.7 earthquake plus power outage.
October 10 2006 Cool Down
October 4 2006 Warm Up Cryo-cooler head contamination.
September 25 2006 Cool Down
September 19 2006 Filter Change Y filter removed, BrG filter installed.
September 15 2006 Warm Up Vaccum less than perfect and coming filter change.
April 5 2006 Cool Down
April 3 2006 Warm Up Summit power outage.
June 17 2005 Cool Down For 1st engineering run with 4 science chips which occured June 26th.
Early June 2005 Warm Up To install 4 science-grade detectors.
May 13 2005 Cool Down For 3rd engineering run with E-chips.
Early May 2005 Warm Up To put in new Lyot mask to align with spider arms.
April 16 2005 Semi Warm Up Fault somewhere - reason unknown. Warmed up to 130K then was pumped and cooled back down.
April 8-10 2005 Cool Down For 2nd engineering run with E-chips.
April 1-5 2005 Warm Up To fix problem with a Hall sensor inside the dewar.
March 18 2005 Cool Down For first engineering run with 2 E-chips.

Telescope Related Changes

Date ActionComment
October 1 2007 Mirror C02 Cleaning Don't expect ZP change.
August 20-24 2007 Mirror Aluminizing Expect a zero point change. Megacam got increases of 0.20 in u, 0.12 in g, 0.05 in r and 0.10 in z.
December 7 2006 Primary Mirror Wash No noticeable zero point change on WIRCam data.
December 15 2005 Primary Mirror Wash No obvious zero point jump in i,z Megacam data.