MegaCam Status

This page is intended to provide you with information on the June 17, 2007 failure of MegaCam, its consequences, and the recovery efforts to put it back on the telescope. This page will be updated as fresh news are available.

July 27 July 15
June 28 June 22 June 20 June 18

July 27, 2007 -5pm HST

In short, MegaCam is operational and stable! Still missing its i' filter though...

The end of the recovery process was a bumpy road:
- July 16, 17 and 18 were spent modifying the control software (with many safeguards added) and performing more tests.
- July 19 in the morning, a wrap-up meeting with the CEA and CFHT teams took place in Waimea. It allowed to assess the situation of the camera, the tasks to be scheduled later in the year or in the spring of 2008.
- The camera was installed on the instrument later on that day. Cooling started, anticipating an installation of the MegaPrime upper-end the following day.
- Friday July 20, MegaPrime upper-end was indeed installed. A few glitches were found and solved. It was late in the evening and the night was going to be lost to weather. The final setup was therefore postponed to Saturday, an opportunity to give some rest to the recovery team and the night was indeed lost to weather.
- Saturday July 21, a newly installed switch for the shutter was found to fail once on the telescope! It was decided to give up that day as the weather was not looking good and the team was getting really tired. Indeed the night was lost to weather, but for perhaps a couple of hours that could have been used on the sky.
- Sunday July 22, a fix was found for the shutter switch and the camera was left working well at the beginning of the night. It failed again quickly after and the crew could not do much in term of remote diagnosis when they arrived in Waimea, as a power failure in the Waimea area shut down all computers and communications at the headquarters for more than an hour. It was decided that it had been enough work for a day and to solve the problem the following day, even if it meant losing the night.
- Monday July 23, indeed the problem was solved (delay adjustments between the various switches checking the position of the shutter) and the camera was good to go. Early in the night, the focus stage unit was found not to work properly, while it was doing fine a short while before... Up again for our valiant crew, this time to only clear the space around a limit switch... The rest of the night was a breeze for the camera.
- Tuesday July 24, the safeguards added to the software sent a message that something was wrong with the jukebox. Even though it meant traveling to the summit again, it was actually good news because it signaled the problem which caused the incident in the first place back on June 17: a failure of the filter identification reading system, something that we know happens rarely, and even more rarely with catastrophic consequences. A new system is actually being looked at. The system was checked as being safe for the continuation of the observations, which went well for the rest of the night.
- Since then, MegaCam, and MegaPrime as a whole, have been working smoothly.

July 15, 2007 -5pm HST

Much work has been done on MegaCam these past fifteen days.
- The new set of rails were sent to the Mainland and came back on schedule nicely anodized.
- Cleanup, adjustment and many checks and measurements have been made on various mechanisms of MegaCam (clamping latch, loading arm,...)..
- The shutter was mechanically reset and modifications in its environment were made to accommodate the slight deformation of the rotating half disk, which lost its planarity with time. Many adjustments and tests were also made to ensure a long and uneventful life of the shutter mechanism.
- More brainstorming and tests led to the confirmation of the 
initial diagnosis of the source of the jukebox problem which led to the loss of the i filter. Safeguards on the software side will prevent such an event to happen again, while the hardware problem at the source of the incident (errors in the scanning of the filters) is being traced to a couple of components.

With this thorough work done to insure that MegaCam will return on the sky more robust than ever, thanks to the experience garnered on the way, more time was spent than initially anticipated. The new schedule calls for MegaPrime on the sky Wednesday July 18 at the earliest instead of Monday July 16 as initially planned.

It should be noted that all these activities have been a joint
very productive collaborative effort,by CFHT staff and CEA colleagues who came to participate in the MegaCam recovery.

A new i' filter was ordered on July 5. Procurement time is 10 to 12 weeks.  It could be there for the October run.

June 28, 2007 -11am HST

Good progress has been made in the MegaCam recovery process.
- The manufacturing of filter slide rails is nearly completed, The two pieces should leave the Island for the West Coast today or tomorrow to be anodized.They should be back in time for reassembly starting on July 9.
- Cleanup and various alignment checks were done on MegaCam.

CEA and CFHT will work together on the reassembly and tests of the filter and shutter mechanisms the week of July 9 at the summit, with preparatory work on July 5 and 6. We are still on good track for MegaCam to be on the sky on July 16. However, we are going to take whatever time it takes to make sure that MegaCam is in excellent shape before going back on the telescope.

On the filter side, we received a firm quote for a new i' filter with a delivery time of 10 to 12 weeks.  A few things to check on the specifications and a purchase order will be sent. Still on good track for the filter on the sky by October 1...

June 22, 2007 -6pm HST

We have now a better assessment of the damages on the filter system and a timeline for recovery.
- The rails holding the filter while it travels in the beam and when it is in an observing position are bent and new ones have to be fabricated. Machining has already started in our shop in Waimea. See a couple of pictures of the old rails here.
- Three of the eight rails on one of the jukebox side panels are also slightly bent. They will be rectified but empty filter frames will be placed there as a precaution (there are five more left, enough for ugrz and whatever narrow band filter would be needed).
- Various alignments will be carefully checked when re-assembling everything.
- In parallel with all the mechanical work, the code controlling the many motions involved in the filter system will be modified to better test and flag the conditions that could lead to a filter unloaded on top of the jukebox.

The current recovery schedule brings the camera back on the sky on July 16, i.e. ten days later than initially planned in the 2007A schedule. Obviously, any unforeseen problem showing up in this process is likely to postpone this date.

What about the i' filter? A new one could be available on October 1st. It is however to be confirmed. More on this later...

The fabrication of new jukebox side panels will be subcontracted soon to an outside shop. These panels will be installed at a later date (the operation does not require much disassembly of the system), giving us new components in areas where wear and tear is definitely an issue: a good thing for the future!

June 20, 2007 -6pm HST

Thanks to the work of many over the past two days, we had WIRCam on the telescope for science operation yesterday and ESPaDOnS set up and ready to start for a long run on Friday night. We are therefore covered for the coming twenty nights.

On the MegaCam side, we have now a scenario which explains how a filter could have ended up on top of the jukebox, half engaged in the beam and smashed by the jukebox itself on its way up. It is actually the only plausible scenario we found so far. Skip the following paragraph if you are not interested in technical details...

    - The filter ended up on top of the jukebox at some point while the jukebox was at its low (observing) position.
    - The jukebox started to move up and at some point the filter it was carrying on top of it became aligned with the loading rails and started to move in the beam thanks to the gravity, the telescope looking West.
    - The jukebox kept moving, caught and smashed the filter before it had enough time to get fully inserted
in the beam under its own weight and clear the way for the jukebox.
Is it really possible for the filter to end up on top of the jukebox? We found indeed a way to end up in this unfortunate state after the failure of one of the jukebox subsystems, without the control system knowing about it and acting as if filters were either stored or one in the beam. We have now to confirm this scenario and take the necessary actions to prevent this from happen again.

A closer look at the various components of MegaPrime does not show obvious catastrophic problems so far (but for the loss of the i' filter). Various bent rails and other mechanical components will have to be re-manufactured. The jukebox will be in Waimea by the end of the week for a thorough examination.

Contacts have been made with SAGEM/REOSC for the procurement of a clone of the defunct i' filter. News should be available in a couple of days.

We intend to have a first timeline of the recovery efforts by the end of the week. Stay tuned...

A few more pictures here.

June 18, 2007 -3pm HST

Last evening, following troubles with MegaCam operations, it was found that the jukebox (the mechanism used to house the filters and change them as needed)  had moved while a filter was partially inserted in the beam underneath the camera. The cause of the incident is unknown at this time. The observing night was therefore completely lost.

A first look at the damages shows the following:
    - The i' filter is destroyed and its frame is bent.
    - All the other filters are good.
    - One of the rails of the filter slides is seriously bent.

WIRCam engineering will take place tonight, following changes on the controllers to reduce crosstalk issues.

The main consequence so far is that no i' filter will be available on MegaCam for the months to come - 6 months seems like a conservative estimate - though alternative solutions are beeing looked at.

Images of the visible damages as of last night (with MegaPrime still on the telescope) are here.