MegaCam
Status
This page
is intended to provide
you with information on the June 17, 2007 failure of MegaCam, its
consequences, and
the recovery efforts to put it back on the telescope. This page will be
updated as
fresh news
are available.
July
27, 2007 -5pm
HST
In short, MegaCam
is operational and stable! Still missing its i' filter though...
The end of the recovery process was a bumpy road:
- July 16, 17 and 18 were spent modifying the control software (with
many safeguards added) and performing more tests.
- July 19 in the morning, a wrap-up meeting with the CEA and CFHT teams
took place in Waimea. It allowed to assess the situation of the camera,
the tasks to be scheduled later in the year or in the spring of 2008.
- The camera was installed on the instrument later on that day. Cooling
started, anticipating an installation of the MegaPrime upper-end the
following day.
- Friday July 20, MegaPrime upper-end was indeed installed. A few
glitches were found and solved. It was late in the evening and the
night was going to be lost to weather. The final setup was therefore
postponed to Saturday, an opportunity to give some rest to the recovery
team and the night was indeed lost to weather.
- Saturday July 21, a newly installed switch for the shutter was found
to fail once on the telescope! It was decided to give up that day as
the weather was not looking good and the team was getting really tired.
Indeed the night was lost to weather, but for perhaps a couple of hours
that could have been used on the sky.
- Sunday July 22, a fix was found for the shutter switch and the camera
was left working well at the beginning of the night. It failed again
quickly after and the crew could not do much in term of remote
diagnosis when they arrived in Waimea, as a power failure in the Waimea
area shut down all computers and communications at the headquarters for
more than an hour. It was decided that it had been enough work for a
day and to solve the problem the following day, even if it meant losing
the night.
- Monday July 23, indeed the problem was solved (delay adjustments
between the various switches checking the position of the shutter) and
the camera was good to go. Early in the night, the focus stage unit was
found not to work properly, while it was doing fine a short while
before... Up again for our valiant crew, this time to only clear the
space around a limit switch... The rest of the night was a breeze for
the camera.
- Tuesday July 24, the safeguards added to the software sent a message
that something was wrong with the jukebox. Even though it meant
traveling to the summit again, it was actually good news because it
signaled the problem which caused the incident in the first place back
on June 17: a failure of the filter identification reading system,
something that we know happens rarely, and even more rarely with
catastrophic consequences. A new system is actually being looked at.
The system was checked as being safe for the continuation of the
observations, which went well for the rest of the night.
- Since then, MegaCam, and MegaPrime as a whole, have been working
smoothly.
July
15, 2007 -5pm
HST
Much work has been done on MegaCam these past fifteen days.
- The new set of rails were sent to the Mainland and came back on
schedule nicely anodized.
- Cleanup, adjustment and many checks and measurements have been made
on various mechanisms of MegaCam (clamping latch, loading arm,...)..
- The shutter was mechanically reset and modifications in its
environment were made to accommodate the slight deformation of the
rotating half disk, which lost its planarity with time. Many
adjustments and tests were also made to ensure a long and uneventful
life of the shutter mechanism.
- More brainstorming and tests led to the confirmation of the initial
diagnosis of the
source of the jukebox problem which led to the loss of the i filter.
Safeguards on the software side will prevent such an event to happen
again, while the hardware problem at the source of the incident (errors
in the scanning of the filters) is being traced to a couple of
components.
With this thorough work done to insure that MegaCam will return on the
sky more robust than ever, thanks to the experience garnered on the
way, more time was spent than initially anticipated. The new schedule
calls for MegaPrime on the sky Wednesday July 18 at the earliest
instead of Monday July 16 as initially planned.
It should be noted that all these activities have been a joint very
productive collaborative
effort,by CFHT staff and CEA colleagues who came to participate in the
MegaCam recovery.
A new i' filter was ordered on July 5. Procurement time is 10 to 12
weeks. It could be there for the October run.
June
28, 2007 -11am
HST
Good progress has been made in the MegaCam recovery process.
-
The manufacturing of filter slide rails is nearly completed, The two
pieces should leave the Island for the West Coast today or tomorrow to be
anodized.They
should be back in time for reassembly starting on July 9.
- Cleanup and various alignment checks were done on MegaCam.
CEA and CFHT will work together on the reassembly and tests of the
filter and shutter mechanisms the week of July 9 at the summit, with
preparatory work on July 5 and 6. We are still on good track for
MegaCam to be on the sky on July 16. However, we are going to take
whatever time it takes to make sure that MegaCam is in excellent shape
before going back on the telescope.
On the filter side, we received a firm quote for a new i' filter with a
delivery time of 10 to 12 weeks. A few things to check on the
specifications and a purchase order will be sent. Still on good track
for the filter on the sky by October 1...
June
22, 2007 -6pm
HST
We have now a better assessment of the damages on the filter system and
a timeline for recovery.
- The rails holding the filter while it travels in the beam and when it
is in an observing position are bent and new ones have to be
fabricated. Machining has already started in our shop in Waimea. See a
couple of pictures of the old rails here.
- Three of the eight rails on one of the jukebox side panels are also
slightly bent. They will be rectified but empty filter frames will be
placed there as a precaution (there are five more left, enough for ugrz
and whatever narrow band filter would be needed).
- Various alignments will be carefully checked when re-assembling
everything.
- In parallel with all the mechanical work, the code controlling the
many motions involved in the filter system will be modified to better
test and flag the conditions that could lead to a filter unloaded on
top of the jukebox.
The current recovery schedule brings the camera back on the sky on July
16, i.e. ten days later than initially planned in the 2007A schedule.
Obviously, any unforeseen problem showing up in this process is likely
to postpone this date.
What about the i' filter? A new one could be available on October 1st.
It is however to be confirmed. More on this later...
The fabrication of new jukebox side panels will be subcontracted soon
to an outside shop. These panels will be installed at a later date (the
operation does not require much disassembly of the system), giving us
new components in areas where wear and tear is definitely an issue: a
good thing for the future!
June
20, 2007 -6pm
HST
Thanks to the work of many over the past two days, we had WIRCam on the
telescope for science operation yesterday and ESPaDOnS set up and ready
to start for a long run on Friday night. We are therefore covered for
the coming twenty nights.
On the MegaCam side, we have now a scenario which explains how a filter
could have ended up on top of the jukebox, half engaged in the beam and
smashed by the jukebox itself on its way up. It is actually the only
plausible scenario we found so far. Skip the following paragraph if you
are not interested in technical details...
- The filter ended up on top of the jukebox
at some point while the jukebox was at its low (observing) position.
- The jukebox started to move up and at some point
the filter it was carrying on top of it became aligned with the loading
rails and started to move in the beam thanks to the gravity, the
telescope looking West.
- The jukebox kept moving, caught and smashed the
filter before it had enough time to get fully inserted in
the beam under
its own weight and clear the way for the jukebox.
Is it really possible for the filter to end up on top of the jukebox?
We found indeed a way to end up in this unfortunate state after the
failure of one of the jukebox subsystems, without the control system
knowing about it and acting as if filters were either stored or one in
the beam. We have now to confirm this scenario and take the necessary
actions to prevent this from happen again.
A closer look at the various components of MegaPrime does not show
obvious catastrophic problems so far (but for the loss of the i'
filter). Various bent rails and other mechanical components will have
to be re-manufactured. The jukebox will be in Waimea by the end of the
week for a thorough examination.
Contacts have been made with SAGEM/REOSC for the procurement of a clone
of the defunct i' filter. News should be available in a couple of days.
We intend to have a first timeline of the recovery efforts by the end
of the week. Stay tuned...
A few more pictures here.
June
18, 2007 -3pm
HST
Last evening,
following troubles with MegaCam operations,
it was found that the jukebox (the mechanism used to house the filters
and change them as needed) had moved while a filter was partially
inserted in the beam underneath the camera. The cause of the incident
is unknown at
this
time. The observing night was therefore completely lost.
A
first look at the damages shows
the following:
- The i' filter
is destroyed and its frame is bent.
- All the other
filters are good.
- One of the
rails of the filter slides is seriously
bent.
WIRCam
engineering will take place tonight, following changes on the
controllers to reduce crosstalk issues.
The
main consequence so far is
that no i' filter will be available on
MegaCam for the months to come - 6 months seems like a conservative
estimate - though alternative solutions are beeing looked at.
Images
of the visible damages as
of last night (with MegaPrime still on
the telescope) are here.