Dome shutter failure recovery progress

This page provides updates on the progress of the recovery efforts following a serious dome shutter failure which happened in the early morning of Saturday April 14, 2012.

Wednesday 20 June 2012

Clear skies! The Observatory was released for normal operations by the Director of Engineering today at 14:10. After 67 days of difficult work and heroic efforts, the teams reported that the Shutter is now in an operational state. Observing will start tonight, if weather cooperates. A very big thank you to every member of the staff who was involved in this enormous and successful endeavor.

Tuesday 19 June 2012

Work is ongoing on the final adjustments of the shutter motors. The plan is to begin operations on Wednesday night, if all goes well.

Monday 18 June 2012

High winds at the summit prevented any work on the shutter. Activities should resume tomorrow.

Sunday 17 June 20123

The crew worked all day yesterday to put us back on the sky and was able to perform a first complete shutter mouvement since the failure. Unfortunately, some of the components had tangancial forces applied to them so some adjustment is needed. Observing is dealyed to at least Tuesday June 19 2012.

Friday 15 June 2012

The shutter crew is working hard on getting us ready to observe Saturday night. The plan is to make sure that no tangancial forces are applied on the gearbox shafts.

Tuesday 12 June 2012

Refurbishment of #8 hub continues today. On #8 hub, 5 of 6 pins needed attention. Looking at the currents from last week, #7 seems to have failed 2 times, it had a 1.5 second delay on opening.

Friday 8 June 2012

The daycrew got the new motor on #5 but the one taken off was a different one from the other units. Tangancial loads maybe the cause for the bending shafts. The teams are looking into this.

Wednesday June 6 2012

Motor #5 was removed and will be disasembled tomorrow. as plan is being prepared to test the shutter motions.

Tuesday 29 May 2012

The shutter was opened for the first time today since the failure. The crew did several tests and learned a lot about the behavior of the system. In Waimea, the controls team went over the proposed changes to the PLC after the tests done Thursday last week.

Wednesday 23 May 2012

Work is ongoing on a list of items to be completed to attempt opening the shutter next week.

Monday 21 May 2012

Controls group investigating operating and did the shutter hit the hard stops. There is no evidence that the shutter has hit the hard stops since OAP has been used for operation. Drive motor unit #1 is in, now bolted and 3 phase power was hooked up. It is ready to be tested/adjusted.

Thursday 17 May 2012

The daycrew has the rebuild unit #1 located up on the shutter and secured in place. The inspection of the rack tooth profiles continues.

Tuesday 15 may 2012

Work continues on the repairs of the dome shutter. The control group now understands in detail all unexplained shutter actions during initial investigations. Also, forensics analysis from a metallurgical lab on the sheared output shaft concludes that the cause is a severe torsional overload. On the mechanical side, detailed inspection of the shutter rack has started. The rack will be checked for damage as well as each teeth profile using rack tooth profile gauges.

Friday 11 May 2012

The control group has a now a good understanding of the events that occured on the day of the failure, thanks to the test controller they assembled. The controller also allowed them to understand some shutter related events that occured before the failure. The mechanical team is also making progres. The rebuild of unit #1 is now complete with the successful installation of the pinion gear of the shaft. The summit crew inspected and put back unit #2 and #6. The plan for next week is to start a close inspection of the shutter rack. Work will resume on Monday.

Wednesday 09 May 2012

Great progress was made today by the two tiger teams. The controls team finished building their test unit. They started performing tests and already explained events that occured on the day of the failure that were, up to now, difficult to explain. They will continue testing for a couple of days. On the mechanical side, the summit crew has unts #6 and #2 off the shutter and have put motor #5 back in. In Waimea, some adjustements were needed to fit motor #1 to the new gearbox assembly. The work continues on the examination of the pinion rollers and of the shutter rack.

Tuesday 08 May 2012

Work has resumed on Monday and is progressing. One of the new gearboxes was successfully mounted on the #1 drive frame. More work is needed to complete the assembly and to have a usable gearbox. Work has started on the examination of the pinion rollers using dye tests.

The control group moving forward with the test controller. They are also completing the video investigation of the shutter rack.

Friday 04 May 2012

The inspection of motors #2, 5 and 6 at the summit has started and is advancing well. Non-destructive tests on pinion hubs 1, 3 and 4 are in the works. We will look into dye penetrant and ultrasonic tests. The hope is that these will give insight into the causes of the failures. The control team has started building the test controllor for the shutter motor. Also implementations were made in the OAP software to increase the amount of information captured for future dome shutter mouvements.

Work will resume on Monday

Wednesday 02 May 2012

The work continues for both the mechanical and the control teams. It was found that motor #1 pinion gear was misaligned with the gearbox frame. The gearbox was transferred to a mechanical shop in Hilo for diagnosis. We are also awaiting diagnosis on the broken shafts. The plan to move forward and the schedule are still in the works.

Monday 30 April 2012

The #1 Motor and gear assembly are now down in Waimea. The teams spend the day planning the upcoming work and assessing our knowledge of the problem. Work is underway to assess the state of the gear assembly. In the meantime, the third gear reducer is now on it's way to Waimea.

Saturday 28 April 2012

Some members of the control group went to the summit to take video of both sides of the dome shutter rack. This will help assess the state of the racks and evaluate if they contributed to the failures. The videos will be analyzed next week.

Friday 27 April 2012

The daycrew was able to finish the removal of motor #1. They worked hard until late in the afternoon to get this done. The motor was brought back down to Waimea for inspection. Many thanks to everyone for their extra efforts on this Friday night. The control group has continued their work on explaining the odd shutter behavior seen before and during the failure.

Thursday 26 April 2012

The control team continues to find important clues regarding the events of April 14. On the mechanical side, work continues with the removal of motor #1.

Wednesday 25 April 2012

The control team has found important clues to explain some of the weird behavoir of the shutter that was witnessed at the time of the failure. They continue their work toward gaining a better understanding of the shutter controls. The mechanical team is ready to draw motor #1 out. This will happen tomorrow. The plan is to reassess the strategy after inpection of that shutter motor.

Tuesday 24 April 2012

The mechanical team has started the removal of Motor #1 for inspection while the control team is in the process of updating the documentation of the shutter control system wiring. At this point, there does not appear to be any anomalous control system function that could explain or are likely implicated in the failures of the drive system gear reducer / pinion mechanics.

The Dome DIMM was removed from it's current position at the bottom of the shutter and relocated away from the work area. Also, a camera was installed at the top of the cute to record the repair activities.

Monday 23 April 2012

The control group made good progress in understanding the shutter controls. They developed a plan to build a standalone test unit with the overload breaker, fuses and drive unit to help understand the operation of the motor controller. On the mechanical side, the plan is now to dismantle and inspect the four lowest moters on the shutter. Work is underway to do this safely.

Saturday 21 April 2012

The teams are taking a well deserved break. By the end of the week, the mechanical team was making progress on the plan to acccess the lowest 4 drives so that they can either be serviced or replaced. An inspection of the motors revealed that motor #1 pinion gear may be out of alignment. They are also looking into consulting with other companies. In the mean time, the group is also looking into the availability of gearboxes. For the moment, there are no plans to order more gearboxes. The Control group has made progress into understanding the shutter control systems. Work will resume on Monday.

Thursday 19 April 2012

The Tiger teams are working hard. The Control team has made progress in trying to understanding the shutter system and how it is controlled and what its limits are. The Mechanical team is moving forward with the planning to access the shutter motors. The new gear reducers from Suitomo have arrived. They will need to be mechanically modified before being suitable for installation on the shutter. Shutdown is still thought to last on the order of a few weeks.

Monday 16 April 2012

Planning of the repairs were completed and Tiger teams were formed to study the access to the drive assemblies for replacement and repairs of the drive and gearbox. The teams will also investigate the possible causes for the failure. At this moment, work is expected to last for at least a few weeks. Gearbox replacements were identified and ordered from Suitomo. They should arrive at CFHT within a few days.

Sunday 15 April 2012

It was determined that the failure was a rupture of the pinion drive shaft of motor #4. Work on the plans to repair the shutter and understand the causes of the failure have begun. A first meeting was held at headquarters to layout plans for the upcoming work.