2006 October 15 Earthquake Aftermath at CFHT

January 11 2007

View the earthquake motion.

The page at this link requires java-enabled browser.

November 25

The bump on the horseshoe at 00:22 in hour angle has been carefully ground and is now not seen in the observations anymore. The image below shows a long exposure passing over the area.
The measurements of the horseshoe profile done after the earthquake have shown may other smaller "bumps", unrelated to the Earthquake and not seen in the observations. They are likely smooth enough to allow the telescope guiding system to compensate for them. However, they will also be ground to make the horseshoe area where the encoder roller travels as smooth as possible.

This is likely the last entry in this "earthquake aftermath" journal.

November 17

The jump in declination seen a few times since the earthquake was similar to right ascension jumps seen before the erthquake. Actually, we also got a couple of such RA jumps since November 2. A careful look at the circumstances of their occurence showed that they are indeed similar to the rare jumps we saw happening before the earthquake. So far, the current investigation points to a glitch introduced somewhere in the hardware servo circuitry of the Telescope Control System, which is 30-year old... Could be something else too! At least, it is not erthquake related and there is nothing obviously wrong with the telescope structure.

We are only left with the small nuisance of this small bump on the horseshoe encoder track that prevent observations around 00:22 in Hour Angle.

November 2

A few nights ago, we also found a problem appearing on one image, this time related to the Declination axis. On this exposure, the bright stars exhibited a long trail in the NS direction, extending over around 150 arc-seconds. The image was repeated and appeared to be fine the second time. A close look on the encoders showed that both the absolute and the incremental encoders saw the jump and the telescope control system brought the telescope at the right place on the sky. The telescope tube itself had been indeed moving with respect to the beams of the telescope mount.

An easy explanation is that something was loose somewhere at the connection between the telescope tube and the beams of the mount: When the telescope is coming close to moving from East to West, the telescope tube is drifting along its declination axis and the declination helicoidal gear transforms this sliding into a North-South motion. Is this explanation the right one? Hard to know! The bolts tightening the declination axis to the caisson central were checked and a few of them were indeed loose on one side and they were tightened up yesterday. Will it cure the problem? Wait and see... By the way, this declination sudden jump was only seen twice so far. It is a minor nuisance but could be a warning for something more severe. It is also very similar to a few occurrences of sudden jumps in Right Ascension seen before the earthquake! More thinking and testing needed...

Declination jump as seen on a MegaPrime image

October 25

On the first night of clear weather, Sunday night (Oct 22), three exposures with MegaPrime exhibited poor images: something funny was happening with the tracking and guiding of the telescope! Each exposure was taken again and looked fine the second time. A closer look the following day showed that these three bad images were all taken at the same "hour angle", which means at the same position of the horseshoe with respect to the telescope mount. More tests were made on Monday night, while the dome was closed due to the fog on the mountain. Going through that specific hour angle value (around 00:22) definitely made the tracking weird, likely from the new encoder having some problems at that location. The telescope was used for the few hours with clear weather that night without any glitches, the "00:22" zone being carefully avoided.

The horseshoe was looked at on Tuesday (Oct 24) and the source of the problem was found: when the telescope bumped into the earthquake restraints at the peak of the earthquake, not only was the encoder smashed and crushed! The edge of the horseshoe was also dented by one the two restraints. Why would this affect the tracking? After the earthquake, the location of the new encoder was slightly shifted toward the edge of the horseshoe to avoid the area where the old one smashed and damaged the surface of the horseshoe. Getting closer to the edge, it is now rolling on the dimple made by the earthquake restraint...

For now, we avoid the area and everything is fine. As we are in queued service observing mode for the time being, it is relatively easy to do without affecting the observations. However, we want to solve the problem for good and there are two ways to do it: Remove the dent by a fine grinding of the horseshoe, or relocate the encoder roller with respect to the horseshoe so that it avoids the dent on one side and the dimple from the roller smash on the other side. Measurements will be done tomorrow to assess the feasibility of the latter solution. Meanwhile, smooth operation of the telescope will continue by avoiding this infamous "00:22" area!

Right Ascension star wandering as seen on a MegaPrime image

October 19 - 10:00pm

To make a long story short, it was a long day of work, but we seem to be back on the sky with a telescope behaving as it should...

The new encoder assembly was installed back where the encoder used to be before the earthquake (West). Not much luck there: it was not really possible to put back the encoder roller in nice alignment with the horseshoe. As the structure supporting the encoder assembly had been severely bumped, the new assembly was moved on the East side stand where it got in nice contact. Time to use the telescope from the control system (TCS) to check if the new encoder was sending some information back. It was and the telescope was then moved all around through TCS, while the telescope was closely watched in the dome for unusual noises. It moved very smoothly!

Next step required the sky, and it started after sunset with a hole in the clouds but not enough time to really take useful images before the fog came in. Fortunately, humidity went down and the sky cleared up. We were able to take a first image and check the pointing. We were of by a few tens of arc-seconds, not a surprise as the absolute encoders are likely to have moved with respect to the telescope itself. So far, so good. Now, had the telescope been bent, or some of the optics put of alignment, or... by the earthquake? Subsequent pointing all around the sky were as good as usual once the initial offsets had been removed. So, nothing really changed and the telescope was back as good as before.

All functionalities of MegaPrime were checked and MegaPrime guiding is doing well. However, we found that the tracking rate was not too good: again, not a surprise as the encoder roller is not exactly the same as before. Conversion factor between bits and arc-seconds was fine-tuned and should now be good enough. Fog came back and we decided to give the telescope back to science (where it belongs) so that they could use any clear sky to come :)

The last item to be checked once the sky is clear again with a decent seeing is the image quality over the MegaCam field. More on this when the weather cooperates.

October 19 - 9:00am

MegaCam is operational again! Electronic boards were checked yesterday afternoon and the camera was switched on. By the end of the day, images were checked and all chips are alive and well! As of this morning, MegaPrime looks very happy! WIRCam is now cold and needs now to be checked... more news later!

The new encoder assembly is now at the summit. Time to mount it. Stay tuned!

October 18 - mid-afternoon

One more day of hard work and only good news to report! In Waimea, a new encoder assembly has been assembled and is ready to go to the summit tomorrow morning. More troubleshooting of the computers needed for image archiving and data processing narrowed the remaining potential problems to only a couple of machines.

On the instrument side, Pueo, the Adaptive Optics Bonnette, has been thoroughly checked and is in good shape. The vacuum of the MegaCam cryostat is holding well and the camera is cold! The electronics board have still to be checked before switching the electronics on.

The telescope structure was carefully examined and a few loose or bent bolts found on the way were either replaced or tightened. No way to really know if it comes from the earthquake or not... The earthquake clips were also checked and look good. The Right Ascension and Declination pins and brakes were carefully checked: they are working well.

The telescope hydraulics has been restarted: the gap at the hydraulic pads with pressure on is nominal. The telescope was moved in Right Ascension by hand very slowly over 40 degrees and its motion was found to be very smooth, with no sign of anything bad in the gearbox: a real relief! The teeth on the drive gear of the horseshoe that were in contact with the gearbox at the time of the earthquake were examined and they look fine. The horseshoe areas that were in contact with the pads at that time are clearly scratched and show an excursion along the RA axis of around 3.5 cm, similar to the one we can infer from the shaving on the encoder housing (see picture from yesterday below). A similar excursion is seen on the South journal of the RA axis. These surfaces will be cleaned up tomorrow, but there is no apparent damage bad enough that it would prevent a smooth motion of the telescope. The Declination encoders were inspected and they look good. The final check will be done once they are really exercised.

Tomorrow, the new encoder assembly will be installed and the telescope moved through its control system. Weather permitting, the shutter should be fully opened: so far, it was checked only open half-way. The MegaCam electronics will be checked. If there is no bad surprise, the telescope could be ready to go with MegaPrime for tomorrow night, with likely the first half of the night spent checking that pointing and tracking are good and that the telescope is behaving normally. The rest of the night could go to science... So, stay tuned!

October 17 - mid-afternoon

At the headquarters, the main remaining issue is related to the computers running the Elixir pipelines and the archives. Some of the machines were badly shaken, and the system is not back and operational yet. A quick fix could be made available quickly if needed, while troubleshooting of the current system is underway.

At the summit, the dome has been realigned and is now back and operational. The dome shutter is working well too. On the telescope side, some checks were made on the overall structure, and so far nothing bad was found on the horseshoe itself. The smashed encoder assembly and the spare one were brought down in the shop in Waimea. The base plate of the assembly is being machined to allow the remounting of the encoder assembly on a different location on the horseshoe track so that the encoder roller avoids the area smashed as the time of the earthquake. The work should be completed soon and the assembly ready to go on the telescope by the middle of the day tomorrow.

WIRCam has been pumped and is now cooling down happily. MegaCam shutter and jukebox are working well. The focal plane has been visually checked through the cryostat window and appears to be intact. MegaCam is being pumped and so far there does not seem to be any leak. Electronics board will be checked before switching the camera on again. ESPaDOnS looks good in the coudé room.

Tomorrow, the telescope hydraulics will be started as soon as more checks of the structure have been completed. Then the telescope will be moved by hand, carefully cranking the gear box. This remains one of the main question marks: the telescope moved up and down along the RA axis by a couple of centimeters and we don't know how well the gears in the gearbox handled that. Another big question mark concerns the declination encoder: How well did it handle this big shake? It will be looked at once the RA axis has been cleared up. So, stay tuned for more news tomorrow. We don't expect to have anything operational for at least three more nights: even if we can put the telescope back together quickly, engineering will be needed on the sky to check telescope pointing and tracking. If the gearbox or the declination axis have issues, that could take longer!

RA encoder assembly in the shop in Waimea for repairs

Close-up on encoder roller and housing. Some of the horseshoe material is melted in the roller

Close-up from the side showing the shaving of the encoder housing by the horseshoe...

One of the two bearings pushing the roller on the track is broken

October 16 - mid-afternoon

Looking at the headquarters, you would hardly believe that something happened. Thanks to the hard work of all the Waimea staff not involved in summit work, the offices are now nice and clean. The computers are back on again, and the observatory is again linked to the world through the Internet. The library looks much better (but for its ceiling), and for now the large conference room is off-limit. The ceiling will have to be redone.

At the summit, the adjustment of the dome is moving ahead nicely. It already moved by 120 degrees and should be operational again tomorrow by the end of the day. The destroyed RA encoder was dismounted, as was its spare. Most of the computers hosting instruments and data flows are back and working. Telescope main mirror looks good, as does M2 (which is on the floor). Too early to know how MegaPrime is doing. So far, not so bad!

October 15 - evening

Conference room

Libray

Cleaning up