2006 October 15 Earthquake Aftermath at CFHT|
January 11 2007
View the earthquake motion.
The page at this link requires java-enabled browser.
The bump on the horseshoe at 00:22 in hour angle has been carefully ground and is now not seen in the observations anymore. The image below shows a long exposure passing over the area.
The measurements of the horseshoe profile done after the earthquake have shown may other smaller "bumps", unrelated to the Earthquake and not seen in the observations. They are likely smooth enough to allow the telescope guiding system to compensate for them. However, they will also be ground to make the horseshoe area where the encoder roller travels as smooth as possible.
This is likely the last entry in this "earthquake aftermath" journal.
The jump in declination seen a few times since the earthquake was similar to right ascension jumps seen before the erthquake.
Actually, we also got a couple of such RA jumps since November 2. A careful look at the circumstances of their occurence showed that they are indeed similar to the rare jumps we saw happening before the earthquake. So far, the current investigation points to a glitch introduced somewhere in the hardware servo circuitry of the Telescope Control System, which is 30-year old... Could be something else too! At least, it is not erthquake related and there is nothing obviously wrong with the telescope structure.
We are only left with the small nuisance of this small bump on the horseshoe encoder track that prevent observations around 00:22 in Hour Angle.
A few nights ago, we also found a problem appearing on one image, this time related to the Declination axis.
On this exposure, the bright stars exhibited a long trail in the NS direction, extending over around 150 arc-seconds. The image was repeated and appeared to be fine the second time.
A close look on the encoders showed that both the absolute and the incremental encoders saw the jump and the telescope control system brought the telescope at the right place on the
sky. The telescope tube itself had been indeed moving with respect to the beams of the telescope mount.
An easy explanation is that something was loose somewhere at the connection between the telescope tube and the beams of the mount: When the telescope is coming close to moving from East to West, the telescope tube is drifting along its declination axis
and the declination helicoidal gear transforms this sliding into a North-South motion. Is this explanation the right one? Hard to know! The bolts tightening the declination axis to the caisson central were checked and a few of them were indeed loose on one
side and they were
tightened up yesterday. Will it cure the problem? Wait and see... By the way, this declination sudden jump was only seen twice so far. It is a minor nuisance but could be a warning for something more severe. It is also very similar to a few occurrences of
sudden jumps in Right Ascension seen before the
earthquake! More thinking and testing needed...
|Declination jump as seen on a MegaPrime image
On the first night of clear weather, Sunday night (Oct 22), three exposures with MegaPrime exhibited poor images: something funny was happening with the tracking and guiding of the telescope! Each exposure was taken again and looked fine
the second time.
A closer look the following day showed that these three bad images were all taken at the same "hour angle", which means at the same position of the horseshoe with respect to the telescope mount. More tests were made on Monday night, while the dome was closed
due to the fog on the mountain. Going through that specific hour angle value (around 00:22) definitely made the tracking weird, likely from the new encoder having some problems at that location. The telescope was used for the few hours with clear weather
that night without any glitches, the "00:22" zone being carefully avoided.
The horseshoe was looked at on Tuesday (Oct 24) and the source of the problem was found: when the telescope bumped into the earthquake restraints at the peak of the earthquake, not only was the encoder smashed and crushed!
The edge of the horseshoe was also dented by one the two restraints. Why would this affect the tracking? After the earthquake, the location of the new encoder was slightly shifted toward the edge of the horseshoe to avoid the area where the old one
smashed and damaged the surface of the
horseshoe. Getting closer to the edge, it is now rolling on the dimple made by the earthquake restraint...
For now, we avoid the area and everything is fine. As we are in queued service observing mode for the time being, it is relatively easy to do without affecting the observations. However, we want to solve the problem for good and there are two ways to do it: Remove
the dent by a fine grinding of the horseshoe, or relocate the encoder roller with respect to the horseshoe so that it avoids
the dent on one side and the dimple from the roller smash on the other side. Measurements will be done tomorrow to assess the feasibility of the latter solution. Meanwhile, smooth operation of the telescope will continue by avoiding this infamous "00:22"
|Right Ascension star wandering as seen on a MegaPrime image
October 19 - 10:00pm
To make a long story short, it was a long day of work, but we seem to be back on the sky with a telescope behaving as it should...
The new encoder assembly was installed back where the encoder used to be before the earthquake (West). Not much luck there: it was not really possible to put back the encoder roller in nice alignment with the horseshoe. As the structure supporting the
encoder assembly had been severely bumped, the new assembly was moved on the East side stand where it got in nice contact. Time to use the telescope from the control system (TCS) to check if the new encoder was sending some information back. It was and
the telescope was then moved all around through TCS, while the telescope was closely watched in the dome for unusual noises. It moved very smoothly!
Next step required the sky, and it started after sunset with a hole in the clouds but not enough time to really take useful images before the fog came in. Fortunately, humidity went down and the sky cleared up. We were able to take a first image and
check the pointing. We were of by a few tens of arc-seconds, not a surprise as the absolute encoders are likely to have moved with respect to the telescope itself. So far, so good. Now, had the telescope been bent, or some of the optics put of alignment,
or... by the earthquake? Subsequent pointing all around the sky were as good as usual once the initial offsets had been removed. So, nothing really changed and the telescope was back as good as before.
All functionalities of MegaPrime were checked and MegaPrime guiding is doing well. However, we found that the tracking rate was not too good: again, not a surprise as the encoder roller is not exactly the same as before. Conversion factor between bits and
arc-seconds was fine-tuned and should now be good enough. Fog came back and we decided to give the telescope back to science (where it belongs) so that they could use any clear sky to come :)
The last item to be checked once the sky is clear again with a decent seeing is the image quality over the MegaCam field. More on this when the weather cooperates.
October 19 - 9:00am
MegaCam is operational again! Electronic boards were checked yesterday afternoon and the camera was switched on. By the end of the day, images were checked and all chips are alive and well! As of this morning, MegaPrime looks very happy! WIRCam is now cold
and needs now to be checked... more news later!
The new encoder assembly is now at the summit. Time to mount it. Stay tuned!
October 18 - mid-afternoon
One more day of hard work and only good news to report! In Waimea, a new encoder assembly has been assembled and is ready to go to the summit tomorrow morning. More troubleshooting of the computers needed for image archiving and data processing narrowed
the remaining potential problems to only a couple of machines.
On the instrument side, Pueo, the Adaptive Optics Bonnette, has been thoroughly checked and is in good shape. The vacuum of the MegaCam cryostat is holding well and the camera is cold! The electronics board have still to be checked before
switching the electronics on.
The telescope structure was carefully examined and a few loose or bent bolts found on the way were either replaced or tightened. No way to really know if it comes from the earthquake or not... The earthquake clips were also checked and look good. The Right Ascension and Declination pins and
brakes were carefully checked: they are working well.
The telescope hydraulics has been restarted: the gap at the hydraulic pads with pressure on is nominal. The telescope was moved in Right Ascension by hand very slowly over 40 degrees and its motion was found to be very smooth, with no sign of anything bad in the gearbox: a
real relief! The teeth on the drive gear of the horseshoe that were in contact with the gearbox at the time of the earthquake were examined and they look fine. The horseshoe areas that were in contact with the pads at that time are clearly
scratched and show an excursion along the RA axis of around 3.5 cm, similar to the one we can infer from the shaving on the encoder housing (see picture from yesterday below). A similar excursion is seen on the South journal of the RA axis. These surfaces
will be cleaned up tomorrow, but there is no apparent damage bad enough that it would prevent a smooth motion of the telescope.
The Declination encoders were inspected and they look good. The final check will be done once they are really exercised.
Tomorrow, the new encoder assembly will be installed and the telescope moved through its control system. Weather permitting, the shutter should be fully opened: so far, it was checked only open half-way. The MegaCam electronics will be checked. If there is no bad surprise, the telescope could be ready to go with MegaPrime for tomorrow night, with likely the first half of the night spent
checking that pointing and tracking are good and that the telescope is behaving normally. The rest of the night could go to science... So, stay tuned!
October 17 - mid-afternoon
At the headquarters, the main remaining issue is related to the computers running the Elixir pipelines and the archives. Some of the machines were badly shaken, and the system is not back and operational yet. A quick fix could be made available quickly if needed, while troubleshooting of the current system is underway.
At the summit, the dome has been realigned and is now back and operational. The dome shutter is working well too. On the telescope side, some checks were made on the overall structure, and so far nothing bad was found on the horseshoe itself. The smashed encoder assembly and the spare one were brought down in the shop in Waimea. The base plate of the assembly is being machined to allow the remounting of the encoder assembly on a different location on the horseshoe track so that the encoder roller avoids the area smashed as the time of the earthquake. The work should be completed soon and the assembly ready to go on the telescope by the middle of the day tomorrow.
WIRCam has been pumped and is now cooling down happily. MegaCam shutter and jukebox are working well. The focal plane has been visually checked through the cryostat window and appears to be intact. MegaCam is being pumped and so far there does not seem to be any leak. Electronics board will be checked before switching the camera on again.
ESPaDOnS looks good in the coudé room.
Tomorrow, the telescope hydraulics will be started as soon as more checks of the structure have been completed. Then the telescope will be moved by hand, carefully cranking the gear box. This remains one of the main question marks: the telescope moved up and down along the RA axis by a couple of centimeters and we don't know how well the gears in the gearbox handled that. Another big question mark concerns the declination encoder: How well did it handle this big shake? It will be looked at once the RA axis has been cleared up. So, stay tuned for more news tomorrow. We don't expect to have anything operational for at least three more nights: even if we can put the telescope back together quickly, engineering will be needed on the sky to check telescope pointing and tracking. If the gearbox or the declination axis have issues, that could take longer!
|RA encoder assembly in the shop
in Waimea for repairs
|Close-up on encoder roller and housing.
Some of the horseshoe material is melted in the roller
|Close-up from the side showing the shaving
of the encoder housing by the horseshoe...
| One of the two bearings pushing the
roller on the track is broken
October 16 - mid-afternoon
Looking at the headquarters, you would hardly believe that something happened. Thanks to the hard work of all the Waimea staff not involved in summit work, the offices are now nice and clean. The computers are back on again, and the observatory is again linked to
the world through the Internet. The library looks much better (but for its ceiling), and for now the large conference room is off-limit. The ceiling will have to be redone.
At the summit, the adjustment of the dome is moving ahead nicely. It already moved by 120 degrees and should be operational again tomorrow by the end of the day. The destroyed RA encoder was dismounted, as was its spare. Most of the computers
hosting instruments and data flows are back and working. Telescope main mirror looks good, as does M2 (which is on the floor). Too early to know how MegaPrime is doing. So far, not so bad!
October 15 - evening
As you might know, Hawaii was hit by an important earthquake on Sunday, October 15. In fact three consecutive earthquakes of magnitude 6.6, 5.8 and 4.2, all centered on a few miles off the Kona coast, hit the Big Island and the other islands.
The good news is that nobody in the CFHT staff and their families were injured.
Damages to the headquarters in Waimea are somewhat limited: There are no structural problems and no issue with plumbing and electrical wiring, but for a few light fixtures here and there. Roof, windows and doors are OK. Many offices are a big mess, as many
books are no longer on their shelves and many items on tables or desks are also now on the floor
too, as are a few ceiling tiles, causing dirt from above the ceiling to cover the desks... The most badly hit rooms are the large conference room (with its ceiling in poor shape) and the library (with cabinets knocked down and ceiling in bad shape too), as you can see on the pictures below.
A first assessment of the summit shows no apparent severe damage to the building and no obvious leak of any kind. The telescope clearly moved and bent some of the earthquake restraints, which played their role! The right ascension encoder looks smashed. The dome moved too,
and it will have to be readjusted before moving again.
| Conference room
| Cleaning up