Metrology lost

If the metrology seems unstable, look in the opcongtk window, and you may see that there is a "Heat" error. This can be fixed by power cycling the laser. Press the N_MET button or S_MET button on k108gtk, wait 5 seconds, then press it again. Off is on and on is off for that button. This process takes about 5 minutes. When finished, you will need to re-zero the cart in OPCON (otherwise you will never find the fringes!)

Issues initializing the PAVO camera
If the PAVO xwindow does not come up successfully and produces the following error, then there has been some issue with cross-mounting directories between argus and peleas:

Could not chdir to home directory /export/observer: No such file or directory
/usr/bin/xauth: error in locking authority file /export/observer/.Xauthority
/usr/local/susi/bin/run_pavo: line 14: /usr/local/susi/tmp/pavo.log: Permission denied
Connection to closed.

This shouldn't typically be an issue, but if it does happen, an argus reboot will fix it (su then reboot):
1. su
2. mount /home
3. mount /user/local/susi
(then commands from /etc/rc.local)
/usr/local/susi/bin/thor_wheel /dev/ttyUSB0 >& /usr/local/susi/tmp/filter.err &
/usr/local/susi/bin/shutter_server >& /usr/local/susi/tmp/shutter_server.err &

Attempting to reset the cable car leads to "reset timeout/check camera"

If the cart seems to be lost, with multiple attempts to "CAB RESET" resulting in "Cable car reset timeout! Check camera" (preventing absolute or relative slews), it is possible that the power to PLC is off. If so, turn "PLC" to on in K108. This would likely happen in the rare event of a major power outage.

No star in acquisition...

Check the offsets in the siderostat window. Type "zoff" to zero then, then stop and track. Note that you can type "setoff 50 100" or whever the offsets are, to start the offsets in the right place. This means that it is a good idea to record the offsets each night.

Note that if the pointing is way off, it might help to "track off" in acqlabjackgtk, stop and track the siderostat, then try a really fast spiral, e.g. "sp 10" in the acqlabjack server.

Cross-hairs in ACQLABJACK camera have shifted away from the centre

This happens when PAVO locks the tip-tilt on noise instead of an actual target, which may be due to a variable PAVO background. You will need to spiral search to re-find the star on the ACQLABJACK camera. Once you have recovered the star and centred it, click "Set Org" in the ACQLABJACK window to bring the cross-hairs back to a nominal position. You may also want to try higher PAVO gains. For bright stars, you can also set the tilt lock auto levels higher (e.g. “nal 400 800” and “sal 400 800”). It probably also makes sense to manually make a backup of /usr/local/susi/etc/acqlabjack_origin.dat with the date.

Time not in Astromod data file

The astromod file is on an NFS mount. This means there is sometimes a slight lag before the file becomes available. The solution seems to be:
wait a few seconds

However, this problem can also occur if the time on the siderostat controller is seriously out of whack.

Siderostat "Slewing failed"

If you get a "Slewing failed" message from a siderostat, it is probably OK. The error tolerance is still just a bit too tight. In the server window, type "sb" (start background). Check that it is in a reasonable place, and type "track". It will likely start tracking just fine.

Siderostat overshoots during slew

Generally, it will turn around and go back, and be OK. This seems to be a hardware dependent problem, some sids undershoot, some overshoot. I don't know if there is anything intelligent to be done about it. If it goes way too far, try "stop" and "track" again.

Recovery from Power Outage

The SUSI system should come up perfectly after a power outage . . . but this isn't always the case. One of the worst things that can happen is that the file server (peleas) comes up after all the other computers. One solution would be to reboot everything else, but alternatively, you need to (as root):
  1. "mount -a" on all computers except for peleas.
You may also need to:
  1. Restart the pico_server (copy and paste the line from rc.local on peleas) a couple of minutes after powering on the picos. See below.
  2. Cool the PAVO camera down (with coolon -70).

Can Not Find Star
Sometimes this problem relates to an encoder losing its position. This can be fixed with "elindx" and "azindx" typed into the siderostat server windows. This problem especially applies to the south siderostats as of May 2012 because of the mis-matched line drivers and receivers.

Check that the pointing offsets make sense. If pavo started servoing on noise, the pointing offsets might be absurd values. Use "zoff" in the siderostat server window to zero the offsets.

Check the autocol position with the laser. Ask for help if you don't know what this means.

The PICO server doesn't seem to work

The pico server is set up with a script to check if the pico hardware is alive, and if so, attempt to restart the server. If the hardware does not respond to a ping, it will be power cycled, and after a 45 seconds or so, the server will be restarted. There is, however, one scenario where the hardware will respond to a ping, but it will not allow the server to connect. I believe this occurs, in particular, if peleas goes down unexpectedly, and the hardware believes the connection is still open. In this case, the script gets into an infinite loop trying to restart the server. The solution is to power cycle the hardware using the K108 and then wait about a minute for the server to come back.

The PAVO fringe display has jumps, especially when there are no fringes

This is probably not a problem. Try typing "dispersion" into the PAVO server and see if the problem goes away. If so, this is just the software choosing different values of dispersion. Obviously, this is not desirable for low S/N data. In that case, type "dispersion" to turn the dispersion calculation off (and assume it is zero).

Secret PAVO Commands

OK - someone (volunteers?) needs these commands to be better documented and probably in the GUI:
  • Tip/tilt flux limits. If tip/tilt won't lock, then you need to set "nal OFFLIM ONLIM" and "sal OFFLIM ONLIM". Just like snon and snoff, the ONLIM is the total flux limit for locking the tip/tilt servo, and the OFFLIM is the total flux limit for unlocking the tip/tilt servo.

Shutter Server

Occasionally (or perhaps often for the Red Table), the shutter controller USB will disconnect and reconnect (power flicker?). This means the server software will no longer be talking to it. At the moment, the only thing that needs to be done is for the server to be restarted.

1. Find the correct process:

[observer@gareth ~]$ ps aux | grep shutter_server
observer 24022 21.4 0.0 19972 1192 ? R Aug08 239:57 shutter_server

2. Kill it:

[observer@gareth ~]$ kill 24022

3. And restart it:

[observer@gareth ~]$ /usr/local/susi/bin/shutter_server >& /usr/local/susi/tmp/shutter_server.err &

Hopefully, in the not too distant future, this will be made more user friendly.

Auto-collimate the siderostats
In peleas, invoke "astromod -a" to start give the siderostats something to track on. The auto-collimatied coordinates are stored in /usr/local/susi/etc/sidstats.

Siderostat slewing close to its clockwise (cw) or counter clockwise (ccw) limit
If you can see the string wrapped around the siderostat and the mirror is slewing away from the vidmon camera, then it is near its cw limit. If you cannot see the string wrapped around the siderostat and the mirror is slewing towards from the vidmon camera, then it is near its ccw limit. Stop the siderostat immediately! If they cannot be stop, power off the siderostat by clicking its corresponding button on K108GTK. Wait for a few minutes before switching the power on again. Upon powering up, start the SIDCON server. Slew the siderostat away from the limit using the ABS AZ and ABS EL command on its gtk client.

LDC crown wedge number is ??
Try change the setting the rate of the crown wedge and try indexing.

Envirogtk is not getting data from the EI-1050 sensor at S4
The EI-1050 has been removed from the system. You aren't going to get data.

LDC is Screwed

Apparently, you can screw up the LDC by indexing *then* escaping from limits. I think this is a software bug. If at a limit, escape from limits first.

If you go past the South limit then stall, the only way to get out of this is "set crown glass rate" in the "setup" menu (positive if you're stuck on the south, only about 1mm/s is good), then "escape from limits" then "index".

Unable to communicate with ark siderostat

ark3 -> N1 (/dev/ttyAP4)
ark8 -> N3 (/dev/ttyAP6)
ark9 -> N4 (/dev/ttyAP7)
ark7 -> S1 (/dev/ttyAP0)
ark4 -> S2 (/dev/ttyAP1)
ark5 -> S3 (/dev/ttyAP2)
ark6 -> S4 (/dev/ttyAP3)

If you can ping the ark computer, I've never seen this. Maybe the server isn't running.

If you cannot ping the ark computer, you can try resetting the PPP connection. On Peleas, as SU in /mfs/usr/local/susi_bin/ (or /usr/local/susi/bin/), launch the "restart_sid" script for the siderostat that's not answering. Example, for S3:
bash-4.1# restart_sid s3

If you still can't ping the ark computer after this, try power cycling the ark computer using k108gtk (for S3, the "S3" button), and after a few minutes to allow it to boot, try the "restart_sid" command again as above.

False indication of periscope being "IN" even when it is not (or vice versa).

If you are certain that the periscope is not indicating correctly, it can be overridden using the command 'op ?' where ? is n or s. This tells the software to ignore what the hardware is telling it and assume the periscope is where it was last commanded to go.