Software:InfraTroubleshooting

Some backbone software troubleshooting info

Redis DB is down ?!

Many symptoms will reflect that the redis database cannot be accessed. Such as:

• The camera display and other scripts throw Redis DB unavailable - pass. messages all around the place.
• Keywords are all wrong.
• ScexaoStatus display doesn't update.
• Generally, everything seems to work half wrong.

What's going on?

Most likely, scexao3 rebooted due to a brown out, and the Redis server needs to be restarted.

Restarting the redis server.

• Ping scexao3
• Login to scexao3
• Check out uptime -p to make sure you have the diagnosis right
• Run a cc-rightafterboot to restart all computer-config boot stuff.

Finally, feel free to run redis-cli -h $REDIS_DB_HOST ping. Redis should answer PONG.

All is good now.

Long term fixes?

• Put scexao3 on a UPS. Done 03/2023.
• Move the redis DB on another server
• Put cc-rightafterboot in a reboot cronjob.

Restarting scxkw daemons

Ensure the following processes are running

• scexao2 daemons: ssh sc2; tmux a -t scxkw. 2 tabs.
  • scxkw-daemon-all select --fits --csvwr --status (Write fits headers; write gen2+scexao DB log; forward Redis to ScexaoStatus display).
  • scxkw-charis-header-server (Answers CHARIS requests for FITS headers).
• scexao6 daemons: ssh sc6; tmux a -t scxkw. 3 tabs.
  • scxkw-daemon-all select --fits --csvwr --pull --push (Write fits headers; write gen2+scexao DB log; Gen2 -> scexao redis; scexao redis -> Gen2).
  • scxkw-daemon-all select --archiveid --fpack (Request frame IDs for new archivable data; compress said data).
  • scxkw-rung2cam (Gen2 g2cam daemon).