Administration of network nodes

Overview

As of mid-2024, Orcasound nodes are mostly deployed as Raspberry Pi computers with the Pisound HAT. The swarm is managed by cloning a working SD card and modifying the .env file to update the node name and other metadata. Remote access is accomplished via Dataplicity.

A goal for 2024 is to investigate swarm management tools like Balena and Nerves, and ultimately migrate to one of them (from Dataplicity) at the same time we release a new version of the orcanode code base.

Network administration

This section includes step-by-step guides to adding a hydrophone node (live audio "feed") to the Orcasound network, adjusting them once they are functional, and remotely investigating their performance.

Set up a new node in the Orcasound hydrophone network

1. Procure and set up all the recommended Orcasound hardware and software, including a high-endurance SD card with the orcanode software installed.
2. Power up the node computer by attaching it via a live Power Over Ethernet (POE) cable.
3. Use a pair of headphones to monitor the live audio signal (via the output jack on the Pisound DAC HAT) and do a "tap test" on the hydrophone(s) to ensure there is a mono or stereo signal and that the software is running locally (with LOOPBACK enabled).
4. Follow the general Orcasound trouble-shooting procedures to be sure the Internet connection is working and the audio data are streaming to cloud-based storage.
5. Deploy the hydrophone!
6. Do a "rock test" to ensure the hydrophone is still working and to measure end-to-end latency of the streaming system.
7. Add the new node to the feed table for the live-listening web app via the Create button within the orcasite admin UI. If you don't yet have an admin account, first register and then request upgrade to administrator role via email to admin@orcasound.net ...
8. Update the new feed page for the location with deployment details and a site photo.
9. Add the new node to the OrcaHello AI pipeline (manually in 2024, possibly automated in 2025). See Michelle's step-by-step guide to adding a node to the ML pipeline.
10. Add a Mezmo view for viewing the logs. One way to do this is by going to the Orcanode Monitor dashboard, clicking on the Mezmo cell for the node that says "NoView" to bring up a view for that node, and then clicking Save.

Adjust settings on a remote node

Change configuration via environment variables

1. Login to Dataplicity using the shared administrative password
2. Select the node of interest
3. Change to the pi user with this command: su pi
4. Enter the pi user password for that node
5. As user pi change to the home directory: /home/pi with the command cd ~ or cd /home/pi
6. Change to the orcanode directory that holds the Orcasound streaming code (~/orcanode or ~/orcanode/node)
7. Edit the .env file (e.g. with vi or nano)
8. If you want new settings to take effect immediately (as opposed to upon the next scheduled container restart), try these commands:

docker-compose down

and (after ~15 seconds for the Docker container to stop) then

docker-compose up -d

to restart the container.

Adjust cron jobs for container restarts

Pending a re-write of the orcanode code, we use a cronjob to recover from an extended power loss and prevent the streaming audio manifest file from getting too large. Here is the current method for editing the cron jobs (assuming you've gained remote access via e.g. Dataplicity and are user pi):

1. Execute sudo crontab -e to edit the root user cron job table.
2. Edit the last couple lines of the crontab (see example below). The default editor is Vi, so to save any changes and close the editor, use the command ZZ

Crontab jobs (last updated 9/18/2024):

# m h  dom mon dow   command
0 0 * * * /usr/local/bin/docker-compose -f /home/pi/orcanode/node/docker-compose.yml down && /usr/local/bin/docker-compose -f /home/pi/orcanode/node/docker-compose.yml up -d
@reboot sleep 60 && /usr/local/bin/docker-compose -f /home/pi/orcanode/node/docker-compose.yml down && /usr/local/bin/docker-compose -f /home/pi/orcanode/node/docker-compose.yml up -d

It may also be worth checking the local crontab for user pi by executing crontab -e and using the default Pico editor to comment out or delete any jobs so that the crontab takes no actions regarding the Docker container(s).

Access a running container on a remote node

1. Login to Dataplicity using the shared administrative password
2. Select the node of interest
3. Change to the pi user with this command: su pi
4. Enter the pi user password for that node
5. As user pi change to the home directory: /home/pi with the command cd ~ or cd /home/pi
6. Change to the orcanode directory that holds the Orcasound streaming code (~/orcanode or ~/orcanode/node)
7. Use command docker ps to find the name of the running streaming container
8. Get a shell in the running container with this command:

docker exec -it orcanode_streaming_1 /bin/bash

where the string orcanode_streaming_1 matches the name of the running streaming container. If successful, you should get a new prompt -- something like this: root@4860bba5bb76:~#

9. Execute commands to explore directories, files, and processes from perspective of the containerized running orcanode code, e.g.:

ls /tmp
ls /tmp/rpi_sunset_bay/hls
ls -lt --full-time /tmp/rpi_sunset_bay/hls/1717743618/
htop
...

Troubleshooting deployed hydrophone nodes

General trouble-shooting strategies for Orcasound nodes

About once a week, it's good to do verify that all nodes are still online and streaming live audio data from the hydrophone through cloud storage to the web-based player. Below are steps to conduct that weekly check and fix things when they break.

If a node is offline, then either the node computer (e.g. Raspberry Pi) has lost power or its internet connection. Power issues can arise if the whole site loses power, the battery backup fails, or the power over ethernet (POE) system fails. Internet can be lost if there's a WAN outage, the local router loses power, or there's a break in the ethernet connection. Finally, if you have power and ethernet, then you may need to login remotely to troubleshoot the streaming computer itself (e.g. the Raspberry Pi could have had an SD card failure that prevents the Dataplicity software from running).

Step 1: Check node status

1. As of mid-2024 you can check on your node via the Orcasound network status dashboard built by David Thaler.

2. If you are an Orcasound administrator, check to see if the node is online via Dataplicity.com. If it's offline, then the device on Dataplicity will be shown in red and remote access will not be possible:

3. If you don't have Dataplicity admin access, there are a few ways you can assess whether audio data are making it to the Amazon S3 streaming data bucket and/or your browser:

• Look at the timestamp of the latest.txt file in the S3 bucket for Orcasound streamed data via the Quilt file viewer:
  • Orcasound Lab
  • North San Juan Channel
  • Port Townsend
  • Bush Point
  • Sunset Bay
  • Point Robinson

It should look something like this, with the timestamp in the lower right part of your browser:

If it is not within 24 hours of the current time, then the node is offline, so the player may loop in your browser. (In version 3 of the live-listening app we are trying to remove/reduce this behavior, but some devices/browsers/caches remain susceptible.)

• Using the relevant link above, preview the latest.txt file by selecting it. Then copy the Unix epoch datetime from the preview which should look like this:

Then navigate to the hls folder for the node of interest within the streaming bucket and paste/enter the epoch datetime into the search filter. This should list the S3 object named in the latest.txt file which you can then select to see the latest manifest file (the one named live.m3u8). The UI should look something like this:

The date and time for the manifest file should be less than 1-2 minutes old if the node is live streaming successfully. Here is an example from 11/10/2022 that suggests the Bush Point node is still offline and most recently succeeded in updating the manifest file about 2 days ago, on 11/8/2022:

Step 2: Resolve power & Internet issues

Assuming the node is using the current (2023-24) Orcasound hardware and software solution, follow these steps to ensure the node has power and Internet access:

1. Is the UPS (Uninterruptible Power Supply) plugged in and on? If so, it should have a green light on. If not, plug it back in (if necessary) and turn it on by holding the power button down for a couple seconds. If it doesn't turn on, replace it.

2. Is the POE (Power Over Ethernet) injector plugged in and on? If so, it should have a solid green light on. If not, ensure the wall wart power supply is still plugged into the UPS (on the surge and battery side) and that the barrel connector at the termination of the wall wart's wire is inserted fully in the POE injector. If the green light doesn't turn on, then replace at least the wall wart, and possibly the POE injector as well.

Here is a photo of a UPS and POE injector in working condition:

3. If the UPS and POE injector lights are green, proceed to the Raspberry Pi and examine the POE splitter. It should have a solid yellow or green light on indicating that power is reaching it from the injector via the ethernet cable. If the light is not on, try unplugging the ethernet cable, looking for corrosion, cleaning corrosion if necessary, and re-plugging the cable. If the splitter power light still isn't on, look for damage along the existing ethernet cable and try replacing it. If that doesn't work, try replacing the splitter and/or injector.

4. If the POE splitter has power, then look at the Raspberry Pi where the USB C plug brings power to the Raspberry Pi from the POE splitter. The nearest LED should be solid red if the Raspberry Pi has power. If the red light isn't on, then try replacing the POE splitter and/or injector. If that doesn't work, replace the Raspberry Pi.

5. If the Raspberry Pi has power, then check that it also is getting an Internet connection. Look at the ethernet jack on the Raspberry Pi to see if the internet activity light (yellow-orange) is flashing intermittently. If it is not, try unplugging it, looking for corrosion, cleaning corrosion if necessary, and re-plugging the cable. If that doesn't work, consider replacing the POE splitter and/or injector and/or the ethernet cable connecting them.

6. Is the SD card active? Upon a successful bootup, you should see activity on the green LED immediately next to the red power LED. After the bootup period (about ~60 seconds), if the orcanode code is successfully reading data from the pisound device, the green LED should be intermittently, randomly flickering. If it is not (e.g. the green LED is lit continuously, you may have a bad Raspberry Pi or a corrupt SD card. Try the card in a different Raspberry Pi; if it works then the original Raspberry Pi isn't functional. Alternatively or additionally, try use a fresh Raspbian OS SD card to see if the Raspberry Pi boots up ok with it; if it does, then the orginal SD card with the orcanode code is corrupt.

7. Is the Pisound HAT recognized as an audio device? It is rare, but sometimes a Raspberry Pi will boot up, but not find an otherwise functional Pisound ADC. To double-check the Pisound board has been noticed by the Raspberry Pi, you can watch during bootup for a double red LED flash (adjacent to the XLR jacks) and/or run this command once you are logged in remotely as user pi (see next Step): arecord -L which should list about 5 entries related to the pisound audio device.

Step 3: Login remotely to restart the streaming container

This procedure requires you to be an Orcasound administrator. If you are not one and would like to be, please contact admin@orcasound.net

1. Access the node remotely via Dataplicity.com (admin login required). If node is online, then the device on Dataplicity will be shown in green. Select it to enter a remote terminal.

2. Login using the node-specific credentials.

3. Upon successful login, change to the directory where the Docker container is invoked. It should be accessible either in ~/orcanode or ~/orcanode/node (depending on the version of the orcanode code that is installed on the node computer) — so use one of these commands to change directory: ‘cd ~/orcanode’ or ‘cd ~/orcanode/node’

4. Stop any currently-running containers with this command: docker-compose down and wait ~10 seconds for completion.

5. Restart the container with this command: docker-compose up -d and wait ~15 seconds for completion.

6. Wait about a minute and then verify that audio data are streaming successfully from the node (see other troubleshoot topics above).

It may also be informative to remotely monitor the container processes, e.g. with the ‘htop’ command filtered for ‘ffmpeg’ (for the ADC process generating audio files and manifests) or the phrase ‘upload’ (for the file transfers to the S3 data buckets). If you need to study files within the running container (e.g. watching what is getting written to the /tmp file or examine logs), try this docker syntax for getting a shell in a running container:

docker exec -it orcanode_streaming_1 /bin/bash