Monitoring

The SCE Monitoring stack. A diagram of how we aggregate Prometheus metrics into grafana is below:

How does this work?

This use of Grafana/Prometheus/Discord bot alerting is covered in this YouTube series

Setting things up

• Create an ssh key that lives at the location ~/.ssh/id_ed25519. This key should allow for automatic access to the clark machine. To allow for automatic access, run:

ssh-copy-id -i ~/.ssh/id_ed25519 sce@10.31.5.15

• Get a webhook url for your Discord server and create a .env file in this project's directory like below:

DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/SERVER_ID/OTHER_STUFF

• Run the project with

docker-compose up -d

Note: if you are not planning to run grafana behind nginx, expose port 3000 in the yml file and remove the GF_SERVER_ROOT_URL entry in the environment section for the Grafana container.