PRR: Add Terraform modules for metrics and alarms

[raserva]

We should set up terraform modules that set up metrics and alarming on all required metrics. This could be done as an extesnion to the existing monitoring module, which contains a dashboard.

[raserva]

Ensure the following:

• Alerts include links to a playbook
• Metric names and formats follow a standard schema

[sethvargo]

We should grab some time to talk about the "gotchas" here, maybe in the next team meeting. There are some annoying edge cases to talk through around custom metrics registration.

[yolocs]

We have some dashboard before: https://github.com/abcxyz/lumberjack/tree/main/terraform/modules/monitoring