feat: pod monitoring (#156)

* feat: pod monitoring

* fix: review comments

* chore: renaming

* chore: add crd lookup error to log

* fix: add rbac perms for pod monitors

* chore: update podmonitor readme

---------

Co-authored-by: Ben Wilson <ben@3box.io>

k8s/operator/manifests/operator.yaml

@@ -38,6 +38,12 @@ rules:
   - apiGroups: ["keramik.3box.io"]
     resources: ["networks", "networks/status", "simulations", "simulations/status"]
     verbs: ["get", "list", "watch", "patch", "delete"]
+  - apiGroups: ["monitoring.coreos.com"]
+    resources: ["podmonitors"]
+    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch"]
 ---
 # Binding the role to the account
 kind: ClusterRoleBinding

keramik/src/monitoring.md

@@ -18,6 +18,7 @@ spec:
   replicas: 2
   monitoring:
     namespaced: true
+    podMonitor: true
 ```
 
 To view the metrics and traces port-forward the services:
@@ -40,3 +41,17 @@ This is typically a collection of metrics per simulation run and is much lighter
 Scrape the `otel:9465` endpoint if you want on the simulation metrics.
 
 >NOTE: The prometheus-0 pod will scrape all metrics so you can easily inspect all activity on the network.
+
+## Pod Monitoring
+
+This option expects the `PodMonitor` custom resource definition to already be installed in the network namespace.
+
+If `podMonitor` is enabled, the operator will create `podmonitors.monitoring.coreos.com` resources for collecting the metrics from the pods in the network.
+
+If you're using something like the grafana cloud agent, or prometheus-operator, the `podmonitors.monitoring.coreos.com` will be installed already.
+
+You can install the CRD directly from the operator:
+
+```
+    kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
+```