feat: configure ceramic to use postgres

[nathanielc]

This changes the ceramic pod configuration to include a postgres pod (copied from the CAS postgres pod) and configures Ceramic to use the postgres pod.

This update is breaking because the ceramic stateful_set is changed in ways that k8s doesn't allow without deleting and recreating the stateful_set. Therefore we will need to determine a rollout strategy. Can we simply rebuild all networks? If not we need to investigate how to update.

[linear]

WS1-1356 Add new Postgres pod to Keramik for indexing

[smrz2001]

This changes the ceramic pod configuration to include a postgres pod (copied from the CAS postgres pod) and configures Ceramic to use the postgres pod.

This update is breaking because the ceramic stateful_set is changed in ways that k8s doesn't allow without deleting and recreating the stateful_set. Therefore we will need to determine a rollout strategy. Can we simply rebuild all networks? If not we need to investigate how to update.

Is this preferable to having separate PG stateful sets for each Ceramic stateful set? Definitely simpler to have it part of the pod.

I don't think any of our current networks can't just be recreated. cc @3benbox

[smrz2001]

LGTM, just one question.

[3benbox]

Is this preferable to having separate PG stateful sets for each Ceramic stateful set? Definitely simpler to have it part of the pod.

It's fine for the container to be in the same pod for dev, but we should not run "in production" like that.

Since keramik is to be used for actual deployments, can we get an option to provide a DB connection string by referencing a k8s secret in the Network spec?

I don't think any of our current networks can't just be recreated. cc @3benbox

No problems recreating, the nodes aren't in use outside testing afaik.

[nathanielc]

Is this preferable to having separate PG stateful sets for each Ceramic stateful set? Definitely simpler to have it part of the pod.

Yes, a statefulset dictates the lifetime of the state it manages. In this case the state in the postgresdb is 1:1 with the state in the ipfs and ceramic containers. As such it makes sense to have all the containers each have their own pvc within that pod.

can we get an option to provide a DB connection string by referencing a k8s secret in the Network spec?

I agree exposing a connection string would be good improvement on this work. However I do not see it as being useful currently as the ceramic pod is fully managed by keramik. Are we alright adding that in a later PR?

[3benbox]

Is this preferable to having separate PG stateful sets for each Ceramic stateful set? Definitely simpler to have it part of the pod.

Yes, a statefulset dictates the lifetime of the state it manages. In this case the state in the postgresdb is 1:1 with the state in the ipfs and ceramic containers. As such it makes sense to have all the containers each have their own pvc within that pod.

Agree that the current usage patterns this is fine, for production we'll have to communicate the =ephemeral nature of these deployments for users of keramik, since it's all internal nbd atm.

can we get an option to provide a DB connection string by referencing a k8s secret in the Network spec?

I agree exposing a connection string would be good improvement on this work. However I do not see it as being useful currently as the ceramic pod is fully managed by keramik. Are we alright adding that in a later PR?

Later is fine, I think we should model the entire ceramic config and expose it though the Network spec, and this could be part of that effort.

[3benbox]

LGTM