argo-server doesn't start with secure postgres setup

[tico24]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Testing out #10300, we encountered an issue whereby the argo-server pod doesn't start. It throws the error
mkdir /home/argo/pgcerts: read only filesystem.

Simply apply the config to connect your controller with a secure Postgres database, restart the controller and then restart the argo-server. The argo server will not start up.

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Not a workflow, an addition to the wf controller configmap:

  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 0s
    nodeStatusOffLoad: true
    archive: true
    archiveTTL: 14d
    postgresql:
      host: <redacted>
      ssl: true
      sslMode: require
      port: 5432
      database: argo
      tableName: argo_workflows
      userNameSecret:
        name: argo-postgres-config
        key: username
      passwordSecret:
        name: argo-postgres-config
        key: password

[terrytangyuan]

We just came across this as well. This also happens in the controller.

[terrytangyuan]

I am reverting it in #10736, which needs to be reworked as a follow-up. cc @reddymh

[reddymh]

@terrytangyuan why this needs to be reverted and helm charts changes PR is not yet merged because this postgres ssl changes are not yet released.
This changes has been tested and live in production

argoproj/argo-helm#1807

Note: If postgres db is ssl enabled then the below flag needs to be enabled for adding the certs
readOnlyRootFilesystem: false

[terrytangyuan]

this postgres ssl changes are not yet released.

I believe we are testing the master branch so it doesn't have to be released.

[reddymh]

argoproj/argo-helm#1807

while testing master code the below changes in helm chart or manifest needs to add/update.
Could you please do the below change and test it?

           readOnlyRootFilesystem: false

If you reverted the changes then will encounter the below ssl certificate error

time="2022-12-14T20:28:04.222Z" level=fatal msg="pq: connection requires a valid client certificate"

[tico24]

I actually tested it with both permeations of that setting and it definitely does not work.

I obviously tested with the manifest, not the helm chart as the helm chart is a separate project.

[reddymh]

@tico24 Without postgres certificate how ssl works with argo workflow(on postgres side if ssl is strict) and without giving client certificate how ssl owrks with argo workflow) and clearly throws the below error?

time="2022-12-14T20:28:04.222Z" level=fatal msg="pq: connection requires a valid client certificate"

From @terrytangyuan comments, controller was throwing error mkdir /home/argo/pgcerts: read only filesystem and it clearly says read only so need to add the flag in the readOnlyRootFilesystem: false in the deployment manifests and the same has been updated in argo workflow helm charts and if you need will update manifest file in argo workflow as well.