Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic on invalid persistence config #1564

Closed
dtaniwaki opened this issue Aug 19, 2019 · 2 comments · Fixed by #1851
Closed

Panic on invalid persistence config #1564

dtaniwaki opened this issue Aug 19, 2019 · 2 comments · Fixed by #1851
Labels
type/bug

Comments

@dtaniwaki
Copy link
Member

dtaniwaki commented Aug 19, 2019 •
edited
Loading

Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT

What happened:
When I set up wrong postgresql connection for the Argo persistence feature, the workflow controller became panic with the following log.

time="2019-08-19T03:49:19Z" level=info msg="workflow controller configuration from workflow-controller-configmap:\nmetricsConfig:\n  enabled: true\n  path: /metrics\n  port: 9090\npersistence:\n  connectionPool:\n    maxIdleConns: 100\n    maxOpenConns: 0\n  nodeStatusOffLoad: false\n  postgresql:\n    host: localhost\n    port: 31233\n    database: postgresdb\n    tableName: argo_workflows\n    userNameSecret:\n      name: argo-postgres-config\n      key: username\n    passwordSecret:\n      name: argo-postgres-config\n      key: password\n"
time="2019-08-19T03:49:19Z" level=info msg="Persistence configuration enabled"
time="2019-08-19T03:49:19Z" level=error msg="Error in createPersistenceContext. dial tcp 127.0.0.1:31233: connect: connection refused"
time="2019-08-19T03:49:19Z" level=error msg="Error Creating Persistence context. dial tcp 127.0.0.1:31233: connect: connection refused"
time="2019-08-19T03:49:19Z" level=info msg="Starting workflow TTL controller (resync 20m0s)"
time="2019-08-19T03:49:19Z" level=info msg="Workflow Controller (version: v2.4.0+fcfd745.dirty) starting"
time="2019-08-19T03:49:19Z" level=info msg="Workers: workflow: 8, pod: 8"
time="2019-08-19T03:49:19Z" level=info msg="Watch Workflow controller config map updates"
time="2019-08-19T03:49:19Z" level=info msg="Starting prometheus metrics server at 0.0.0.0:9090/metrics"
time="2019-08-19T03:49:19Z" level=info msg="Detected ConfigMap update. Updating the controller config."
time="2019-08-19T03:49:19Z" level=info msg="workflow controller configuration from workflow-controller-configmap:\nmetricsConfig:\n  enabled: true\n  path: /metrics\n  port: 9090\npersistence:\n  connectionPool:\n    maxIdleConns: 100\n    maxOpenConns: 0\n  nodeStatusOffLoad: false\n  postgresql:\n    host: localhost\n    port: 31233\n    database: postgresdb\n    tableName: argo_workflows\n    userNameSecret:\n      name: argo-postgres-config\n      key: username\n    passwordSecret:\n      name: argo-postgres-config\n      key: password\n"
time="2019-08-19T03:49:19Z" level=info msg="Persistence configuration enabled"
time="2019-08-19T03:49:19Z" level=error msg="Error in createPersistenceContext. dial tcp 127.0.0.1:31233: connect: connection refused"
time="2019-08-19T03:49:19Z" level=error msg="Error Creating Persistence context. dial tcp 127.0.0.1:31233: connect: connection refused"
time="2019-08-19T03:49:19Z" level=info msg="Started workflow TTL worker"
E0819 03:49:19.495073       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/github.com/argoproj/argo/workflow/persist/sqldb/workflow_repository.go:61
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:285
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:232
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:173
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1333
E0819 03:49:19.495073       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/github.com/argoproj/argo/workflow/persist/sqldb/workflow_repository.go:61
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:285
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:232
/go/src/github.com/argoproj/argo/workflow/controller/controller.go:173
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/go/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1333
log: exiting because of error: log: cannot create log: open /tmp/workflow-controller.workflow-controller-5f95656f44-ffz2b.unknownuser.log.ERROR.20190819-034919.1: no such file or directory

What you expected to happen:
It should fail with an appropriate error log. I tried to fix the issue, but it's not that easy because the DB connection becomes nil in the resync config thread w/ no lock.

https://github.com/argoproj/argo/blob/d82de8813910afaf9b3fb77d029faa7953bfee3a/workflow/controller/config_controller.go#L49-L60

How to reproduce it (as minimally and precisely as possible):
Run a workflow controller with the following config.

persistence:
        connectionPool:
          maxIdleConns: 100
          maxOpenConns: 0
        nodeStatusOffLoad: false
        postgresql:
          host: invalid-host
          port: 5432
          database: postgres
          tableName: argo_workflows
          userNameSecret:
            name: argo-postgres-config
            key: username
          passwordSecret:
            name: argo-postgres-config
            key: password

Anything else we need to know?:

Environment:

  • Argo version:
$ argo version
argo: v2.4.0+6d150a1.dirty
  BuildDate: 2019-08-19T04:49:26Z
  GitCommit: 6d150a15eb96183fb21faf6a49b0997e6150880b
  GitTreeState: dirty
  GoVersion: go1.12.4
  Compiler: gc
  Platform: darwin/amd64
  • Kubernetes version :
$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-04-08T17:11:31Z"
  compiler: gc
  gitCommit: b7394102d6ef778017f2ca4046abbaa23b88c290
  gitTreeState: clean
  gitVersion: v1.14.1
  goVersion: go1.12.1
  major: "1"
  minor: "14"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-06-19T16:32:14Z"
  compiler: gc
  gitCommit: e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529
  gitTreeState: clean
  gitVersion: v1.15.0
  goVersion: go1.12.5
  major: "1"
  minor: "15"
  platform: linux/amd64
@MrSaints
Copy link

Related a317fbf ?

@dtaniwaki
Copy link
Member Author

dtaniwaki commented Sep 25, 2019 •
edited
Loading

No, it’s another issue. This panic occurs with wrong database setup.

@logicfox logicfox mentioned this issue Dec 3, 2019
Merged
@sarabala1979 sarabala1979 mentioned this issue Dec 13, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Panic if DB context has issue sarabala1979/argo
3 participants
@MrSaints @dtaniwaki @jessesuen