Is there a way to get process to exit on DB connection errors when rotating database passwords?

[tomi-bigpi]

Hi, we're setting up automatic database password rotation and running into an edge case where the container and the database might not always be in sync with a valid password.

Unfortunately it looks like the server will simply log an error ("Authentication failed against database server at...") instead of exiting the process which would then allow Kubernetes to restart the container and solve the issue (in a subset of the edge cases).

Is there a way to get the server to "crash", i.e. exit the process on fatal database connection errors?

[dosubot[bot]]

The Langfuse server does have a mechanism to exit the process on fatal errors using the shutdown or shutdownAsync methods. These methods ensure that all requests are flushed and pending requests are awaited before the process exits, which is useful in environments like Kubernetes where you want the container to restart on such errors ^[1][2]. However, specific handling for database connection errors isn't detailed in the available information. You might need to implement custom error handling logic to trigger these shutdown methods when a database connection error occurs.

To continue talking to Dosu, mention @dosu.

[marcklingen]

this answer does not make sense, please ignore

[marcklingen]

Yes, use the health check, which should fail if the database is unavailable (for example, when the database password changes). This will allow you to terminate the pods that still have the old credentials.

For more information, visit: https://langfuse.com/docs/deployment/self-host#health-and-readiness-check-endpoint

[tomi-bigpi]

Hmm, I wonder if I've run into a bug then? The both health checks are returning:

{"status":"OK","version":"2.88.0"}

This is for a pod that still has the old credentials, and is failing SSO with the following errors:

Invalid `prisma.account.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`. {
  message: '\n' +
    'Invalid `prisma.account.findUnique()` invocation:\n' +
    '\n' +
    '\n' +
    'Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.\n' +
    '\n' +
    'Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.',
  stack: 'PrismaClientKnownRequestError: \n' +
    'Invalid `prisma.account.findUnique()` invocation:\n' +
    '\n' +
    '\n' +
    'Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.\n' +
    '\n' +
    'Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.\n' +
    '    at Mn.handleRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7753)\n' +
    '    at Mn.handleAndLogRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7061)\n' +
    '    at Mn.request (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:6745)\n' +
    '    at async l (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:130:9633)\n' +
    '    at async getUserByAccount (/app/node_modules/.pnpm/@next-auth+prisma-adapter@1.0.7_@prisma+client@5.20.0_prisma@5.20.0__next-auth@4.24.7_next@14_ldyegm3dulwkgddylfl5nzex6q/node_modules/@next-auth/prisma-adapter/dist/index.js:211:29)',
  name: 'PrismaClientKnownRequestError'
}
[next-auth][error][OAUTH_CALLBACK_HANDLER_ERROR]
https://next-auth.js.org/errors#oauth_callback_handler_error
Invalid `prisma.account.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`. PrismaClientKnownRequestError:
Invalid `prisma.account.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.
    at Mn.handleRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7753)
    at Mn.handleAndLogRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7061)
    at Mn.request (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:6745)
    at async l (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:130:9633)
    at async getUserByAccount (/app/node_modules/.pnpm/@next-auth+prisma-adapter@1.0.7_@prisma+client@5.20.0_prisma@5.20.0__next-auth@4.24.7_next@14_ldyegm3dulwkgddylfl5nzex6q/node_modules/@next-auth/prisma-adapter/dist/index.js:211:29) {
  name: 'GetUserByAccountError',
  code: 'P1000'

And then also for tracing:

2024-11-12T13:52:37.185Z error 	Error verifying auth header:
Invalid `prisma.apiKey.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.
Invalid `prisma.apiKey.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.
PrismaClientKnownRequestError:
Invalid `prisma.apiKey.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.
    at Mn.handleRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7753)
    at Mn.handleAndLogRequestError (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:7061)
    at Mn.request (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:121:6745)
    at async l (/app/node_modules/.pnpm/@prisma+client@5.20.0_prisma@5.20.0/node_modules/@prisma/client/runtime/library.js:130:9633)
    at async p.fetchApiKeyAndAddToRedis (/app/web/.next/server/pages/api/trpc/[trpc].js:343:6672)
    at async p.verifyAuthHeaderAndReturnScope (/app/web/.next/server/pages/api/trpc/[trpc].js:343:4100)
    at async f (/app/web/.next/server/chunks/9739.js:1:1561)
    at async /app/node_modules/.pnpm/@sentry+nextjs@8.33.1_@opentelemetry+api@1.9.0_@opentelemetry+core@1.26.0_@opentelemetry+api@_7w2pabnqpritttg6xtpr2rix5i/node_modules/@sentry/nextjs/build/cjs/common/wrapApiHandlerWithSentry.js:90:43
    at async K (/app/node_modules/.pnpm/next@14.2.15_@babel+core@7.24.3_@opentelemetry+api@1.9.0_@playwright+test@1.47.2_babel-plugin_kbpuz46h3a5badhs6fmvawrksq/node_modules/next/dist/compiled/next-server/pages-api.runtime.prod.js:20:16881)
    at async U.render (/app/node_modules/.pnpm/next@14.2.15_@babel+core@7.24.3_@opentelemetry+api@1.9.0_@playwright+test@1.47.2_babel-plugin_kbpuz46h3a5badhs6fmvawrksq/node_modules/next/dist/compiled/next-server/pages-api.runtime.prod.js:20:17520)
2024-11-12T13:52:37.185Z error 	error_handling_ingestion_event
Invalid `prisma.apiKey.findUnique()` invocation:


Authentication failed against database server at `some.region.rds.amazonaws.com`, the provided database credentials for `username_here` are not valid.

Please make sure to provide valid database credentials for the database server at `some.region.rds.amazonaws.com`.

[tomi-bigpi]

That's running a single pod/replicaCount=1.

[marcklingen]

Oh, sorry! I just learned from @maxdeichmann that health checks for database availability were conducted very early on. However, this practice changed some time ago because it could cause all replicas to restart if the database was briefly unavailable, resulting in significantly more downtime than necessary.

Could you create an issue for this? I think it would be a good idea to allow this as an optional flag, for example, curl http://localhost:3000/api/public/health?failIfDbUnavailable=true.

[tomi-bigpi]

Hmm, frankly that sounds like a misconfigured failureThreshold more than an issue with the health endpoint reflecting actual status...

In any case, I'll create an issue. Thanks.

[marcklingen]

Thank you, this should be a quick fix, will keep you in the loop via the issue!

[tomi-bigpi]

Cross-linking issue:
#4200