E2E: Flaky test errors for MySQL implementation

[s373r]

Describe the bug

We have periodic errors when working with E2E tests for MySQL. On CI it is enough (at the moment) to restart the test when errors occur, but in local development it is sometimes not enough.

The purpose of this ticket is to investigate the causes of deadlocks and how to fix them.

Steps To Reproduce

make test-e2e

Current Behavior

Sometimes, our team members get an error even on re-runs

Expected Behavior

No issues

kamu system info

latest

Logs

left: Err(InternalError { source: CommandError { cmd: ["kamu", "-q", "--system-time", "2024-07-17T00:00:00.000Z", "--e2e-output-data-path", "/tmp/.tmpamARRx/e2e-output-data.txt", "system", "api-server", "--address", "127.0.0.1"], error: CriticalFailure { source: InternalError { source: Internal(InternalError { source: Database(MySqlDatabaseError { code: Some("40001"), number: 1213, message: "Deadlock found when trying to get lock; try restarting transaction" }), backtrace: Some(Backtrace [{ fn: "internal_error::InternalError::new", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 38 }, { fn: "<E as internal_error::ErrorIntoInternal>::int_err", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 76 }, { fn: "<core::result::Result<OK,E> as internal_error::ResultIntoInternal<OK>>::int_err", file: "/home/sergeiz/kamu/kamu-cli/src/utils/internal-error/src/lib.rs", line: 91 }, { fn: "<kamu_accounts_mysql::repos::mysql_account_repository::MySqlAccountRepository as kamu_accounts::repos::account_repository::AccountRepository>::find_account_id_by_provider_identity_key::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/infra/accounts/mysql/src/repos/mysql_account_repository.rs", line: 262 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "<kamu_accounts_services::authentication_service_impl::AuthenticationServiceImpl as kamu_accounts::services::authentication_service::AuthenticationService>::login::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/domain/accounts/services/src/authentication_service_impl.rs", line: 196 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "kamu_cli::commands::system_api_server_run_command::APIServerRunCommand::get_access_token::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 117 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional_with::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 109 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 73 }, { fn: "database_common::transactions::db_transaction_manager::DatabaseTransactionRunner::transactional_with::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/utils/database-common/src/transactions/db_transaction_manager.rs", line: 111 }, { fn: "kamu_cli::commands::system_api_server_run_command::APIServerRunCommand::get_access_token::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 121 }, { fn: "<kamu_cli::commands::system_api_server_run_command::APIServerRunCommand as kamu_cli::commands::Command>::run::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/commands/system_api_server_run_command.rs", line: 132 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/1a648b397dedc98ada3dd3360f6d661ec2436c56/library/core/src/future/future.rs", line: 124 }, { fn: "kamu_cli::app::run::{{closure}}::{{closure}}::{{closure}}", file: "/home/sergeiz/kamu/kamu-cli/src/app/cli/src/app.rs", line:

Anything else?

There is an incompatibility between nextest and sqlx:

• nextest implies multiprocessing,
• while sqlx has a lock on cleanup within the current process (check fix: sqlx::macro db cleanup race condition by adding a margin to current timestamp launchbadge/sqlx#2640 (comment)).

Collaborative work will be supported in launchbadge/sqlx#3334

[s373r]

Waiting for the next PR to be merged

• Fix: nextest cleanup race condition launchbadge/sqlx#3334

[s373r]

Fixed by #767