db-init job is getting failed #998

yildizozgur · 2023-08-31T07:08:46Z

yildizozgur
Aug 31, 2023

**Hi,

When I want to install sentry(23.6.1) with helm packages on K8. I got error from sentry-db-init job.
Error is below. I use existing potgre db which is in version 15.3

could you give me an idea what is the problem ?

Thank you.**

select version();
version
PostgreSQL 15.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4), 64-bit
(1 row)

Image Versions:

docker.io/bitnami/kafka:3.4.0-debian-11-r33
docker.io/bitnami/nginx:1.25.1-debian-11-r48
docker.io/bitnami/rabbitmq:3.11.20-debian-11-r19
docker.io/bitnami/redis:7.0.12-debian-11-r24
docker.io/bitnami/zookeeper:3.8.2-debian-11-r18
getsentry/relay:23.6.1
getsentry/sentry:23.6.1
getsentry/snuba:23.6.1
yandex/clickhouse-server:21.8.13.6-alpine

❯ kgp
NAME READY STATUS RESTARTS AGE
sentrymobil-clickhouse-0 1/1 Running 0 19m
sentrymobil-cron-7ffb7d8794-sxtcj 1/1 Running 0 19m
sentrymobil-db-check-8xtrj 0/1 Completed 0 19m
sentrymobil-db-init-htm9p 0/1 Error 0 85s
sentrymobil-db-init-t8h9p 1/1 Running 0 5s
sentrymobil-kafka-0 1/1 Running 0 19m
sentrymobil-kafka-provisioning-d7hzg 0/1 Completed 0 16m
sentrymobil-nginx-f7995b74f-kbsmk 1/1 Running 0 19m
sentrymobil-rabbitmq-0 1/1 Running 0 19m
sentrymobil-sentry-redis-master-0 1/1 Running 0 19m
sentrymobil-sentry-redis-replicas-0 1/1 Running 0 19m
sentrymobil-snuba-api-79447b4954-cwkpl 1/1 Running 0 19m
sentrymobil-snuba-db-init-cz7ct 0/1 Completed 0 15m
sentrymobil-web-7f5685d9f5-zfsph 1/1 Running 0 19m
sentrymobil-worker-987779bcb-p4vwz 1/1 Running 0 19m
sentrymobil-zookeeper-clickhouse-0 1/1 Running 0 19m

Note: sentrymobil-snuba-migrate-xxx jobs is completed successfully.
after that I kicked this job, but it is getting error.

kc logs sentrymobil-db-init-htm9p -f -->

Running migrations for default
Releases: 100% |# | ETA: --:--:--
Organization Options: 100% |# | ETA: --:--:--
Releases: 100% |# | ETA: --:--:--
Discover Saved Querys: 100% |# | ETA: --:--:--
Dashboards: 100% |# | ETA: --:--:--
Releases: 100% |# | ETA: --:--:--
Query Subscriptions: 100% |# | ETA: --:--:--
Query Subscriptions: 100% |# | ETA: --:--:--
Project Options: 100% |# | ETA: --:--:--
Repositorys: 100% |# | ETA: --:--:--
Project Options: 100% |# | ETA: --:--:--
Sentry Apps: 100% |# | ETA: --:--:--
Repository Project Path Configs: 100% |# | ETA: --:--:--
Repository Project Path Configs: 100% |# | ETA: --:--:--
Repository Project Path Configs: 100% |# | ETA: --:--:--
Integration Features: 100% |# | ETA: --:--:--
Dashboard Widget Querys: 100% |# | ETA: --:--:--
Dashboard Widget Querys: 100% |# | ETA: --:--:--
Project Ownerships: 100% |# | ETA: --:--:--
Alert Rules: 100% |# | ETA: --:--:--
Snuba Querys: 100% |# | ETA: --:--:--
Dashboard Widget Querys: 100% |# | ETA: --:--:--
Projects: 100% |# | ETA: --:--:--
Query Subscriptions: 100% |# | ETA: --:--:--
Query Subscriptions: 100% |# | ETA: --:--:--
Alert Rules: 100% |# | ETA: --:--:--
Alert Rules: 100% |# | ETA: --:--:--
Snuba Querys: 100% |# | ETA: --:--:--
Integrations: 100% |# | ETA: --:--:--
Identity Providers: 100% |# | ETA: --:--:--
External Actors: 100% |# | ETA: --:--:--
Operations to perform:
Apply all migrations: admin, auth, contenttypes, nodestore, replays, sentry, sessions, sites, social_auth
Running migrations:
Applying sentry.0001_squashed_0200_release_indices... OK
Applying contenttypes.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying nodestore.0001_initial... OK
Applying nodestore.0002_nodestore_no_dictfield... OK
Applying replays.0001_init_replays... OK
Applying replays.0002_rename_to_segment_id... OK
Applying replays.0003_add_size_to_recording_segment... OK
Applying sentry.0201_semver_package... OK
Applying sentry.0202_org_slug_upper_idx... OK
Applying sentry.0203_groupedmessage_status_index... OK
Applying sentry.0204_use_project_team_for_team_key_transactions... OK
Applying sentry.0205_semver_backfill... OK
Applying sentry.0206_organization_require_email_verification_flag... OK
Applying sentry.0207_release_adoption... OK
Applying sentry.0208_add_team_scope... OK
Applying sentry.0209_avatar_remove_file_fk... OK
Applying sentry.0210_backfill_project_transaction_thresholds... OK
Applying sentry.0211_add_artifact_count... OK
Applying sentry.0212_appconnectbuilds... OK
Applying sentry.0213_rule_project_status_owner_index... OK
Applying sentry.0214_activity_project_type_datetime_idx... OK
Applying sentry.0215_fix_state... OK
Applying sentry.0216_cdc_setup_replication_index... OK
Applying sentry.0217_debugfile_remove_project_fk... OK
Applying sentry.0218_releasefile_remove_fks... OK
Applying sentry.0219_exporteddatablob_remove_blob_fk... OK
Applying sentry.0220_add_current_release_version_group_resolution... OK
Applying sentry.0221_add_appconnect_upload_dates... OK
Applying sentry.0222_add_datetime_index_to_auditlogentry... OK
Applying sentry.0223_semver_backfill_2... OK
Applying sentry.0224_has_sessions_flag... OK
Applying sentry.0225_latest_appconnect_builds_check... OK
Applying sentry.0226_add_visits... OK
Applying sentry.0227_backfill_visits... OK
Applying sentry.0228_update_auditlog_index_with_entry... OK
Applying sentry.0229_drop_jiratenant... OK
Applying sentry.0230_sentry_app_config_jsonfield... OK
Applying sentry.0231_alert_rule_comparison_delta... OK
Applying sentry.0232_backfill_missed_semver_releases... OK
Applying sentry.0233_recreate_subscriptions_in_snuba... OK
Applying sentry.0234_grouphistory... OK
Applying sentry.0235_add_metricskeyindexer_table... OK
Applying sentry.0236_remove_legacy_key_transactions... OK
Applying sentry.0237_recreate_subscriptions_in_snuba... OK
Applying sentry.0238_remove_scheduleddeletion_aborted... OK
Applying sentry.0239_drop_scheduleddeletion_aborted... OK
Applying sentry.0240_grouphistory_index... OK
Applying sentry.0241_grouphistory_null_actor... OK
Applying sentry.0242_delete_removed_plugin_data... OK
Applying sentry.0243_delete_visualstudio_repo_data... OK
Applying sentry.0244_organization_and_integration_foreign_keys... OK
Applying sentry.0245_delete_itunes_credentials... OK
Applying sentry.0246_incident_snapshots_remove_fks... OK
Applying sentry.0247_add_color_column... OK
Applying sentry.0248_add_popularity_column... OK
Applying sentry.0249_add_avatar_type_back... OK
Applying sentry.0250_backfill_popularity... OK
Applying sentry.0251_sentryappavatar_sentryapp_not_unique... OK
Applying sentry.0252_code_mapping_cascade_delete... OK
Applying sentry.0253_add_widget_type... OK
Applying sentry.0254_org_integration_grace_period_end... OK
Applying sentry.0255_delete_code_mappings_with_no_integration... OK
Applying sentry.0256_create_docintegration_table... OK
Applying sentry.0257_add_target_id_and_type_to_integrationfeature... OK
Applying sentry.0258_create_docintegrationavatar_table... OK
Applying sentry.0259_delete_codeowners_and_code_mappings_with_no_integration... OK
Applying sentry.0260_backfill_integrationfeature... OK
Applying sentry.0261_prepare_remove_sentry_app_column... OK
Applying sentry.0262_drop_sentry_app_from_integrationfeature... OK
Applying sentry.0263_remove_not_null_integrationfeature... OK
Applying sentry.0264_use_booleanfield_docintegration... OK
Applying sentry.0265_add_userrole... OK
Applying sentry.0266_add_dashboard_widget_detail_field... OK
Applying sentry.0267_sentry_release_version_btree... OK
Applying sentry.0268_rename_issue_widget_query_fields... OK
Applying sentry.0269_alertrule_remove_unique_name... OK
Applying sentry.0270_group_history_project_date_added_index... OK
Applying sentry.0271_add_codeowners_auto_sync_setting... OK
Applying sentry.0272_seq_scan_indexes... OK
Applying sentry.0273_fix_grouplink_seqscans... OK
Applying sentry.0274_add_dashboardwidgetquery_columns_aggregates... OK
Applying sentry.0275_rule_fire_history... OK
Applying sentry.0276_rulefirehistory_date_added_index... OK
Applying sentry.0277_backfill_dashboard_widget_query_columns_aggregates... OK
Applying sentry.0278_backfill_codeowners_auto_sync_setting... OK
Applying sentry.0279_add_limit_dashboard_widget... OK
Applying sentry.0280_extend_commit_author_email_length... OK
Applying sentry.0281_add_new_indexer_table... OK
Applying sentry.0282_add_field_aliases_dashboard_widget_query... OK
Applying sentry.0283_extend_externalissue_key... OK
Applying sentry.0284_metrics_indexer_alter_seq... OK
Applying sentry.0285_add_organization_member_team_role... OK
Applying sentry.0286_backfill_alertrule_organization... OK
Applying sentry.0287_backfill_snubaquery_environment... OK
Applying sentry.0288_fix_savedsearch_state... OK
Applying sentry.0289_dashboardwidgetquery_convert_orderby_to_field... OK
Applying sentry.0290_fix_project_has_releases... OK
Applying sentry.0291_add_new_perf_indexer... OK
Applying sentry.0292_migrate_sessions_subs_user_counts... OK
Applying sentry.0293_restore_metrics_based_alerts... OK
Applying sentry.0294_alertrule_type... OK
Applying sentry.0295_backfill_alertrule_type... OK
Applying sentry.0296_alertrule_type_not_null... OK
Applying sentry.0297_backfill_alert_owners... OK
Applying sentry.0298_add_dashboard_level_filter_fields... OK
Applying sentry.0299_update_commitfilechange_filename_type... OK
Applying sentry.0300_deploy_date_finished_index... OK
Applying sentry.0301_mep_move_type_to_snuba_query... OK
Applying sentry.0302_mep_backfill_and_not_null_snuba_query_type... OK
Applying sentry.0303_backfill_msteams_default_installation_type... OK
Applying sentry.0304_sentry_functions... OK
Applying sentry.0305_create_releaseactivity_table... OK
Applying sentry.0306_sentry_functions_code... OK
Applying sentry.0307_remove_alert_type_state... OK
Applying sentry.0308_remove_alert_type_column... OK
Applying sentry.0309_fix_many_to_many_field... OK
Applying sentry.0310_sentry_functions_add_webhooks... OK
Applying sentry.0311_backfill_slack_settings... OK
Applying sentry.0312_has_profiles_flag... OK
Applying sentry.0313_sentry_functions_env_variables... OK
Applying sentry.0314_bit_int_for_org_and_project_id... OK
Applying sentry.0315_add_type_to_group... OK
Applying sentry.0316_add_context_field_to_groupowner... OK
Applying sentry.0317_groupedmessage_type_index... OK
Applying sentry.0318_add_is_homepage_to_saved_query_model... OK
Applying sentry.0319_projectownership_fallthrough_false... OK
Project Code Ownerss: 100% |# | ETA: --:--:--
App Connect Builds: 100% |# | ETA: --:--:--
Saved Searchs: 100% |##########################################| Time: 0:00:00
Saved Searchs: 100% |##########################################| Time: 0:00:00
Organizations: 100% |# | ETA: --:--:--
Monitors: 100% |# | ETA: --:--:--
Applying sentry.0320_rule_history_event_id... OK
Applying sentry.0321_suspect_committer_auto_assignment... OK
Applying sentry.0322_projectownership_fallthrough_revert... OK
Applying sentry.0323_backfill_ownership_for_projects_with_codeowners... OK
Applying sentry.0324_set_auto_assignments_true_by_default... OK
Applying sentry.0325_turn_off_suspect_committer_auto_assignment_as_default... OK
Applying sentry.0326_add_first_transaction_to_release_project... OK
Applying sentry.0327_add_appid_str... OK
Applying sentry.0328_backfill_app_id_str... OK
Applying sentry.0329_make_app_id_nullable... OK
Applying sentry.0330_add_project_replay_sent_flag... OK
Applying sentry.0331_redirect_app_id... OK
Applying sentry.0332_release_user_agent... OK
Applying sentry.0333_remove_app_id... OK
Applying sentry.0334_repositorypath_automatically_generated... OK
Applying sentry.0335_cleanup_old_savedsearches... OK
Applying sentry.0336_mark_deprecated_savedsearch_is_default_nullable... OK
Applying sentry.0337_drop_deprecated_savedsearch_columns... OK
Applying sentry.0338_add_saved_search_visibility... OK
Applying sentry.0339_mark_old_pinned_savedsearches... OK
Applying sentry.0340_drop_deprecated_savedsearch_columns_db_op... OK
Applying sentry.0341_reconstrain_savedsearch_pinning_fields... OK
Applying sentry.0342_add_spike_protection_flags_to_project... OK
Applying sentry.0343_drop_savedsearch_userdefault_fk_constraints_and_remove_state... OK
Applying sentry.0344_add_idp_flags... OK
Applying sentry.0345_create_organizationmapping... OK
Applying sentry.0346_add_flags_field_to_team_model... OK
Applying sentry.0347_add_project_has_minified_stack_trace_flag... OK
Applying sentry.0348_add_outbox_and_tombstone_tables... OK
Applying sentry.0349_issue_alert_fallback... OK
Applying sentry.0350_add_codecov_access_flag... OK
Applying sentry.0351_project_has_checkins_monitors... OK
Applying sentry.0352_add_org_role_to_team_model... OK
Applying sentry.0353_add_missing_uuid_unique_indexes... OK
Applying sentry.0354_break_saved_search_foreign_key... OK
Applying sentry.0355_break_groupowner_user_foreign_key... OK
Applying sentry.0356_break_release_owner_foreign_key... OK
Applying sentry.0357_add_region_scheduled_deletions... OK
Applying sentry.0358_break_group_related_user_fks... OK
Applying sentry.0359_checkin_file_field... OK
Applying sentry.0360_authenticator_config_type_change... OK
Applying sentry.0361_monitor_environment... OK
Applying sentry.0362_break_project_integration_fk... OK
Applying sentry.0363_debug_id_artifact_bundle... OK
Applying sentry.0364_remove_project_id_from_environment... OK
Applying sentry.0365_breaking_more_fks... OK
Applying sentry.0366_add_slug_to_monitors... OK
Applying sentry.0367_migrate_monitor_guid_to_slug... OK
Applying sentry.0368_monitor_remove_slug_nullable... OK
Applying sentry.0369_break_useroption_org_fk... OK
Applying sentry.0370_integrate_monitor_environment... OK
Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/sentry/runner/init.py", line 193, in main
func(**kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
return ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 99, in upgrade
_upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/upgrade.py", line 50, in _upgrade
dj_call_command(
File "/usr/local/lib/python3.8/site-packages/django/core/management/init.py", line 148, in call_command
return command.execute(*args, **defaults)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
post_migrate_state = executor.migrate(
File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/monkey/executor.py", line 79, in apply_migration
return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.8/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
return super().apply(project_state, schema_editor, collect_sql)
File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 527, in database_forwards
alter_together(
File "/usr/local/lib/python3.8/site-packages/django_zero_downtime_migrations/backends/postgres/schema.py", line 347, in alter_index_together
super().alter_index_together(model, old_index_together, new_index_together)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 379, in alter_index_together
self._delete_composed_index(model, fields, {'index': True}, self.sql_delete_index)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 394, in _delete_composed_index
raise ValueError("Found wrong number (%s) of constraints for %s(%s)" % (
ValueError: Found wrong number (2) of constraints for sentry_monitor(organization_id, slug)
Applying sentry.0371_monitor_make_org_slug_unique...%

INDEXes that are created by the same job

sentry=> \d sentry_monitor

Indexes:
"sentry _monitor_pkey" PRIMARY KEY, btree (id)
"sentry_monitor _guid_key" UNIQUE CONSTRAINT, btree (guid)
"sentry _monitor _organization_id_27f5b6ee" btree (organization_id)
"sentry _monitor_organization_id_slug_c4ac3a42_idx" btree (organization_id, slug)
"sentry_monitor_organization_id_slug_c4ac3a42_uniq" UNIQUE CONSTRAINT, tree (organization_id, slug)
"sentry_monitor_project_id_16569fdd" btree (project_id)
"sentry_monitor_slug_ad0f637f" btree (slug)
"sentry_monitor_slug_ad0f637f_like" btree (slug varchar_pattern_ops)
"sentry_monitor _type _next_checkin_65a5eb3a_idx" btree (type, next_checkin)
Check constraints:
"sentry_monitor _organization_id_check' CHECK (organization_id >= 0)
"sentry _monitor_project_id_check" CHECK (project_id >= 0)
"sentry_monitor_status_check" CHECK (status >= 0)
"sentry_monitor _type _check" CHECK (type >= 0)
Referenced by:
TABLE "sentry _monitorcheckin" CONSTRAINT "sentry_monitorcheckin_monitor_id_c15ff0f0_fk_sentry_monitor_id" FOREIGN KEY (monitor_id) REFERENCES sentry_monitor (id) DEFERRABLE INITIALLY DEFERRED TABLE "sentry _monitorenvironment" CONSTRAINT "sentry_monitorenviro_monitor_id_1877d3ea_fk_sentry_mo" FOREIGN KEY (monitor_id) REFERENCES sentry_monitor (id) DEFERRABLE INITIALLY DEFERRED

yildizozgur · 2023-10-10T12:43:42Z

yildizozgur
Oct 10, 2023
Author

I tested again, and it seems the reason is about using the existing postgre database. When I prefer to postgre that is created by sentry chart(postgre in a container), It is running, everything is ok. However, when I want to use an existing postgre database(a new setup, blank database) it throws an exception as mentioned above.All the versions are same with the postgre in container. I searched for it on the forums and postgre db creates indexes for primary keys automatically. "PostgreSQL automatically creates a unique index when a unique constraint or primary key is defined for a table. The index covers the columns that make up the primary key or unique constraint (a multicolumn index, if appropriate), and is the mechanism that enforces the constraint."

How can I overcome this problem? Do you have any idea?

`postgresql:
enabled: false
nameOverride: sentry-postgresql
auth:
database: sentry
replication:
enabled: false
readReplicas: 2
synchronousCommit: "on"
numSynchronousReplicas: 1
applicationName: sentry

externalPostgresql:
host: host.minikube.internal
port: 5432
username: sentryowner
password: blabla
database: sentrylocal

0 replies

patsevanton · 2024-08-30T01:20:08Z

patsevanton
Aug 30, 2024

Check after create new release: 25.2.0

add --wait --timeout=1000s
For example:

helm repo add sentry https://sentry-kubernetes.github.io/charts
helm repo update
helm install my-sentry sentry/sentry --wait --timeout=1000s

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

db-init job is getting failed #998

{{title}}

Replies: 2 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

db-init job is getting failed #998

yildizozgur Aug 31, 2023

Replies: 2 comments

yildizozgur Oct 10, 2023 Author

patsevanton Aug 30, 2024

yildizozgur
Aug 31, 2023

yildizozgur
Oct 10, 2023
Author

patsevanton
Aug 30, 2024