Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: internal migration in postgres #2559

Merged
merged 4 commits into from
Oct 14, 2022

Conversation

Sidddddarth
Copy link
Member

Description

Rudder server triggers internal migration when required. No longer job fetching and then copying into new DS.
Does so using a insert into newDS (select * from oldDS where non-terminal) statement.

Notion Ticket

Perform internal migrations using insert into select statements

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Oct 12, 2022

Codecov Report

Base: 43.86% // Head: 43.95% // Increases project coverage by +0.09% 🎉

Coverage data is based on head (beb21b6) compared to base (8444433).
Patch coverage: 85.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2559      +/-   ##
==========================================
+ Coverage   43.86%   43.95%   +0.09%     
==========================================
  Files         187      187              
  Lines       39084    39059      -25     
==========================================
+ Hits        17144    17169      +25     
+ Misses      20856    20808      -48     
+ Partials     1084     1082       -2     
Impacted Files Coverage Δ
services/streammanager/bqstream/bqstreammanager.go 71.59% <69.69%> (+38.69%) ⬆️
jobsdb/jobsdb.go 68.05% <92.53%> (+0.38%) ⬆️
config/backend-config/namespace_config.go 70.83% <0.00%> (-3.13%) ⬇️
utils/httputil/server.go 92.30% <0.00%> (+11.53%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@Sidddddarth Sidddddarth force-pushed the refactor.internalMigration branch 2 times, most recently from 94eef24 to bb5607a Compare October 12, 2022 13:45
@atzoum atzoum force-pushed the refactor.internalMigration branch from bb5607a to a326aa5 Compare October 13, 2022 05:58
@atzoum
Copy link
Contributor

atzoum commented Oct 13, 2022

Please add a test verifying that incomplete jobs are migrated properly

@Sidddddarth Sidddddarth force-pushed the refactor.internalMigration branch from d95a7d2 to 9f8e250 Compare October 13, 2022 16:14
@Sidddddarth Sidddddarth marked this pull request as ready for review October 14, 2022 00:35
jobsdb/integration_test.go Outdated Show resolved Hide resolved
jobsdb/integration_test.go Outdated Show resolved Hide resolved
@Sidddddarth Sidddddarth force-pushed the refactor.internalMigration branch from 6b464ae to 772d97c Compare October 14, 2022 09:28
Co-authored-by: Aris Tzoumas <atzoumas@rudderstack.com>
@Sidddddarth Sidddddarth force-pushed the refactor.internalMigration branch from 772d97c to beb21b6 Compare October 14, 2022 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants