feat(warehouse): degraded workspace id #2627

lvrach · 2022-10-27T19:21:41Z

Description

Feature

Degradation is a common concept in rudder-stack infrastructure, where a service runs in a limited way allowing some operations to happen safely. In the warehouse domain, degraded mode means:

No processing or database updates are taking place.
Will not accept any new request. Most APIs are not registered.

Workspace degradation is a feature that enables applying the degradation mode for specific workspaceIDs.

Motivation

Migrate to multi-tenant cluster without downtime for customers that are not being migrated but are residents in the source or destination of migration.
Isolate problematic workspaceIDs if the need arises/
Set the foundations for a multi-master warehouse system.

Implementation

A new component is introduced, multitenant.Manager is responsible for:

be the source of truth for degraded workspaceIDs
filter degraded workspaceIDs from backend config
map sourceID to workspaceID

Places that have been affected:

HTTP API handlers will not respond with http.StatusServiceUnavailable, if a workspace id is degraded.
archiver will not archive degraded workspaceID
WatchConfig has replace backendconfig.Subcribe so that workers related to degraded mode ids won't start.

Archive refactoring

To properly test the changes of the archiver:

I have added assertions for minio contents and db states
Added table test for workspaceId functionality
Convert from gingko to go native tests, as I had trouble setting up table tests
Small refactor of the archive, introducing struct and DI (surely further work remains)

limitation and out-of-scope

Degraded workspaces are statically defined. A service restart will be required every time we change them. The manager is design is a way to easily support dynamic workspace ids in the future.
The manager is not properly injected in all the cases. To use proper DI for the manager significant refactoring will be required.
async framework is not in the scope of this PR.

Notion Ticket

https://www.notion.so/rudderstacks/workspace-degraded-mode-5f231fa3899b4a79bf230d2ee12e85ca

Security

The code changed/added as part of this pull request won't create any security issues with how the software is being used.

…t.degraded-workspace-id

codecov · 2022-11-04T01:20:23Z

Codecov Report

Base: 45.60% // Head: 45.44% // Decreases project coverage by -0.15% ⚠️

Coverage data is based on head (f324099) compared to base (0d061ff).
Patch coverage: 47.46% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2627      +/-   ##
==========================================
- Coverage   45.60%   45.44%   -0.16%     
==========================================
  Files         287      289       +2     
  Lines       47790    47928     +138     
==========================================
- Hits        21793    21783      -10     
- Misses      24619    24762     +143     
- Partials     1378     1383       +5

Impacted Files	Coverage Δ
warehouse/archive/cron.go	`0.00% <0.00%> (ø)`
warehouse/identities.go	`1.04% <0.00%> (ø)`
warehouse/warehouse.go	`8.42% <2.59%> (-0.26%)`	⬇️
warehouse/api.go	`70.41% <42.85%> (ø)`
warehouse/archive/archiver.go	`65.63% <50.81%> (ø)`
warehouse/upload.go	`18.45% <58.33%> (+0.09%)`	⬆️
warehouse/multitenant/manager.go	`100.00% <100.00%> (ø)`
warehouse/retry.go	`87.58% <100.00%> (ø)`
warehouse/table_upload.go	`93.92% <100.00%> (ø)`
processor/stash/stash.go	`41.56% <0.00%> (-24.28%)`	⬇️
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

…dder-server into feat.degraded-workspace-id

achettyiitr · 2022-11-04T11:39:23Z

@lvrach
Since we have added an archiver component, Should we consider moving it to a separate package?

lvrach · 2022-11-04T12:26:00Z

@lvrach Since we have added an archiver component, Should we consider moving it to a separate package?

Yeah, I was considering this as well. I will give it a try and see how it goes.

lvrach · 2022-11-04T16:25:39Z

@lvrach Since we have added an archiver component, Should we consider moving it to a separate package?

Yeah, I was considering this as well. I will give it a try and see how it goes.

@achettyiitr archiver has its package now, but it has some changes to upload types to avoid circular imports.

achettyiitr · 2022-11-04T17:05:35Z

warehouse/multitenant/manager.go

+}
+
+// Run is a blocking function that executes manager background logic.
+func (m *Manager) Run(ctx context.Context) error {


Since we are always returning nil. Should we remove the error it?

warehouse/multitenant/manager.go

achettyiitr · 2022-11-04T17:10:27Z

warehouse/multitenant/manager.go

+		}
+
+		m.sourceIDToWorkspaceID = make(map[string]string)
+		m.excludeWorkspaceIDMap = make(map[string]struct{})


We are not using this anywhere?

I believe we are using degradedWorkspaceIDs as excludeWorkspace. Do we still need this?

achettyiitr · 2022-11-04T17:37:03Z

Kudos @lvrach. It was really nice going through this and learning a lot from it.

abhimanyubabbar

This looks really good. 👍🏼

abhimanyubabbar · 2022-11-08T07:05:04Z

warehouse/archive/archiver.go

-func backupRecords(args backupRecordsArgs) (backupLocation string, err error) {
-	pkgLogger.Infof(`Starting backupRecords for uploadId: %s, sourceId: %s, destinationId: %s, tableName: %s,`, args.uploadID, args.sourceID, args.destID, args.tableName)
+type Archiver struct {
+	DB          *sql.DB


I like the idea of having a separate package for archiver and explicitly listing down the dependencies. For having a better DI, should we expose a NewArchiver function so that the system doesn't allowing creating archiver with nil values. Wdyt ?

I prefer this approach for packages that are used as internal components. Let me explain,

If we were to introduce a NewArchive method, is going to need 5 args, and more will be added in the future. In this case, we could use opts fn or a separate struct for the options that still require validation.

In general, is common in Go to initialise struct like this (http.Server, for example).

If we need validation we can do it lazily as part of .Run method for example, but personally, for internal packages, I find it overkill.

abhimanyubabbar · 2022-11-08T07:09:31Z

warehouse/archive/archiver_test.go

@@ -0,0 +1,228 @@
+//go:build !warehouse_integration
+
+package archive_test


Amazing job with the test case ! Moreee tests !! 🥳 🎉

feat(warehouse): added support for bigquery custom partition for workspaceIDs (#2679) chore: by default enable max concurrent gw request limit. (#2648) doc: create SECURITY.md (#2656) chore: use bugsnag in router and processor goroutines (#2686) test: using arm64 compatible images if necessary (#2670) chore: regulation worker avoid panic in case of timeout (#2657) feat(warehouse): degraded workspace id (#2627) Release-As: 1.4.0

exp: PoC

b16559d

github-actions bot added warehouse-team with tests labels Oct 27, 2022

lvrach changed the title ~~exp: PoC~~ feat(warehouse): degraded workspace id Oct 31, 2022

exp: introduce new component for multitenant

a00cdd7

lvrach force-pushed the feat.degraded-workspace-id branch from c015372 to a00cdd7 Compare October 31, 2022 12:24

lvrach added 5 commits November 3, 2022 16:13

Merge branch 'master' of github.com:rudderlabs/rudder-server into fea…

c72c54d

…t.degraded-workspace-id

chore: add tests

9a83299

Merge branch 'master' of github.com:rudderlabs/rudder-server into fea…

08661b4

…t.degraded-workspace-id

fix: broken merge

38a26b1

minor fixes

1d0db4e

lvrach marked this pull request as ready for review November 3, 2022 16:53

lvrach requested review from achettyiitr and abhimanyubabbar November 3, 2022 16:53

lvrach marked this pull request as draft November 3, 2022 17:20

lvrach added 7 commits November 4, 2022 01:58

refactor: improve tests

a975b10

chore: fmt

61ebb88

refactor: no need to override global stats

850a69e

chore: deep source fixes

8e9967f

fix: truncate time to deal with db accuracy issues

7e8d5b2

chore: fmt

abf105a

chore: deep source fixes

a8e1efe

lvrach added 3 commits November 4, 2022 09:55

chore: deep source fixes

1823988

Merge branch 'master' into feat.degraded-workspace-id

902d85d

Merge branch 'feat.degraded-workspace-id' of github.com:rudderlabs/ru…

794c521

…dder-server into feat.degraded-workspace-id

lvrach marked this pull request as ready for review November 4, 2022 07:56

chore: deep source fixes

5e0dd6c

lvrach added 3 commits November 4, 2022 14:33

refactor: add prepare statements

d0c4f17

refactor: move archive to its own package

e0922df

Merge branch 'master' into feat.degraded-workspace-id

769d8ac

fix: testdata

882a0a9

achettyiitr reviewed Nov 4, 2022

View reviewed changes

warehouse/multitenant/manager.go Show resolved Hide resolved

achettyiitr reviewed Nov 4, 2022

View reviewed changes

achettyiitr approved these changes Nov 4, 2022

View reviewed changes

lvrach added 2 commits November 4, 2022 20:28

Merge branch 'master' into feat.degraded-workspace-id

0c4308e

pr fixes

43fe99b

lvrach force-pushed the feat.degraded-workspace-id branch from f6afbc1 to 43fe99b Compare November 4, 2022 18:52

Merge branch 'master' into feat.degraded-workspace-id

8d51558

lvrach requested a review from sivashanmukh November 7, 2022 13:56

lvrach added 2 commits November 7, 2022 19:01

Merge branch 'master' into feat.degraded-workspace-id

e00857b

Merge branch 'master' into feat.degraded-workspace-id

f324099

abhimanyubabbar approved these changes Nov 8, 2022

View reviewed changes

lvrach merged commit a0da8f7 into master Nov 8, 2022

lvrach deleted the feat.degraded-workspace-id branch November 8, 2022 11:18

This was referenced Nov 8, 2022

chore: release 1.3.0 #2562

Closed

chore: release 1.3.0 #2695

Closed

chore: prerelease 1.4.0-rc.1 #2696

Closed

This was referenced Dec 1, 2022

chore: release 1.3.0 #2749

Closed

chore: prerelease 1.4.0-rc.1 #2750

Merged

chore: release 1.4.0 #2752

Merged

This was referenced Dec 1, 2022

chore: prerelease 1.4.0-rc.1 #2757

Closed

chore: prerelease 1.4.0-rc.2 #2759

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(warehouse): degraded workspace id #2627

feat(warehouse): degraded workspace id #2627

lvrach commented Oct 27, 2022 •

edited

Loading

codecov bot commented Nov 4, 2022 •

edited

Loading

achettyiitr commented Nov 4, 2022 •

edited

Loading

lvrach commented Nov 4, 2022

lvrach commented Nov 4, 2022

achettyiitr Nov 4, 2022 •

edited

Loading

achettyiitr Nov 4, 2022

achettyiitr Nov 4, 2022 •

edited

Loading

achettyiitr commented Nov 4, 2022

abhimanyubabbar left a comment

abhimanyubabbar Nov 8, 2022

lvrach Nov 8, 2022

abhimanyubabbar Nov 8, 2022

		@@ -0,0 +1,228 @@
		//go:build !warehouse_integration

		package archive_test

feat(warehouse): degraded workspace id #2627

feat(warehouse): degraded workspace id #2627

Conversation

lvrach commented Oct 27, 2022 • edited Loading

Description

Feature

Motivation

Implementation

Archive refactoring

limitation and out-of-scope

Notion Ticket

Security

codecov bot commented Nov 4, 2022 • edited Loading

Codecov Report

achettyiitr commented Nov 4, 2022 • edited Loading

lvrach commented Nov 4, 2022

lvrach commented Nov 4, 2022

achettyiitr Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

achettyiitr Nov 4, 2022

Choose a reason for hiding this comment

achettyiitr Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

achettyiitr commented Nov 4, 2022

abhimanyubabbar left a comment

Choose a reason for hiding this comment

abhimanyubabbar Nov 8, 2022

Choose a reason for hiding this comment

lvrach Nov 8, 2022

Choose a reason for hiding this comment

abhimanyubabbar Nov 8, 2022

Choose a reason for hiding this comment

lvrach commented Oct 27, 2022 •

edited

Loading

codecov bot commented Nov 4, 2022 •

edited

Loading

achettyiitr commented Nov 4, 2022 •

edited

Loading

achettyiitr Nov 4, 2022 •

edited

Loading

achettyiitr Nov 4, 2022 •

edited

Loading