CBG-4420: handle rev tree in history on processRev #7245
Closed
+13,907
−2,172
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…se on main) * CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code
* CBG-3210: Updating HLV on Put And PutExistingRev (#6366) * CBG-3209: Add cv index and retrieval for revision cache (#6491) * CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code * Add support for PutExistingCurrentVersion * updated to remove function not used anymore * remove duplicated code from dev time * fix linter errors + add assertions on body of doc update * address commnets * updates to add further test cases for AddNewerVersions function + fix some incorrect logic * updates to chnage helper function for creation of doc for tests. Also adress further comments * lint error * address comments, add new merge function for merge versions when hlv is in conflict. * updates to remove test case and test * remove unused function * rebase * missed current version name change * more missing updates to name changes
* CBG-3255: Add current version to log entry for population on the channel cache. Pre-requisite for my work on adding CV to change entries. Only adds CV to log entry from docs seen over DCP at this time pending work on channel cache backfill * add comments and protect against panic in channel cache population * add more commnets * updated to move test and few lines populating log entry
#6590) * CBG-3607: disable the ability to set shared_bucket_access to false. In future we will probably look to fully remove the config param but for now this will protect against panics * remove comment slashes + add test comments * updates off review + failing test in integration test run * missed test update * lint error fix * add skip for lint?
* CBG-3356: add CV to change entry, test that it corretcly populates when calling for changes. Tests need to activate a channel cache as backfill for channel cache not yet implemented * updates to fix failing tests. Added cv to version type returned by Putting a doc and deleting a doc to make testing easier * minor changes * updates after rebase * fix for test failure * updates from rebase * updated comment * changes in response to commmets * updates to fix test failures * rebase + lint skip * updates to update the doc id changes test I have to actually test the codepath
…nts. (#6614) * - Rename `SourceAndVersion` to just `Version`, and rename `SourceAndVersion.Version` to `Value`. - Add/Improve comments. * Update db/hybrid_logical_vector.go
* CBG-3354 Channel query support for current version Adds current version to marshalled _sync.rev property for use with existing indexes. New struct RevAndVersion handles marshal/unmarshal of the rev property, and supports rev only (string) and rev/src/version (map). New structs SyncDataJSON and SyncDataAlias are used to encapsulate this handling at the persistence/marshalling layer. This avoids changes to use of SyncData.CurrentRev, and also avoids potential errors by not duplicating cv in SyncData. * Test updates based on PR feedback
* CBG-3212: add api to fetch a document by its CV value * test fix * rebased SourceAndVersion -> Version rename * Update currentRevChannels on CV revcache load and doc.updateChannels * fix spelling * Remove currentRevChannels * Move common GetRev/GetCV work into documentRevisionForRequest function * Pass revision.RevID into authorizeUserForChannels * Update db/crud.go Co-authored-by: Tor Colvin <tor.colvin@couchbase.com> --------- Co-authored-by: Ben Brooks <ben.brooks@couchbase.com> Co-authored-by: Tor Colvin <tor.colvin@couchbase.com>
* `teh` -> `the` * Remove typo'd TODO by implementing assertion * `comapre` -> `compare` * `exsiting` -> `existing`
* CBG-3210: Updating HLV on Put And PutExistingRev (#6366) * CBG-3209: Add cv index and retrieval for revision cache (#6491) * CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code * CBG-3503 Update HLV on import (#6572) * Beryllium: Rename `SourceAndVersion` to `Version` / Improve HLV comments. (#6614) * - Rename `SourceAndVersion` to just `Version`, and rename `SourceAndVersion.Version` to `Value`. - Add/Improve comments. * Update db/hybrid_logical_vector.go * CBG-3254: pull replication for v4 protocol * updates to the btcRunner * tidy of comments * fix linter * updates to change way hlv is represented on doc revision in rev cache. Also added temporary methods to use db operation in btcRunner test to put and update docs * updates to fix linters and remove unused function * more lint stuff * address commnets after rebase * updates to fix failing test and comments * updates to address comments --------- Co-authored-by: Adam Fraser <adam.fraser@couchbase.com> Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>
* CBG-3213 Version support for channel removals Adds cv (source and version) to removals in _sync.channels (ChannelMap). Uses RevAndVersion to support query (the same approached used for _sync.rev). Required moving RevAndVersion to channels package for usage within ChannelMap. Changes in crud.go required to support the case where the removal version needs to be set via macro expansion. * Use standard function to update testBackingStore document channels
…6655) * CBG-3719: change in memory format of hlv to match XDCR/CBL format * updates after rebase * updates to fix missed type swap in channels package * update to add encoded bucket UUID to db contect, this allows us to avoid overhead associated with encoding bucketUUID each time a HLV is updated * updates after rebase * updates based off review
* CBG-3788 Support HLV operations in BlipTesterClient Switches the BlipTesterCollectionClient to maintain client HLV and (linear) revtree per document. Switches the docs struct to a map of a new BlipTesterDoc struct, instead of a map of revs per document. BlipTesterDoc still maintains a history of all rev messages received (revMessageHistory) to support test evaluation of received messages, but also defines a linear revTreeId history or an HLV (depending on protocol enabled for the test). Includes a refactor of revID to revTreeID in RevAndVersion, as a step toward standardizing ‘revID’ as the generic property used during replication (which can be currentRev or cv), and revTreeID as a traditional revtree revision ID. * Fixes based on PR review
* CBG-3255 Push replication support for HLV Adds push replication support for HLV clients. Delta sync and attachments are not yet supported (pending CBG-3736, CBG-3797). On proposeChanges, checks whether the incoming CV and parent version represent a new document, known version, valid update, or conflict. Uses the same handling as revTreeID (conflict if parent version isn’t the server’s current version), with the additional non-conflict case where the incoming CV and server CV share the same source and the incoming CV is a newer version. For the incoming rev, detects conflict based on the incoming cv (based on the implicit hierarchy in an HLV, where cv > pv > mv). Includes some test helpers to support writing tests with simplified versions (e.g. 1@abc) while still asserting for encoded source and version. * Test fixes * Fixes/cleanup based on PR review --------- Co-authored-by: Gregory Newman-Smith <gregory.newmansmith@couchbase.com>
HLV clients don't consider revpos, and evaluate whether they need to request an attachment based on the existing set of attachments on the document. SGW still needs to persist revpos into _attachments to support revtree clients. For new attachments added by HLV client, revpos is set to the generation of SGW's computed revTreeID for the incoming revision. Co-authored-by: Gregory Newman-Smith <gregory.newmansmith@couchbase.com>
Modifies document marshal and unmarshal to support a set of xattrs (_sync, _vv), and does the same for parsing DCP stream events (including user xattr).
* CBG-4253 create interfaces for integration testing * fixups: - move code to _test.go - force removal of BucketID for couchbase lite peers - remove uneeded SyncGatewayPeerID, now defined at replication time - use a better name to define a database (since peer ids are unique, but test names are too long) - define CouchbaseLitePeerType but the only implementation is a mock peer
Adds collectionID to revCacheValue to support key computation at eviction time (anemone doesn’t include key in revCacheValue as separate keys are needed for CV and revTreeID maps) Also includes test fixes/enhancements for CV handling in memory-bounded rev cache tests.
(Cherry pick to rebased anemone)
- Use unique name for each test single actor test, this used to use the same name for each topology, but not each test. This ensures the documents are not lingering from previous tests. Debugability: - rename rosmar buckets to bucket1,bucket2 - Log full bucket + sourceID with the peer - log replication directions - add prealloc to skipped linter to avoid having to preallocate test cases - add GoString to deep print HLV/Mou
* CBG-4317 uptake fix for TLS without certs for import feed * set auto import which is off by default for CE * Remove root certificates like other credentials on cbgtManager close * reset cert pools only in a test * add missing file
Under some race conditions rosmar XDCR isn't incrementing the 'target newer' stat for the target->source replication, for a mutation that was successfully replicated from source->target. Switch the test to avoid dependency on this stat by switching to docs written instead of docs processed.
* CBG-4265 avoid panic in rosmar xdcr tests - return an error if xdcr is already running when Start is called, or already stopped when Stop is called - allow rosmar xdcr to be restarted via Start/Stop/Start by resetting terminator - don't return empty topology in Topologies which causes test panic - enable rosmar multi actor conflict tests - remove a test that is a duplicate of existing test * lock setting up collections in the case that dcp feed is running when stopping and starting very quickly * switch to require and CollectT * improve debug message * skip test with cbs
* CBG-4389: extract cv from knwon revs and store backup rev by revID * update comments * fix backup revs * further tidy up * udpated to address comments and fix flaking tests * fix incorrect fetch format by cv in getCurrentVersion
* Move tests into separate files to make it easier to enable/disable groups * shorten document names for readability * add missing tests
adamcfraser
reviewed
Dec 16, 2024
torcolvin
force-pushed
the
release/anemone
branch
from
9da680a to
3d0adec
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CBG-4420
Pre-review checklist
fmt.Print,log.Print, ...)base.UD(docID),base.MD(dbName))docs/apiIntegration Tests
GSI=true,xattrs=truehttps://jenkins.sgwdev.com/job/SyncGateway-Integration/000/