Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove unused reference.conf from DistributedData.LightningDb #7294

Merged
merged 4 commits into from
Jul 22, 2024

Conversation

Aaronontheweb
Copy link
Member

Changes

These configuration values are read from and used by DistributedData/reference.conf, so this unused duplicate file is confusing and unnecessary.

These configuration values are read from and used by `DistributedData/reference.conf`, so this unused duplicate file is confusing and unnecessary.
@Aaronontheweb Aaronontheweb added configuration akka-ddata akka-ddata-durable LMDB implementation for persisting durable data labels Jul 22, 2024
@Aaronontheweb Aaronontheweb changed the title remove unused reference.confg from DistributedData.LightningDb remove unused reference.conf from DistributedData.LightningDb Jul 22, 2024
@Aaronontheweb Aaronontheweb enabled auto-merge (squash) July 22, 2024 19:37
@Aaronontheweb
Copy link
Member Author

Akka.Cluster.Sharding.Tests.PersistentShardingMigrationSpec.Migration_should_allow_migration_of_remembered_shards_and_not_allow_going_back failed due to the following error:

Expected subject to be a collection with 3 item(s), but {"1", "3"}"
"contains 1 item(s) less than"
"{"1", "2", "3"}.

On this block of code:

 WithSystem(_configForNewMode, typeName, "NewMode", (system, region, rememberedEntitiesProbe) =>
            {
                AssertRegionRegistrationComplete(region);
                var probe = CreateTestProbe(system);
                region.Tell(new Message(1), probe.Ref);
                probe.ExpectMsg("ack");
                ImmutableHashSet.Create(
                    rememberedEntitiesProbe.ExpectMsg<string>(),
                    rememberedEntitiesProbe.ExpectMsg<string>(),
                    rememberedEntitiesProbe.ExpectMsg<string>()).Should().BeEquivalentTo("1", "2", "3"); // 1-2 from the snapshot, 3 from a replayed message
                rememberedEntitiesProbe.ExpectNoMsg();
            });

The issue occurred because the sharding system had to retry remembering entity 1, which you can see in the logs here:

Migration: Started entity [[akka://NewMode/system/sharding/Migration/1/1#976670556]] with entity id [1] in shard [1]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0017][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/1/RememberEntitiesStarter] Found [1] remembered entities waiting for StartEntityAck, retrying
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0017][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/1] Migration: Request to start entity [1] (Already started)
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0017][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3/RememberEntitiesStore] Recovery completed for shard [3] with [1] entities
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3] Migration: Restarting set of [1] entities
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3] Migration: Shard initialized
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3/RememberEntitiesStarter] Shard starting [1] remembered entities using strategy [all]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration] Migration: Shard was initialized [3]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3] Migration: Request to start entity [3] (in state [RememberedButNotCreated])
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/3] Migration: Started entity [[akka://NewMode/system/sharding/Migration/3/3#1471812385]] with entity id [3] in shard [3]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2/RememberEntitiesStore] Recovery completed for shard [2] with [1] entities
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2] Migration: Restarting set of [1] entities
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2] Migration: Shard initialized
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2/RememberEntitiesStarter] Shard starting [1] remembered entities using strategy [all]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration] Migration: Shard was initialized [2]
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2] Migration: Request to start entity [2] (in state [RememberedButNotCreated])
[NewMode][DEBUG][07/22/2024 19:48:04.674Z][Thread 0018][akka.tcp://NewMode@0.0.0.0:56596/system/sharding/Migration/2] Migration: Started entity [[akka://NewMode/system/sharding/Migration/2/2#336867493]] with entity id [2] in shard [2]

All 3 entities get started, but due to the retry to entity 1 the rememberedEntitiesProbe gets a second message. I'm going to see if I can harden the spec so it can tolerate that in the future.

// due to retries in the remember-entities system, we have to tolerate
// potentially receiving a duplicate message for the same entity
// therefore, we need to wait for at least 3 distinct messages or until the timeout
var maxTimeout = TimeSpan.FromSeconds(5);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardened the spec here - the comment above explains the change.

Copy link
Contributor

@Arkatufus Arkatufus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb Aaronontheweb merged commit a6daed7 into akkadotnet:dev Jul 22, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
akka-ddata akka-ddata-durable LMDB implementation for persisting durable data configuration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants