Akka.Cluster.Sharding: LeastShardAllocationStrategy ArgumentException during rebalance

[Aaronontheweb]

Version Information
Version of Akka.NET? not sure yet - waiting on reply in Discord
Which Akka.NET Modules? Akka.Cluster.Sharding

Describe the bug

System.ArgumentException: An element with the same key but a different value already exists. Key: 'akka.tcp://NuBreedSys@prod-nubreed-quote-process-managers-0.prod-nubreed-quote-process-managers-service:9233'
   at System.Collections.Immutable.ImmutableDictionary2.HashBucket.Add(TKey key, TValue value, IEqualityComparer1 keyOnlyComparer, IEqualityComparer1 valueComparer, KeyCollisionBehavior behavior, OperationResult& result)
   at System.Collections.Immutable.ImmutableDictionary2.AddRange(IEnumerable1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
   at Akka.Cluster.Sharding.Internal.AbstractLeastShardAllocationStrategy.RegionEntriesFor(IImmutableDictionary2 currentShardAllocations)
   at Akka.Cluster.Sharding.LeastShardAllocationStrategy.Rebalance(IImmutableDictionary2 currentShardAllocations, IImmutableSet1 rebalanceInProgress)
   at Akka.Cluster.Sharding.ShardCoordinator.Active(Object message)
   at Akka.Cluster.Sharding.DDataShardCoordinator.<Activate>b__45_0(Object msg)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Phobos.Actor.PhobosActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

Expected behavior

Actual behavior
What actually happened and how did it differ from your expectations?

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

Using state-store-mode = ddata

Additional context

Not sure yet, still working on getting thoughts organized here

[Aaronontheweb]

This is the line that throws:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/ShardAllocationStrategy.cs

Line 169 in 5e2bd0e

var sortedRegionEntries = RegionEntriesFor(currentShardAllocations).OrderBy(i => i, ShardSuitabilityOrdering.Instance).ToImmutableList();

When it's invoking this function:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/Internal/AbstractLeastShardAllocationStrategy.cs

Lines 127 to 142 in 5e2bd0e

	protected ImmutableList<RegionEntry> RegionEntriesFor(IImmutableDictionary<IActorRef, IImmutableList<string>> currentShardAllocations)
	{
	var addressToMember = ClusterState.Members.ToImmutableDictionary(m => m.Address, m => m);
	return currentShardAllocations.Select(i =>
	{
	var regionAddress = i.Key.Path.Address.HasLocalScope ? SelfMember.Address : i.Key.Path.Address;
	var memberForRegion = addressToMember.GetValueOrDefault(regionAddress);
	// if the member is unknown (very unlikely but not impossible) because of view not updated yet
	// that node is ignored for this invocation
	if (memberForRegion != null)
	return new RegionEntry(i.Key, memberForRegion, i.Value);
	return null;
	}).Where(i => i != null).ToImmutableList();
	}

I'm guessing it's the ToImmutableDictionary call that is the problem here based on the stack trace provided by the user

[Aaronontheweb]

Also need to investigate why Akka.Cluster.Sharding never automatically healed from this issue either - that seems like a separate bug