Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempting to retrieve FetchedJobs crashes due to duplicates #370

Open
Jameak opened this issue Jun 14, 2024 · 0 comments
Open

Attempting to retrieve FetchedJobs crashes due to duplicates #370

Jameak opened this issue Jun 14, 2024 · 0 comments

Comments

@Jameak
Copy link

Jameak commented Jun 14, 2024

We're getting an exception when attempting to retrieve the list of fetched jobs via the Monitoring API.

This line of code:
JobStorage.Current.GetMonitoringApi().FetchedJobs("default", 0, int.MaxValue)

Causes the following exception:

System.ArgumentException: An item with the same key has already been added. Key: 24014
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(IEnumerable`1 jobIds)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(String queue, Int32 from, Int32 perPage)
   at Somewhere.In.My.Code

The issue seems to occur in the following situation. This is purely based on logs from our environments, I have not had the time to attempt to reproduce it locally:

  1. A hangfire server is currently processing jobs.
  2. The hangfire server disappears from the active server list (such as when the server gets restarted or the application crashes)
  3. The other hangfire servers (or the 'new' server if it was restarted) notices that the jobs from step 1 might have been aborted, because they're marked as being processed by a server that is not in the active server list.

If you attempt to get the list of fetched jobs after step 3 has occurred, it seems to (sometimes?) crash. This eventually resolves itself, presumably when the job gets successfully processed.

Versions:
Hangfire.PostgreSql: 1.20.8
Hangfire: 1.8.7
.NET version 8.0.202

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant