[fix] delayed_sidekiq race condition

[skcc321]

In the PR I'm addressing two issues with delayed_sidekiq strategy.

1. There is a chance for race conditions exactly in this place.

            unless redis.zrank(timechunks_key, timechunk_key) # read
              redis.zadd(timechunks_key, at, timechunk_key) # write

I eliminated it using Lua script and making the combination of operations atomic.

2. Also, there was a possibility to bloat Redis if there is an issue with reindexing (ES connection timeout for instance)
   Before submitting the PR make sure the following are checked:

• The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
• Wrote good commit messages.
• Commit message starts with [Fix #issue-number] (if the related issue exists).
• Feature branch is up-to-date with master (if not - rebase it).
• Squashed related commits together.
• Added tests.
• Added an entry to the changelog if the new code introduces user-observable changes. See changelog entry format for details.

[konalegi]

:wow: great!

[konalegi]

please ping me once you are ready to merge. I'll release it

[konalegi]

Please fix rubocop issues, and something is odd with our specs and CI in general :/

[konalegi]

Please take a look at your PR to see what's wrong. I created a separate PR to check CI and it seems to be working. #942

[ebeagusamuel]

@skcc321 Please try rebasing on toptal/chewy master branch.

[skcc321]

Sorry for the delay, guys.
Yep, I rebased it on upstream/master today.
So, we've been testing it on prod 2 weeks. Looks like it completely resolved the race condition issue.
@konalegi, @ebeagusamuel should be ready to merge now. Could you guys restart the CI to verify the PR?

[konalegi]

Do we need mock_redis gem after that?

[konalegi]

Since you've started using real redis in specs, you have to redis service, smth like and provide proper env variables for sidekiq/redis depending how you want to test it.

jobs:
  ruby-3:
    services:    
      redis:
        # Docker Hub image
        image: redis
        # Set health checks to wait until redis has started
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

[skcc321]

Since you've started using real redis in specs, you have to redis service, smth like and provide proper env variables for sidekiq/redis depending how you want to test it.

jobs:
  ruby-3:
    services:    
      redis:
        # Docker Hub image
        image: redis
        # Set health checks to wait until redis has started
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

done. let's try again the CI run.

[konalegi]

@skcc321 still failing :/

[skcc321]

@skcc321 still failing :/

OK, I'll take a look tonight.

[skcc321]

@konalegi one more attempt please

[skcc321]

@konalegi looks like two failed checks are not related to the changes I made.
could you restart them?

[skcc321]

@konalegi one of them has passed. Could you restart the last one?

[konalegi]

Great, merging!