[Bug]: Direct Runner sometimes rises RuntimeError: dictionary changed size during iteration

[rafalpotempa]

What happened?

Apache Beam: 2.46.0
Direct Runner

We are running some parts of our pipelines using Direct Runners, since we are processing small batches of data there. The rest is using Dataflow runner, which has some auto-recovery. For direct runners it fails the run and triggers alerts for SRE team.

I couldn't reproduce the issue myself, the issue is non-deterministic.
The part of our company's proprietary system sometimes (not very often ~100 runs) fails with:

apache_beam/runners/worker/sdk_worker.py in shutdown_inactive_bundle_processors at line 585

Exception in thread Thread-37:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 72, in run
    reraise(*_capture_exception())
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/threading.py", line 70, in run
    return old_run_func(self, *a, **kw)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/data_plane.py", line 228, in run
    self._function(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/runners/worker/sdk_worker.py", line 585, in shutdown_inactive_bundle_processors
    for descriptor_id, last_access_time in self.last_access_times.items():
RuntimeError: dictionary changed size during iteration

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

we should be able to fix this by using a lock in shutdown_inactive_bundle_processors, same as

beam/sdks/python/apache_beam/runners/worker/sdk_worker.py

Line 470 in 41e6628

with self._lock:

.

[anneyshas]

Hi! It's my first time contributing and I would love to give this a shot!

[anneyshas]

.take-issue

[tvalentyn]

great, thanks a lot, @Anneysha7

[tvalentyn]

Hey @Anneysha7 , just wondering if you are still interested or need help. Thanks!

[anneyshas]

Hi, sorry for the late reply, still interested! I'll come back to you asap if I need help.

…

On Tue, 8 Aug 2023, 12:20 tvalentyn, ***@***.***> wrote: Hey @Anneysha7 <https://github.com/Anneysha7> , just wondering if you are still interested or need help. Thanks! — Reply to this email directly, view it on GitHub <#27501 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWUSWJCTS3535W7GYL2QPLXUGO73ANCNFSM6AAAAAA2J7D2LU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tvalentyn]

Hi @Anneysha7 i'd like to have this issue fixed in next version of Beam esp since it should be a simple fix. I suggest we unassign it for now in case someone else has time to take a look - no worries you can always pick this or any other issue when you have time to work on Beam.

[Mr-Magnificent]

Hey, would like to give this a go. Might take some time since it would be my first time contributing to apache beam. I'll assign it to myself, feel free to reassign if the criticality requires faster fix @tvalentyn. Thanks

[Mr-Magnificent]

.take-issue

[Mr-Magnificent]

Realised I am having bandwidth issues, incase issue still open post couple of week, I'd like to give it a try.

[lrakla]

.take-issue

[lrakla]

Hi, I have set up my dev environment and will give it a try. It is my first time contributing too, so I am grateful for any guidance. For now, I'll add a lock in the required section

[tvalentyn]

@lrakla Sounds like a plan, thanks for your help. We also have a dev wiki at https://s.apache.org/beam-python-dev-wiki, which you might find helpful.