Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(framework) Stop launching simulation in thread if asyncio loop detected #3472

Merged
merged 71 commits into from
Jul 11, 2024
Merged
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
57032f2
wip
jafermarq Apr 22, 2024
b885f22
w/ previous
jafermarq Apr 22, 2024
33ada3e
default `create_run`
jafermarq Apr 22, 2024
54539d9
w/ previous
jafermarq Apr 22, 2024
ea946b7
Merge branch 'main' into in-memory-driver
jafermarq Apr 23, 2024
83ff757
Merge branch 'main' into in-memory-driver
jafermarq Apr 24, 2024
13b0e7b
init
jafermarq Apr 25, 2024
242883e
clientdriverproxytests good
jafermarq Apr 25, 2024
6138968
unfusing
jafermarq Apr 25, 2024
0f3ab98
Merge branch 'main' into in-memory-driver
jafermarq Apr 25, 2024
a8ac6ff
Merge branch 'fuse-grpc-driver-and-helper' into in-memory-driver
jafermarq Apr 25, 2024
49ea5df
flower-simulation uses `InMemoryDriver`
jafermarq Apr 25, 2024
8e808f8
Merge branch 'main' into fuse-grpc-driver-and-helper
jafermarq Apr 25, 2024
c2b2f1e
Merge branch 'fuse-grpc-driver-and-helper' into in-memory-driver
jafermarq Apr 25, 2024
46d0268
Merge branch 'main' into in-memory-driver
jafermarq Apr 29, 2024
91443ce
revert and merge w/ main
jafermarq Apr 29, 2024
91a7b31
use InMemoryDriver
jafermarq Apr 29, 2024
e63fd1c
Merge branch 'main' into in-memory-driver
jafermarq Apr 29, 2024
6455809
Merge branch 'main' into in-memory-driver
jafermarq Apr 30, 2024
b2dcc86
updated /w fab
jafermarq Apr 30, 2024
966b55a
wip
jafermarq Apr 30, 2024
2302dc3
completed unittest
jafermarq Apr 30, 2024
944d8f6
Merge branch 'main' into in-memory-driver
jafermarq Apr 30, 2024
a150844
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 1, 2024
68e77c6
Merge branch 'main' into in-memory-driver
jafermarq May 1, 2024
7e211c6
Merge branch 'main' into in-memory-driver
panh99 May 1, 2024
ece1774
ensure tasks are deleted after pull
jafermarq May 1, 2024
7eb41c5
w/ previous
jafermarq May 1, 2024
9e6fc95
fix
jafermarq May 1, 2024
da9ed0b
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 1, 2024
c49df89
Merge branch 'main' into in-memory-driver
jafermarq May 6, 2024
e94874e
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 6, 2024
f8963d3
state-agnostic driver definition
jafermarq May 6, 2024
f4b244c
minor tweak
jafermarq May 6, 2024
5768b08
Merge branch 'main' into in-memory-driver
jafermarq May 6, 2024
96b4c6a
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 6, 2024
b3e4407
Merge branch 'main' into in-memory-driver
jafermarq May 8, 2024
f1a59c7
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 8, 2024
76eab14
Merge branch 'main' into in-memory-driver
jafermarq May 8, 2024
9b1599d
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 8, 2024
a2c2239
Merge branch 'main' into in-memory-driver
jafermarq May 16, 2024
202e2f4
Merge branch 'main' into in-memory-driver
jafermarq May 18, 2024
69349df
Merge branch 'in-memory-driver' into use-inmemory-driver-in-simulations
jafermarq May 18, 2024
b08ba2c
Merge branch 'main' into use-inmemory-driver-in-simulations
danieljanes May 19, 2024
12be6fe
Merge branch 'main' into use-inmemory-driver-in-simulations
jafermarq May 19, 2024
5f803e6
init
jafermarq May 19, 2024
f5d7e87
init
jafermarq May 19, 2024
d6b8cc4
init
jafermarq May 19, 2024
aed45e8
init
jafermarq May 19, 2024
53e11a0
init
jafermarq May 19, 2024
1427958
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 10, 2024
2a04fe7
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jun 10, 2024
5548ce2
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 11, 2024
438154c
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jun 11, 2024
b0a47bd
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jun 14, 2024
aafb61b
Merge branch 'main' into simplify-sim-double-queue
jafermarq Jul 2, 2024
5c0ed99
merge
jafermarq Jul 2, 2024
39f7ba1
fix
jafermarq Jul 2, 2024
723318a
Merge branch 'simplify-sim-double-queue' into simplify-sim-no-asyncio…
jafermarq Jul 2, 2024
cb6c647
remove asyncio cleanup in tests
jafermarq Jul 2, 2024
02a11b9
Merge branch 'main' into simplify-sim-no-asyncio-backends
danieljanes Jul 10, 2024
02c5d1d
refactor(framework) Remove `asyncio` from core Simulation Engine (#3470)
jafermarq Jul 10, 2024
84727f5
Merge branch 'main' into simplify-sim-no-asyncio-backends
danieljanes Jul 10, 2024
cc655d8
Merge branch 'main' into simplify-sim-no-asyncio-backends
jafermarq Jul 10, 2024
f5d2119
py3.8 compatible
jafermarq Jul 10, 2024
d73437f
merge
jafermarq Jul 10, 2024
2404aba
w/ previous
jafermarq Jul 10, 2024
6904497
merge
jafermarq Jul 10, 2024
e75f057
merge
jafermarq Jul 11, 2024
b5cf50b
Merge branch 'simplify-sim-no-asyncio-events' into simplify-sim-alway…
jafermarq Jul 11, 2024
4ce0506
Merge branch 'main' into simplify-sim-always-in-main-thread
danieljanes Jul 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 9 additions & 20 deletions src/py/flwr/simulation/run_simulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,12 +216,7 @@ def _main_loop(
server_app: Optional[ServerApp] = None,
server_app_attr: Optional[str] = None,
) -> None:
"""Launch SuperLink with Simulation Engine, then ServerApp on a separate thread.

Everything runs on the main thread or a separate one, depending on whether the main
thread already contains a running Asyncio event loop. This is the case if running
the Simulation Engine on a Jupyter/Colab notebook.
"""
"""Launch SuperLink with Simulation Engine, then ServerApp on a separate thread."""
# Initialize StateFactory
state_factory = StateFactory(":flwr-in-memory-state:")

Expand Down Expand Up @@ -376,7 +371,6 @@ def _run_simulation(
# Convert config to original JSON-stream format
backend_config_stream = json.dumps(backend_config)

simulation_engine_th = None
args = (
num_supernodes,
backend_name,
Expand All @@ -390,31 +384,26 @@ def _run_simulation(
server_app_attr,
)
# Detect if there is an Asyncio event loop already running.
# If yes, run everything on a separate thread. In environments
# like Jupyter/Colab notebooks, there is an event loop present.
run_in_thread = False
# If yes, disable logger propagation. In environmnets
# like Jupyter/Colab notebooks, it's often better to do this.
asyncio_loop_running = False
try:
_ = (
asyncio.get_running_loop()
) # Raises RuntimeError if no event loop is present
log(DEBUG, "Asyncio event loop already running.")

run_in_thread = True
asyncio_loop_running = True

except RuntimeError:
log(DEBUG, "No asyncio event loop running")
pass

finally:
if run_in_thread:
if asyncio_loop_running:
# Set logger propagation to False to prevent duplicated log output in Colab.
logger = set_logger_propagation(logger, False)
log(DEBUG, "Starting Simulation Engine on a new thread.")
simulation_engine_th = threading.Thread(target=_main_loop, args=args)
simulation_engine_th.start()
simulation_engine_th.join()
else:
log(DEBUG, "Starting Simulation Engine on the main thread.")
_main_loop(*args)

_main_loop(*args)


def _parse_args_run_simulation() -> argparse.ArgumentParser:
Expand Down