Uvicorn with reload hangs when using a ProcessPoolExecutor

[lukin0110]

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

When at last 1 task is submitted to a ProcessPoolExecutor uvicorn fails to reload when a file change has been detected. It detects the file change and the server is shutdown but it doesn't start again. As long as no tasks are submitted uvicorn is able to reload properly.

To reproduce

"""ProcessPoolExecutor Example.

Run
---
uvicorn main:app --reload

Versions
--------
fastapi~=0.63.0
uvicorn[standard]~=0.13.3
"""
from concurrent.futures import ProcessPoolExecutor
from typing import Any, Dict

from fastapi import FastAPI

app = FastAPI(title="Example API")
POOL = ProcessPoolExecutor(max_workers=1)


def task() -> None:
    """."""
    print("Executed in process pool")


@app.get("/")
def index() -> Dict[str, Any]:
    """Index."""
    POOL.submit(task)
    return {"message": "Hello World"}

Expected behavior

Uvicorn should reload when file changes are detected.

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [9042] using watchgod
INFO:     Started server process [9044]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
WARNING:  WatchGodReload detected file change in '['/Users/maartenhuijsmans/main.py']'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [9044]
INFO:     Started server process [9047]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:61607 - "GET / HTTP/1.1" 200 OK

Actual behavior

Uvicorn doesn't start

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [9054] using watchgod
INFO:     Started server process [9056]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:61615 - "GET / HTTP/1.1" 200 OK
Executed in process pool
WARNING:  WatchGodReload detected file change in '['/Users/maartenhuijsmans/main.py']'. Reloading...
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [9056]

Debugging material

Environment

• macOS 10.13.6 / python 3.8.6 / uvicorn 0.13.3
• uvicorn main:app --reload

Additional context

[euri10]

incidentally it seems to not happen on #853

[lukin0110]

@euri10 thanks. I'll be patient and wait for the MR

[kellen]

Similar issue, probably same root cause: when using a lock to mutate shared state, if the lock is acquired when a reload starts, the new process can never be joined, so the reload hangs. @euri10 is #853 stalled?

import threading
lock = threading.Lock()

@app.get("/")
def index() -> Dict[str, Any]:
    with lock:
        time.sleep(5.0)
        # any reload here will hang
    return {"message": "Hello World"}

Edit: hmm, appears I should have been using multiprocessing.Lock, which doesn't seem to exhibit this behavior. Now deadlocks 🙃

[euri10]

definitely stalled yes
you could check with #1069 maybe to see if that solves your thing, would be a good test, let us know thanks

[Kludex]

This is not a bug. It works as intended.

On the BaseReload.restart() method, we terminate() the server process, and then we join(). As the process still have the ProcessPool alive, it will never terminate.

That said, we have two options:

1. On Uvicorn's side, as join() accepts a timeout parameter, we can set it to a value that makes sense (?).
2. Consider this as a user issue, which the solution here would be to shut down the pool on the shutdown event.

On 2 you'd have:

from concurrent.futures import ProcessPoolExecutor
from typing import Any, Dict

from fastapi import FastAPI

app = FastAPI(title="Example API")
POOL = None

@app.on_event("startup")
def startup():
    global POOL
    POOL = ProcessPoolExecutor(max_workers=1)


@app.on_event("shutdown")
def shutdown():
    global POOL
    POOL.shutdown()

def task() -> None:
    """."""
    print("Executed in process pool")


@app.get("/")
def index() -> Dict[str, Any]:
    """Index."""
    POOL.submit(task)
    return {"message": "Hello World"}

Given that the snippet above already satisfies this issue, and that I've provided explanation about what's happening, I'll be closing this issue. If there's a good argument, and a proposal to implement 1 in uvicorn, we can reconsider it.