Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent hangs in CI during parallel run compilation #2844

Merged
merged 10 commits into from
Jun 26, 2023
Merged

Prevent hangs in CI during parallel run compilation #2844

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
93df59f
each pytest has it's own torch extension compilation directory
mrwyattii Feb 16, 2023
e74d157
Merge branch 'master' into seperate-compilation-dirs
mrwyattii Feb 16, 2023
8191a97
remove system package upgrade
mrwyattii Feb 16, 2023
c4818f6
Merge branch 'master' into seperate-compilation-dirs
mrwyattii Feb 16, 2023
e5c9093
Update test_aio.py
mrwyattii Feb 17, 2023
65077e6
move cpu test to self-hosted
mrwyattii Feb 17, 2023
e8887d5
Merge branch 'master' into seperate-compilation-dirs
loadams Feb 17, 2023
cc1b6c3
Merge branch 'master' into seperate-compilation-dirs
tjruwase Feb 21, 2023
17baa5d
Merge branch 'master' into seperate-compilation-dirs
tjruwase Mar 1, 2023
8f04321
Merge branch 'master' into seperate-compilation-dirs
loadams Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/nv-torch-latest-cpu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ concurrency:

jobs:
unit-tests:
runs-on: ubuntu-20.04
runs-on: [self-hosted, cpu]

steps:
- uses: actions/checkout@v3
Expand Down
8 changes: 8 additions & 0 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,14 @@ def validate_version(expected, found):
return found == expected


# This fixture prevents hangs when 2+ pytest processes try to compile the same
# code at once and causes deadlocks / hangs
@pytest.fixture(scope="session", autouse=True)
def set_torch_ext_dir(worker_id):
torch_ext_dir = os.environ["TORCH_EXTENSIONS_DIR"]
os.environ["TORCH_EXTENSIONS_DIR"] = os.path.join(torch_ext_dir, worker_id)


@pytest.fixture(scope="session", autouse=True)
def check_environment(pytestconfig):
expected_torch_version = pytestconfig.getoption("torch_ver")
Expand Down