Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set OPENBLAS_NUM_THREADS=1 on local Distributed workers #47803

Merged
merged 1 commit into from
Dec 5, 2022
Merged

Set OPENBLAS_NUM_THREADS=1 on local Distributed workers #47803

merged 1 commit into from
Dec 5, 2022

Conversation

staticfloat
Copy link
Member

This should prevent LinearAlgebra from trying to increase our OpenBLAS thread count in its __init__() method when we're not trying to enable threaded BLAS.

DilumAluthge
DilumAluthge approved these changes Dec 5, 2022
View reviewed changes
Copy link
Member

@DilumAluthge DilumAluthge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also check if the environment variable is already defined by the user, and if it is, we don't override it?

@DilumAluthge
Copy link
Member

Can we backport to 1.8 and/or 1.9?

@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers
29faa50 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.
@staticfloat staticfloat force-pushed the sf/windows_openblas_thread branch from 471bbc0 to 29faa50 Compare December 5, 2022 18:08
@DilumAluthge
Copy link
Member

For context, we hope that this will fix the Windows OOMs that we are seeing on CI.

@giordano giordano added the ci Continuous integration label Dec 5, 2022
@staticfloat staticfloat added backport 1.6 Change should be backported to release-1.6 backport 1.8 Change should be backported to release-1.8 backport 1.9 Change should be backported to release-1.9 labels Dec 5, 2022
@PallHaraldsson
Copy link
Contributor

Does this also work for Pkg.add? I'm not sure how it forks Julia processes, or if this trick is already used there. If it uses Distributed, then this would automatically cover it?

I tried locking where add is implemented, I think at least this file is related:
https://github.com/JuliaLang/Pkg.jl/blob/d5ac7caf57c603dde7eb63f74549898a4618bc5e/src/API.jl#L141

@staticfloat
Copy link
Member Author

Does this also work for Pkg.add?

Do you mean for the precompilation step that runs at the end of Pkg.add()? That works by invoking Base.compilecache(), which eventually ends up here, and that already has OPENBLAS_NUM_THREADS set to 1, so it doesn't need this change.

@staticfloat
Copy link
Member Author

I can confirm that this reduces the commit charge significantly! Compare the commit values versus the working set values:

image

Compare against an equivalent run on master at about the same point in the test suite:

image

Let's let this ride for a bit on master, and if this cleans up the OOM errors, we may even be able to reduce the Windows VM memory usage a bit to 3GB/core, instead of the currently extremely wasteful 4GB/core.

@staticfloat staticfloat merged commit a8b3994 into master Dec 5, 2022
@staticfloat staticfloat deleted the sf/windows_openblas_thread branch December 5, 2022 19:40
staticfloat added a commit to JuliaCI/sandboxed-buildkite-agent that referenced this pull request Dec 5, 2022
@staticfloat
Reduce Windows VM memory allocation to 3GB/core
ae0160f 
Now that JuliaLang/julia#47803 has been merged, we are using [significantly less commit charge](JuliaLang/julia#47803 (comment)), and should be capable of running with only 3GB/core allocated, rather than the currently rather wasteful 4GB/core.
@staticfloat staticfloat mentioned this pull request Dec 5, 2022
Merged
@KristofferC
Copy link
Member

Maybe fixes #45434

This was referenced Dec 5, 2022
Closed
Closed
KristofferC pushed a commit that referenced this pull request Dec 8, 2022
@staticfloat @KristofferC
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
dba443d 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
This was referenced Dec 8, 2022
Merged
Merged
KristofferC pushed a commit that referenced this pull request Dec 14, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
8f114fc 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
KristofferC pushed a commit that referenced this pull request Dec 14, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
dd0e839 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
KristofferC pushed a commit that referenced this pull request Dec 14, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
9bcb6c6 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
KristofferC pushed a commit that referenced this pull request Dec 14, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
8830c26 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
@KristofferC KristofferC removed the backport 1.8 Change should be backported to release-1.8 label Dec 16, 2022
@KristofferC KristofferC mentioned this pull request Dec 21, 2022
Closed
78 tasks
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
8069c38 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
22489a3 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
42c3fe8 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
7a4c4b0 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
2ce254a 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
KristofferC pushed a commit that referenced this pull request Dec 21, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
7479889 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
staticfloat added a commit that referenced this pull request Dec 23, 2022
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
2bfab32 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
@KristofferC KristofferC removed the backport 1.9 Change should be backported to release-1.9 label Dec 27, 2022
staticfloat added a commit to JuliaCI/sandboxed-buildkite-agent that referenced this pull request Feb 14, 2023
@staticfloat
Reduce Windows VM memory allocation to 3GB/core (#31)
597f19d 
Now that JuliaLang/julia#47803 has been merged, we are using [significantly less commit charge](JuliaLang/julia#47803 (comment)), and should be capable of running with only 3GB/core allocated, rather than the currently rather wasteful 4GB/core.
@Pangoraw Pangoraw mentioned this pull request Sep 13, 2023
Merged
@fonsp fonsp mentioned this pull request Sep 18, 2023
Merged
vchuravy pushed a commit to JuliaLang/Distributed.jl that referenced this pull request Oct 6, 2023
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (JuliaLang/…
a89ae75 
…julia#47803)

This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit f77b8a6)
(cherry picked from commit 72cb2fd)
KristofferC pushed a commit that referenced this pull request Oct 10, 2023
@staticfloat @KristofferC
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
b901d10 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.

(cherry picked from commit a8b3994)
(cherry picked from commit 8830c26)
Keno pushed a commit that referenced this pull request Jun 5, 2024
@staticfloat
Set OPENBLAS_NUM_THREADS=1 on local Distributed workers (#47803)
f77b8a6 
This should prevent LinearAlgebra from trying to increase our OpenBLAS
thread count in its `__init__()` method when we're not trying to enable
threaded BLAS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DilumAluthge DilumAluthge DilumAluthge approved these changes

@vchuravy vchuravy vchuravy approved these changes

Assignees
No one assigned
Labels
backport 1.6 Change should be backported to release-1.6 ci Continuous integration
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@staticfloat @DilumAluthge @PallHaraldsson @KristofferC @vchuravy @giordano