-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More lock contention in Julia 1.6 to 1.8 than in 1.5.1 (I/O with parallelism strictly not the reason) #39598
Comments
For binarytrees, if you put |
Indeed, if you try something like |
Looks like probably #37816 |
The fix, if it's in, seems only for Distributed, not Threads. Should this hold up a new RC? EDIT: Looking at my timing, I see it's only minor regression for the latter, so no, but for the former, I would really like it confirmed that the fix is in... |
Nothing has been changed with respect to this. Removing the precompile signatures in #37816 was intentional since they caused a larger sysimage and invalidations and it isn't clear that it is worth it. |
Threading is getting noticeably slower on 1.7 (52% slower than 1.5.1; 20% slower than rc2):
This time I'm testing under some load (i.e. Firefox running), but that's also a valuable test.
Maybe, but that's only for the other kind of slowdown (not fixed on 1.7). |
With 4 threads:
I think you're mostly benchmarking I/O? Also, I know you love hyperfine (and it's a great tool to benchmark applications!), but it always includes Julia startup time. |
Yes, I kind of love hyperfine! And it's good to know the real cause, but it's also important to fix. When I looked into IO, Julia seemed to have too much locking (I tried unsuccessfully to lower it), and it's known to be a problem/slower than in other languages, but still it shouldn't be more of a problem in later Julia versions. The number of allocations seems to be exactly the same each time "71.69 k allocations", without threading, and even across versions 1.5.1 and 1.7, but is not with threading on, in older, or here for 1.7:
|
FYI: Actually 1.6.0 is faster up to 2 threads, just 1.5.1 keeps getting faster up to about 7 threads while 1.6 gets slower:
|
Strange Julia error, but first reporting in chronological order the timings: It's not better with 1.8 (and similar on 1.7):
These numbers are with Firefox on, and without I get at least 1 sec better. julia-1.6.0-rc2 may be slightly faster on same conditions (I tried suspending the main firefox process, while "Web Content" processes weren't), and julia-1.5.1 almost certainly still is, getting down to 10 sec. with -O0.
The strange error, first working fine:
Might this be a hint to the possible slowness? I'll wait a bit with killing these processes. The fan is annoying, and I doubt something can be debugged, but let me know then. Could this have anything to do with memory running out? I see in /var/log/syslog from BEFORE those julia processes started (but I' still puzzled by this, I believe I haven't used julia today exert for benchmarking):
Soon after, it's now 14:17, some minutes ago, Mozilla Crash reporter appeared. Firefox often just crashes, with memory running out (OOM in Linux). I don't see that happening in the log for julia or hyperfine (nor firefox which is unusual, I would usually notice, possibly not now since the reporter kicked in), and I stopped hyperfine myself, and julia didn't got killed. Can memory running out still be affecting? Are other processes involved or just the Linux kernel? I get "no"/"same" output for either julia process:
|
Same problem as in previous comment, while now hung earlier:
Just after, I could run with out hyperfine and then:
I doubt this is a hyperfine problem, as then it worked fine. Still I can't rule it out as I've only seen the problem with it. Is it a likely interaction? Since it seems to just run the program in a shell? Actually just trying that seems might be the problem as this just hung:
|
#41411 likely fixes threading hang (untested), but I tested commenting out printing ("write" and "unsafe_write") statements, and fasta.jl went from (min) 2.793 sec to (min) 2.401 sec. or 14% reduction. I took out some wait statements (not all) and time went further down to (min) 2.207 sec, and I still got hangs, but if I take out the last two waits, i.e. the following lines, the time went down to (min) 497.0 ms (82% reduction), but I think the program exited before the threads had completed running the code.
and
At some point after, i.e. I think I had eliminated all waits (but I'm not sure, I was testing back and forth) I did get:
then julia processes were left running and load went up to 30, and I started seening "Measuring shell spawning time" and progress bar for long before "Initial time measurement" and high measurements:
|
@vtjnash I did try to uncomment back the write statements (continuing from previous comment), while keeping the wait statements I had commented out, still commented out, and I predictable get the output in some garbled order, but at least it's fast... so the I/O isn't strictly slow, nor seems the cause of the hang, only the lock contention. And it seems to have eliminated the hang.
|
I just downloaded a nightly and it seems to be getting worse, i.e. "user" time, while "real" maybe slightly better:
|
The "real" time is getting a little better on the latest master, while with --compile=min much better:
|
I think this is in our nanosoldier basebenchmarks, so we will notice any changes. |
I ran with Firefox turned off, and all 16 virtual cores were idle when starting (strangely load still about "1,08"):
https://discourse.julialang.org/t/julia-1-6-0-rc1-and-1-7-with-threads-is-slower-considerations-for-the-benchmark-game/54957
The text was updated successfully, but these errors were encountered: