Parallel Target JIT?

[BlythMeister]

I have the following targets as an example

Target "T1" (fun _ ->
    printfn "T1"
    System.Threading.Thread.Sleep(1000)
)

Target "T2.1" (fun _ ->
    printfn "T2.1"
    System.Threading.Thread.Sleep(1000)
)

Target "T2.2" (fun _ ->
    printfn "T2.2"
    System.Threading.Thread.Sleep(1000)
)

Target "T2.3" (fun _ ->
    printfn "T2.3"
    System.Threading.Thread.Sleep(1000)
)

Target "T3" (fun _ ->
    printfn "T3"
    System.Threading.Thread.Sleep(3000)
)

Target "T4" (fun _ ->
    printfn "T4"
    System.Threading.Thread.Sleep(1000)
)

"T1"
    ==> "T2.1"
    ==> "T2.2"
    ==> "T2.3"

"T1"
    ==> "T3"

"T2.3"
    ==> "T4"

"T3"
    ==> "T4"

T1 is my base target,
Off of that the T2 chain consists of 3 steps taking 1 second each
The T3 target is a single and takes 3 seconds
T4 will run when the T2 chain and T3 target have completed.

I would expect the total exectution to be 5 seconds:
T1 - 1 second
T3 & T2 Chain - 3 second
T4 - 1 second

However, T3 runs alongside T2.3.
As a result, the total run time is 7 seconds.

Should T3 not start as soon as it is possible to do so, rather than waiting?

The build output is as follows:

Building project with version: LocalBuild
Running parallel build with 20 workers
Starting Target: T1
T1
Finished Target: T1
Starting Target: T2.1 (==> T1)
T2.1
Finished Target: T2.1
Starting Target: T2.2 (==> T2.1)
T2.2
Finished Target: T2.2
Starting Target: T3 (==> T1)
Starting Target: T2.3 (==> T2.2)
T3
T2.3
Finished Target: T2.3
Finished Target: T3
Starting Target: T4 (==> T2.3, T3)
T4
Finished Target: T4

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target     Duration
------     --------
T1         00:00:01.0015316
T2.1       00:00:01.0009064
T2.2       00:00:01.0009421
T2.3       00:00:01.0007335
T3         00:00:03.0011838
T4         00:00:01.0007754
Total:     00:00:07.0572197
Status:    Ok
---------------------------------------------------------------------

[BlythMeister]

If i update the dependency tree using the <=> syntax as follows:

"T1"
    ==> "T2.1" <=> "T3"
    ==> "T2.2"
    ==> "T2.3"

"T1"
    ==> "T3"

"T2.3"
    ==> "T4"

"T3"
    ==> "T4"

T3 will run earlier, but then T2.2 waits for T3 to finish, which is also undesired as the overall time to complete is still 7 seconds

Building project with version: LocalBuild
Running parallel build with 20 workers
Starting Target: T1
T1
Finished Target: T1
Starting Target: T2.1 (==> T1)
T2.1
Starting Target: T3 (==> T1, T1)
T3
Finished Target: T2.1
Finished Target: T3
Starting Target: T2.2 (==> T2.1, T3)
T2.2
Finished Target: T2.2
Starting Target: T2.3 (==> T2.2)
T2.3
Finished Target: T2.3
Starting Target: T4 (==> T2.3, T2.1, T3)
T4
Finished Target: T4

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target     Duration
------     --------
T1         00:00:01.0009762
T2.1       00:00:01.0013728
T3         00:00:03.0007401
T2.2       00:00:01.0010601
T2.3       00:00:01.0005961
T4         00:00:01.0008781
Total:     00:00:07.0471391
Status:    Ok
---------------------------------------------------------------------

[forki]

I assume you need to use more <=> arrows.

Am 04.10.2016 5:55 nachm. schrieb "Chris Blyth" notifications@github.com:

If i update the dependency tree using the <=> syntax as follows:

"T1"
==> "T2.1" <=> "T3"
==> "T2.2"
==> "T2.3"

"T1"
==> "T3"

"T2.3"
==> "T4"

"T3"
==> "T4"

T3 will run earlier, but then T2.2 waits for T3 to finish, which is also
undesired as the overall time to complete is still 7 seconds

Building project with version: LocalBuild
Running parallel build with 20 workers
Starting Target: T1
T1
Finished Target: T1
Starting Target: T2.1 (==> T1)
T2.1
Starting Target: T3 (==> T1, T1)
T3
Finished Target: T2.1
Finished Target: T3
Starting Target: T2.2 (==> T2.1, T3)
T2.2
Finished Target: T2.2
Starting Target: T2.3 (==> T2.2)
T2.3
Finished Target: T2.3
Starting Target: T4 (==> T2.3, T2.1, T3)
T4
Finished Target: T4

---

Build Time Report

Target Duration

---

T1 00:00:01.0009762
T2.1 00:00:01.0013728
T3 00:00:03.0007401
T2.2 00:00:01.0010601
T2.3 00:00:01.0005961
T4 00:00:01.0008781
Total: 00:00:07.0471391

Status: Ok

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1395 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADgNHkX88ovEnLfMGMcVpUWNvkKlEg6ks5qwnbUgaJpZM4KN15Q
.

[BlythMeister]

If I put the arrows on 2.1 to 2.3 I get a stack overflow.
Where do you suggest then go?

To me, it seems fake will sort of lazy evaluate the parallel tree and run it at the last possible minute.
Whereas I would like it to run the targets eagerly, as if there was a dependency, it would be specified.

[BlythMeister]

I'm guessing it has everything to do with the ordering here
https://github.com/fsharp/FAKE/blob/122939490cad5338d448829f93699a005c8fdb8e/src/app/FakeLib/TargetHelper.fs#L471

Since this orders by level depending, it will run all the level 3s together, then the level 2s, then the level 1s.
But that seems a little odd to me, since it would be more optimal to run the targets as early as possible, as it's impossible to know how long they take.

[forki]

I'm not on pc right now, but IIRC only things that are bound by <=> are run in parallel. ==> is a sequential operator.

[BlythMeister]

Yeah, I basically defined 2 sequential operations that have a common start and end.
Using something similar to the docs.

Whilst the steps do run in parallel, the tree with 1 step starts when the last step of the tree with 3 starts.
Therefore making it not optimal.

If it were to start when 1st step runs it would be more efficient.

I can get parallel targets running without the use of the <=> operator.

[BlythMeister]

To be clearer, the issue is not about parallel targets running/not running, that works fine.

Its more when FAKE starts running the parallel target.

[forki]

Mhm I'm open to improvements in the target runner. But unfortunately currently I don't have time for that,but I definitely would appreciate a pr.

[BlythMeister]

I am happy to look into it (can't promise I'll get it working)

As long as you think my thought pattern on running parallel steps as early as possible rather than as late is a good idea.

Dont want to spend hours getting it to work to have the project rejected. ☺️

[forki]

Tbh I never really cared about parallel targets since most of the time my IO wasn't really meant for that.

[pr-yemibedu]

Hello,
Please change your build.cmd from:
packages\FAKE\tools\FAKE.exe build.fsx %*
to
packages\FAKE\tools\FAKE.exe build.fsx %* "parallel-jobs=2"
I have setup up your targets as such:
Target "All" (fun () -> trace "all") // to unify the end
"T1" ==> "T4"
"T2.1" <=> "T3" <=> "T4"
"T2.1" ==> "T2.2" ==> "T2.3"
"T4" ==> "All"
"T2.3" ==> "All"
You do not need to be redundant with the T1 dependency. <=> means the all have a common parent. So you only need to define the unique paths to follow. Without the parallel takes 8 seconds and with the parallel takes 6 seconds. Thank you. Good day.

[BlythMeister]

Right, I'll give that a go
I think that some of your dependencies differ to mine in that targets dependency.
It should go
T1 then T2.x & T3 then T4

And also don't think I understood the <=> operator

[pr-yemibedu]

Hello,
Recreating your logic. Still runs in 6 seconds. Still using parallel jobs.

"T1" ==> "T3"
"T2.1" <=> "T3"
"T2.1" ==> "T2.2" ==> "T2.3"
"T2.3" ==> "T4"
"T3" ==> "T4"
"T4" ==> "All"

You have a minimum path of 1 plus 3 plus 1. The more threads the better your outcome. This only allowed 2 even though I specified 8.

For <=> see its comment and definition:
https://github.com/fsharp/FAKE/blob/master/src/app/FakeLib/AdditionalSyntax.fs#L83-83

Thank you. Good day.

[BlythMeister]

Ok I'll try and replicate this on my real world scenario.

Should be 5 seconds though surely.
1+3+1=5.

[BlythMeister]

@pr-yemibedu this may run in 6 seconds, but the dependencies are all wrong

Building project with version: LocalBuild
Running parallel build with 4 workers
Starting Target: T2.1
T2.1
Finished Target: T2.1
Starting Target: T1
Starting Target: T2.2 (==> T2.1)
T1
T2.2
Finished Target: T2.2
Finished Target: T1
Starting Target: T2.3 (==> T2.2)
Starting Target: T3 (==> T1)
T2.3
T3
Finished Target: T2.3
Finished Target: T3
Starting Target: T4 (==> T2.3, T2.1, T3)
T4
Finished Target: T4

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target     Duration
------     --------
T2.1       00:00:01.0007302
T1         00:00:01.0017169
T2.2       00:00:01.0009312
T2.3       00:00:01.0005985
T3         00:00:03.0010788
T4         00:00:01.0008892
Total:     00:00:06.0573842
Status:    Ok
---------------------------------------------------------------------

As you can see, T2.1 starts before T1

[BlythMeister]

"You do not need to be redundant with the T1 dependency. <=> means the all have a common parent."

i also believe to not be accurate statement as:

"T1" ==> "T3"
"T1" ==> "T2.1"
"T2.1" <=> "T3"

will force T2.1 to wait for T1 but

"T1" ==> "T3"
"T2.1" <=> "T3"

will allow T2.1 to run when T1 has not run

[BlythMeister]

@forki @pr-yemibedu 1st step, create a failing test...BlythMeister@5a9a2f4

That might explain what i am meaning better.

[pr-yemibedu]

Hello,
What version oFare you running? I am using FAKE (4.40.1). I get 7 seconds using your logic. Even though T2.1 and T3 start together, the T2.1 dependencies wait for T2.1 and T3 to finish. I understand both what you want and what is actually happening. So it is back to the drawing board. Thank you. Good day.

[BlythMeister]

i am using the latest 4.40.1.

I also understand why it is happening due to looking at the code and am working on a way to fix.

[BlythMeister]

Having got the order working, i realised that all parallel targets work a level at a time.
Basically, what i am trying to do with different length trees fundamentally won't work :(

@forki did you want a PR for the re-ordering so they run eagarly, or not as it actually makes no difference to the overall execution

[pr-yemibedu]

Hello,

I posted in chat I was working on making a runTargetAsync library. Currently here is what it can achieve:

Starting Target: T1
T1
Finished Target: T1
Starting Target: T3 (==> T1)
Starting Target: T2.1 (==> T1)
T3
T2.1
Finished Target: T2.1
Starting Target: T2.2 (==> T2.1)
T2.2
Finished Target: T2.2
Starting Target: T2.3 (==> T2.2)
T2.3
Finished Target: T3
Finished Target: T2.3
Starting Target: T4 (==> T2.3, T3)
T4
Finished Target: T4

---

Build Time Report

---

Target Duration

---

T1 00:00:01.0010002
T2.1 00:00:01.0018909
T2.2 00:00:01.0014843
T3 00:00:03.0019320
T2.3 00:00:01.0015257
T4 00:00:01.0016476
Total: 00:00:05.0506471
Status: Ok

It is good enough for me but I have to work on implementing soft dependencies and other test scenarios. The lib is easier than branching for now. Let me know if are things I should include as I get this working. Thank you. Good day.

[BlythMeister]

Ok yeah that looks awesome!
I've got some other tweaks on my PR to show the dependency tree etc on parallel.

But I would love to see this in! It really solves my problem.

[yemibedu]

This library is working for now: Target Concurrency

[BlythMeister]

Could this not be integrated direct into fake as the way all targets are run?

[BlythMeister]

This has been implemented now