[general] Close workers that do not have jobs

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

One aspect about targets that puzzles me is that I just have

31 jobs left

Submitted batch job 19178162
Submitted batch job 19178163
Submitted batch job 19178164
Submitted batch job 19178165
Submitted batch job 19178166
Submitted batch job 19178167
Submitted batch job 19178168
Submitted batch job 19178169
Submitted batch job 19178170
Submitted batch job 19178171
Submitted batch job 19178172
Submitted batch job 19178173
Submitted batch job 19178174
Submitted batch job 19178175
Submitted batch job 19178176
1      | 25327   | 34         | 9         | 2       | 8      | 0        | 21:13 40.91Submitted batch job 19181424
1      | 25327   | 31         | 12        | 2       | 10     | 0        | 21:35 27.97

But I still have 37 jobs active

1917813 regular   crew-tier_4-1-270fbaffbd9cca0bb7193 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n16
1917813 regular   crew-tier_4-2-f4c640ba8d38c3daa1987 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n10
1917813 regular   crew-tier_4-3-cdae1c0b332273675213d mangiola  R      4:26:48     19:33:12     2     80G    1 il-n18
1917814 regular   crew-tier_4-4-2a9038759411549a94518 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n18
1917814 regular   crew-tier_4-5-e16b7a6076d0f50c9253b mangiola  R      4:26:48     19:33:12     2     80G    1 il-n18
1917814 regular   crew-tier_4-6-7b18b5b91d58799a27216 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n18
1917814 regular   crew-tier_4-7-79912f67216802be88b4b mangiola  R      4:26:48     19:33:12     2     80G    1 il-n19
1917814 regular   crew-tier_4-8-855a27c3b70ff8c2cb7ae mangiola  R      4:26:48     19:33:12     2     80G    1 il-n19
1917814 regular   crew-tier_4-9-82caf312bf9bf5e768943 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n19
1917814 regular   crew-tier_4-10-9e9d88fd486e4c6beceb mangiola  R      4:26:48     19:33:12     2     80G    1 il-n19
1917814 regular   crew-tier_4-11-b2896f73733bb59909f9 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n19
1917814 regular   crew-tier_4-12-76ca4bab8cad32333d57 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n03
1917814 regular   crew-tier_4-13-63b6e42ba68b9b5d4b55 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n03
1917815 regular   crew-tier_4-14-605f41825e51c4ca13d0 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n02
1917815 regular   crew-tier_4-15-5ccc0f081913379190ab mangiola  R      4:26:48     19:33:12     2     80G    1 il-n07
1917815 regular   crew-tier_4-16-9e6a4847ec4d12e8f0dc mangiola  R      4:26:48     19:33:12     2     80G    1 il-n07
1917815 regular   crew-tier_4-17-25ce78611ebf21a434d7 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n07
1917815 regular   crew-tier_4-18-d98230321d082015230b mangiola  R      4:26:48     19:33:12     2     80G    1 il-n07
1917815 regular   crew-tier_4-19-cc32378ff1057eb09be6 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n05
1917815 regular   crew-tier_4-20-ba9d2c72beb9118e8b29 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917815 regular   crew-tier_4-21-2562406ab247bc9f4e13 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917815 regular   crew-tier_4-22-ef8bb461a67615cad697 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917815 regular   crew-tier_4-23-a4255b63ef2bf8cc9544 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917816 regular   crew-tier_4-24-93a68159bc2f9f4b3a2e mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917816 regular   crew-tier_4-25-fd51a9d57899d78840a6 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n04
1917816 regular   crew-tier_4-26-0b819e7564fc3bfc8d83 mangiola  R      4:26:48     19:33:12     2     80G    1 il-n06
1917816 regular   crew-tier_4-27-fe9520ce2661a35a2409 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n19
1917816 regular   crew-tier_4-28-bdb38bed06945cdd8ca4 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n22
1917816 regular   crew-tier_4-30-1a462d51886eb1a76608 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n24
1917816 regular   crew-tier_4-31-f690fffaf339e917ec93 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n05
1917816 regular   crew-tier_4-32-f3e9e58b8b17a7cce26f mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n06
1917816 regular   crew-tier_4-33-39e1f271b0e89cc9f9e6 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n07
1917817 regular   crew-tier_4-34-a827ad0687aaf72af0ca mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n02
1917817 regular   crew-tier_4-35-26620a7c8563e0228e80 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n03
1917817 regular   crew-tier_4-36-58a88e8fc2751a1559f0 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n04
1917817 regular   crew-tier_4-37-0984c919090fe0848054 mangiola  R      4:26:48     19:33:12     2     80G    1 sml-n09

I think workers that do not have any more jobs should be closed. The fact that they stay alive create also catastrophic situations, where ghost workers remain alive (maybe for memory overflow or failed jobs, never understood the reason) and stop other workers, with other resources from starting.

Am I missing something?

[wlandau]

37 alive workers with 31 dispatched targets can happen if those extra 6 workers recently completed tasks. It depends on your tasks_max and seconds_idle settings. Even after completing a task, a worker can remain in an idle state for seconds_idle seconds if tasks_max is greater than 1. That's actually part of the point: to avoid the startup time of e.g. requesting a new SLURM job, allow the worker to run a little longer if there is a chance it will pick up new tasks.

[stemangiola]

Thanks, I see.

As a comment, if the goal is to reduce inefficiencies, a default seconds_idle << Inf would help. There must be a reasonable value in real-world pipelines. With seconds_idle = Inf (as it is default now), it means that if I have 200 workers but just one job left that takes 5 hours, I keep occupied those resources for hours on hand and block many other tasks with a different worker setup to start.

In the worst-case scenario, how much lag could we have that is worth waiting for? 1 minute?

Just to let me understand, what is happening in the server<->main node that would delay pick-up of tasks?

[wlandau]

Yeah, that's a good point. I just changed the default seconds_idle to 300. It could still lead to a lot of idle time, but it should still enough processing time on the targets main process. In general though it's best to think about a good value for the pipeline at hand and set it explicitly.

[stemangiola]

I set it to 30 seconds, but I don't even know what would a realistic median time lag would be. For example, hypothetically, in SLURM do we expect a worker to remain idle (because of some lag) for 0.5 seconds , but we setup 30 seconds for safety, or the lag is 5-10 seconds realistically, so to not have that worker die we need more 50-100 seconds.

As I don't understand the cause of lag, these thresholds seem still arbitrary to me.