Print x / N for subtargets?

[kendonB]

Prework

• Read and abide by drake's code of conduct.
• Search for duplicates among the existing issues, both open and closed.

Proposal

In cases where the number of subtargets for a given dynamic target is known, it would be great to print something like

subtarget x/N target_name_hash

to give an indicator of progress. Is this feasible? Even when N is unknown you could still print x.

P.S. I have tried out the dynamic branching version of map with clustermq and SLURM and it seems to be working great!

[wlandau]

Glad to hear dynamic branching is working well for you.

Sorry, but I hesitate to add more to the console messages because things tend to get visually busy very easily and there is extra bookkeeping involved that could slow us down (particularly for x). Believe it or not, printing to the console adds a surprising amount of overhead if the number of targets is large and each target is quick. I see it all the time in profiling studies nowadays. I suspect that assigning and keeping track of integer sub-target indices will have a similar cost.

[kendonB]

I would still urge you to consider some optional progress indication.

[wlandau]

How do you feel about progress(), running(), and failed()?

[kendonB]

I haven't used them until now but there's definitely value there.

However, those functions can only tell me about what has happened or is going on now. To really know progress we also need to know how much is to come.

Is there any feaible way to have drake also know N above? Even in a limited number of cases or as a guess? Or for a particular target that has already started and is currently calculating subtargets?

[wlandau]

It does, actually.

drake/R/dynamic.R

Line 262 in daf204f

ndeps <- length(subtargets_build)

Right under here, we could insert config$logger$minor("register", ndeps, "subtargets", target = target).

[wlandau]

That way, you will see it in the console_log_file.

[wlandau]

Implemented.