refactor/generalize autolabor/autohauler/labormanager

[ab9rf]

These three plugins are variations on a theme and share a lot of code by copy and paste. They also conflict with one another; running more than one of them at the same time will generally cause problems. It makes sense to consolidate them.

I also have an idea for a different approach to labor assignment that would use some of the same functionality that's in labormanager now, but would maintain persistent state and operate differently. Rather than simply replace or fork labormanager, I'd rather implement it as an alternative within a common framework, keeping the existing version without adding further duplicated code.

I have long wanted to combine them into a single plugin that has the ability to do the work of any one of them. The idea is to have a single plugin that then will use whichever submodule the user selects.

My initial strategy is going to be along these lines:

• refactor labormanager into two parts, the common core that is shared and the code that is specific to labormanager
• add code to labormanager that allows the user to select which manager to use
• reimplement autolabor and autohauler as alternative managers within this framework

[myk002]

Having all three plugins use a common core (or combining them all into a single plugin) sounds like an excellent idea. I endorse. What are your ideas for the new approach to labor assignment?

[ab9rf]

The main change is that I was going to order the jobs oldest-first (which means maintaining state on when jobs appear on the job list, but that's not really that hard). Then, assign the "best" unit for each job, in order by age of job.

I'm not sure where it goes from there. labormanager has always had the primary goal of avoiding job starvation, and has been willing to sacrifice worker quality to achieve that, but I'm unhappy with some of its behaviors and think it can be improved with some use of persistent state, something it currently does not do. Also, the current code basically creates itself each run and then tears itself back down again, which is ridiculously wasteful. Just making those auxiliary structures persist is likely to improve performance at least somewhat.

[myk002]

Job starvation is actually also why I wrote the prioritize script. That script is based on the observation that there are certain job types that I always want done first, but that the game will ignore indefinitely in busy forts (e.g. CleanSelf, store item in vehicle, etc.). I wonder which approach is fundamentally better suited to solve the job starvation problem. I'm pretty happy with the performance of the prioritize script in testing, but that may just be due to my particular playstyle.

I don't think that it's an either/or, though. Labormanager automatically distributes labors to dwarves. That distribution is just as important as job type prioritization.

[myk002]

It occurs to me that what we're both trying to do is guarantee a minimum level of service for each job type in a constantly changing environment. This is a problem that is well explored in the SRE (Site Reliability Engineering) space. Perhaps we should be looking at online service platform operations methods for inspiration.

To wit, we could measure latency (in ticks) for active jobs (the SLI, or service level indicator) define a target latency, measurement window, and error budget for each job type (the SLO, or service level objective), and emit alerts when labor allocations need to change to keep the latencies within the error budgets. The nice part about this approach is that it automatically adjusts itself to player actions, like if the player manually sets the labors for some dwarves or uses the prioritize or do-job-now scripts to set job priorities. Let me play with it a little and see if I can come up with a simple API. If it works out, this can be the engine for "do I need more dwarves or fewer for labor X". The rest of the code can focus on how to assign the labors (and on mapping the jobs to labors).

What do you think of the concept?

[ab9rf]

I have considered doing this many times, but have never done much more than think about it. It would be a good solution to the problem, but I've not really any good ideas of what the algorithm requires looks like in enough detail to implement it.

[myk002]

I'll see if I can write a standalone proof of concept. I do have concerns about the viability of the SLO approach, especially around getting enough events within a measurement window to make the calculations statistically significant -- we need the graph to be sufficiently smooth in order to draw conclusions from it. If the measurement window has to be large, though, the responsiveness of the algorithm will suffer. There are also other elements we have to calculate that the SLO measurements won't directly tell us, like "will adding another dwarf with labor X actually help getting the tasks done faster?". For example, constructing the floor of an aboveground building will generate a lot of ConstructBuilding jobs, but only a few tiles of flooring can be accessed at a time, so adding more dwarves wouldn't help, and they'd be more productive allocated to other tasks.

I'm also worried about the mismatch between having the SLOs based on job_type but our remediations (that is, the knobs that we tweak) being based on unit_labor. For example, both item hauling and sand collection are controlled by the HAUL_ITEM labor. If there are both many items to haul and lots of sand to collect, adding more dwarves with the HAUL_ITEM labor won't necessarily bring sand collection tasks back within SLO. We may need an additional facility to force a particular out-of-SLO job to be prioritized (i.e. setting the do_now flag) if it would otherwise get ignored (but of course we also need to not prioritize out-of-SLOs jobs if it would cause other, more important job types to go out of SLO).

In your mind, what should the high-level algorithm look like? [s]I think it needs to be more nuanced than "oldest job first", since we'd get all our dwarves assigned to hauling that way.[/s]
edit: actually, maybe that's not true. jobs for hauling are not generated at item creation time. there are limited number of hauling jobs created at one time from available haulable items. Maybe I'm overthinking this and the oldest-job-first approach would give good results. Why don't you give it a try and see how it goes? I'll reshape my plans for the SLO-based engine so that it just measures the effectiveness of any other solution we try.