More efficient scheduling of large ensembles

[cylc]

This is just an idea for consideration. See also #108.

Currently if a "family trigger" is used in the graph to trigger a large ensemble, the family trigger is replaced with the equivalent expression involving all the ensemble members, and at run time every member is represented by its own dedicated task proxy. The efficiency of the scheduling algorithm depends on the size of the task pool, and large ensembles are the most likely cause of task pool bloat. If all ensemble members trigger at once, and if downstream tasks trigger off the entire ensemble, then the suite would be just as well served by a single task proxy representing the entire ensemble. The family proxy would have to take messages from each of its members, keep track of each of their states, know how to submit all of them at once, and so on. This would result in a massive performance boost for very large ensemble suites.

[cylc]

Complications:

1. if an individual member is singled out in a non-family dependency relationship, it could be removed from family control and given its own task proxy. Or perhaps the family proxy could also manage these relationships.

2. how to display these aggregate families in gcontrol etc. - perhaps clicking on a family would just bring up a summary of the member states.

3. what about cylc's internal queues, which work with individual tasks?

4. what about internal dependence among ensemble members?

[cylc]

Here's an idea - keep the individual member proxies but have them interact only amongst themselves in a separate pool (to handle any internal dependence) while they are represented in the main pool by a single family proxy.

[hjoliver]

It was suggested in @cylc/core meeting, that dependency matching could work with shared "dependency objects" rather than individual task proxies. Members of graphed families would automatically share the same dependency objects - would probably solve this issue and #1776 in one whack (if the same shared objects could be used to define graph edges). Bumping up to 'soon' on that basis...

[hjoliver]

The new idea (prev comment) is superior - closing this and moving that to a new Issue.