TL/UCP: reorder ranks for SRA

[shimmybalsam]

What

Add reorder ranks in for Allreduce SRA

Why ?

Performance benfit, especialy when n_extra ranks > 0, in which case we do a complete allreduce between "proxy" and "extra" ranks and if they are on the same node this significantly improves perf.

How ?

By reordring ranks in each step: reduce_scatter_knomial, allgather_knomial.

Note: allgather_knomial is used also for small message allgather and large message bcast (bcast SAG). In these cases reorder ranks will not be used, it is currently only implemented for the usage within allreduce SRA.

[shimmybalsam]

Without extra ranks:

---

With extra ranks:

[shimmybalsam]

bot:retest

[Sergei-Lebedev]

can we instead make map part of knomial pattern. That way we don't need to repeat ucc_ep_map_eval every time we get peer.
In future we can also add ring patter to do the same for ring algorithms

[shimmybalsam]

can we instead make map part of knomial pattern. That way we don't need to repeat ucc_ep_map_eval every time we get peer. In future we can also add ring patter to do the same for ring algorithms

As discussed on call, in my opinion we should merge this PR as is as first step, since it substantially improves performance of SRA. In future we can rewrite knomial pattern functions (which will effect all knomial algorithms in TL/UCP).

Meanwhile here is some more supportive data:

[shimmybalsam]

bot:retest