feat: add async model average algorithm

[NOBLES5E]

No description provided.

[wangraying]

update @NOBLES5E

It seems using LL128 protocol will cause the problem to hang. I have reproduced the problem through a sample code backend by torch+cupy. This is consistent with the results on Bagua.

Using the other two protocols LL and Simple does not hang as far as I know.

I have posted an issue on github, which we could track later on.

[todo]

remove nccl proto check

bagua/bagua/torch_api/algorithms/async_model_average.py

Lines 14 to 19 in f550bcd

	# TODO: remove nccl proto check
	proto_str = os.environ.get("NCCL_PROTO", "")
	if (
	proto_str == ""
	or ("^" not in proto_str and "LL128" in proto_str)
	or ("^" in proto_str and "LL128" not in proto_str)

---

This comment was generated by todo based on a TODO comment in f550bcd in #110. cc @BaguaSys.

[wangraying]

@wangraying fix CI so that we can merge this
We also need a tutorial page on this algorithm

This CI seems broken.

[wangraying]

And do you have any further comment about the usage, since we add a function barrier to end async threads.

@NOBLES5E

[todo]

; remove this after NVIDIA/nccl#549 gets solved

bagua/bagua/torch_api/algorithms/async_model_average.py

Lines 23 to 28 in 1c33cf9

	) # TODO; remove this after https://github.com/NVIDIA/nccl/issues/549 gets solved
	class AsyncModelAverageAlgorithm(Algorithm):
	def __init__(
	self, peer_selection_mode: str = "all", sync_interval_ms: int = 500,

---

This comment was generated by todo based on a TODO comment in 1c33cf9 in #110. cc @BaguaSys.

[NOBLES5E]

I refined the texts and left some comments. @wangraying Please do a final check