Node sampling history of strategy is erased when nodes fail

[sharkovsky]

When nodes fail during an experiment, they are also removed from the sampling history of the strategy class.
This should not happen, because the purpose of the history is to keep track of which nodes had been selected by the strategy, not only those that didn't fail.

Steps to reproduce

on develop commit 850f73d
Based on 101 notebook

1. insert assert False in the training_step of the training plan. This simulates a node failure.
2. execute exp.run
3. the output of exp.strategy()._sampling_node_history is {0: []}, while it should contain the ids of the selected nodes.

Furthermore, the experiment fails with FB401: aggregation crashes or returns an error. Aggregation aborted due to sum of the weights is equal to 0 {}. Sample sizes received from nodes might be corrupted..
However, the fact that no nodes succeeded should be caught before the aggregation is even attempted.

[sharkovsky]

@lchambon and myself will look at this together

[manos-mark]

I think I have the same problem in a real-world experiment. I got the error

Cannot instantiate training plan object

and now I can't reverse it. Even if I changed the training plan to a previous "working" version.
I work with VPN settings and I tried to restart the node and the researcher but there is no difference.

[mvesin]

Hi @manos-mark
I'm not sure to understand how your issue is related to the previous one. Could you give a few info about:

• when does the error occur: aggregation on the researcher ? other ?
• exp.strategy()._sampling_node_history ?
• exp.all_federation_nodes() ?

Thank you for reporting !