Question re: Table 1 in Empowering Federated Learning for Massive Models with NVIDIA FLARE

[hwpang]

Hi! Thanks for the great package! I'm currently reading through the publication at https://arxiv.org/abs/2402.07792, and I'm very surprised by the results in Table 1. It's very surprising to me that the model trained by federated learning can outperform the model trained by centralized learning with combined dataset consistently.

To me, the amount of information the model can learn from the dataset should theoretically be equivalent, regardless of the federated learning vs. centralized learning settings. Moreover, this results would imply that even if one has all the data in one place, one should still apply federated learning (perhaps with simulated clients) rather than centralized learning.

I would appreciate any explanation or insights, thanks!

[YuanTingHsieh]

@holgerroth and @ZiyueXu77 can you help on this? thanks

[ZiyueXu77]

Hi @hwpang, thanks for the question and yes you are spot-on!
Both training and evaluation of LLM can be tricky, especially when comparing different datasets. There is indeed one dilemma here: our trainer requires a fixed-length epoch - instead of considering going over the entire dataset as "an epoch", it first samples a fixed size from the local dataset.
Therefore, here we select equal length for all training cases, and thus even though the "combined" dataset enjoys greater diversity, the training data size is still the same as other training cases - you can consider it 3x diversity, but 1/3 size from each source.
Now you can see why FL can perform better than "combined" in this case: each site has the full size, then by federation, it also enjoyed the 3x diversity.
Indeed for "upper bound", we should perform experiment with 3x length for combined dataset, we recognize it and plan to add it in our future work.

[hwpang]

@ZiyueXu77 Thanks for explaining! That makes a lot of sense