Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reproducibility versus replicability #7

Open
valdanchev opened this issue Dec 24, 2021 · 1 comment
Open

reproducibility versus replicability #7

valdanchev opened this issue Dec 24, 2021 · 1 comment

Comments

@valdanchev
Copy link

Great to see this implemented. Definitions of reproducibility and replication differ across domains, and would probably be helpful to clarify these in a few places. Happy to add these. In probably the most accepted definition now reproducibility would mean the use of the same data sets, techniques, scripts, and framework by independent researchers to obtain the same results. Replication in this setting is a bit tricky though—would the only difference be in the implemented framework, TensorFlow versus PyTorch? Are there other underlying differences between the two frameworks, which may contribute to differences also in how the model is trained or in the results, depending on whether model training or results are replicated?

@VictorSanh
Copy link
Member

thanks for raising that point @valdanchev !

Replication in this setting is a bit tricky though—would the only difference be in the implemented framework, TensorFlow versus PyTorch? Are there other underlying differences between the two frameworks, which may contribute to differences also in how the model is trained or in the results, depending on whether model training or results are replicated?

The main differences for the replication of the training will be:

  • framework: tf vs pytorch
  • optimizer: adafactor vs adam
  • data processing: example packing vs no example packing

(under the folder evaluation, I used "reproduce" but as you noted, a better term would be "replicate")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants