Is there a way to seed experiments?

[bartonp2]

❓ Questions and Help

What is your question?

Is there a way to seed experiments? Attempts at using seed_everything from pytorch lightning do not appear to work (also with the the workers argument set to True).

What's your environment?

• OS: Linux
• Packaging: pip
• Version: 0.5.2

[ethanwharris]

Hey @bartonp2 Thanks for the question 😃 As far as we are aware seed_everything should be working in flash (and I think some of our tests depend on it) so if it isn't then I would consider that a bug. Could you share a reproducible script that illustrates the problem?

[bartonp2]

You are right, the seeding works.

The problem turned out to be in the SemanticSegmentationFolderInput class. Specifically, the load_data() function lists files in an arbitrary order and the intersection() function scrambles the order again. The problem is solved by adding:

files.sort()
mask_files.sort()

to the end of the load_data() function.

This might we be worth changing in order to make results reproducible.

[ethanwharris]

Ah, @bartonp2 thanks for your work debugging this. Would you be interested in opening a PR to fix this? All of our file handling is currently in here:
https://github.com/PyTorchLightning/lightning-flash/blob/master/flash/core/data/utilities/paths.py

Let me know if you're interested and I can assign you to the issue. Also happy to help with it if you'd like 😃

[bartonp2]

I could although I don't have time the next couple of weeks. But I could do it after.

[ethanwharris]

That would be great 😃 I don't think there will be much progress on Flash over the next few weeks anyway