bpo-36302: Sort list of sources

[bmwiedemann]

When building packages (e.g. for openSUSE Linux)
(random) filesystem order of input files
influences ordering of functions in the output .so files.
Thus without the patch, builds (in disposable VMs) would usually differ.

Without this patch, all callers have to be patched individually
ofalk/libdnet#42
sass/libsass-python#212
tahoe-lafs/pycryptopp#41
yt-project/yt#2206
pyproj4/pyproj#142
pytries/datrie#49
but that is an infinite effort.

See https://reproducible-builds.org/ for why this matters.

https://bugs.python.org/issue36302

[merwok]

Could there be negative consequences to the sorting?

I think the two most common idioms for defining sources in setup.py are 1) an explicit list with fixed order (e.g. sources = ["lib/thing.c", "lib/utils.c"]) and 2) a glob result.

Is it ok in 100% of cases to change the order for case 1) ?

[bmwiedemann]

Let's say the case 2 people seem to not have noticed any negative consequences from random order, so I would assume that it is fine for case 1, too. I also mailed this to our expert to be on the safe side.

[bmwiedemann]

Our compiler expert said, order does not matter for .so files. It only matters when using .a files in case of duplicate symbols - but those are for static linking and not for dynamic linking as required by python.

[bmwiedemann]

ping. Anything left to do on my side?

[bmwiedemann]

ping. Still wanting this. Hoping for @vstinner or who else could have a look.

Because I found another of the infinite callers.

[vstinner]

Would it make sense to modify where self.sources is used rather than sorting Extension constructor? For example, read_setup_file() function seems to modify the sources attribute afterwards.

Maybe modify CCompiler.compile()?

[bmwiedemann]

patching build_ext would probably work (edit: does not need test updates). read_setup_file code looks deterministic enough (unless someone autogenerates the setup file nondeterministically).
CCompiler.compile is overridden in msvccompiler.py so might not catch all cases.

link_shared_object might also be a good place to patch instead, because after all it is the linking order of .o files into .so that matters - it just happens to follow the source file order.

[bmwiedemann]

moved the sorting into build_ext

[vstinner]

Thanks for updated PR: sorting in build_extension() looks more reliable than sorting in Extension constructor.