Put our Python sitecustomize.py into the site-packages folder

[Flamefire]

We use a separate folder for that file that handles $EBPYTHONPREFIXES:

easybuild-easyblocks/easybuild/easyblocks/p/python.py

Lines 147 to 148 in 5838f1d

	# Used for EBPYTHONPREFIXES handler script
	self.pythonpath = os.path.join(log_path(), 'python')

This requires putting it into $PYTHONPATH. If we'd put it into the site-packages folder directly we don't need that entry.

Downside is that ebpythonprefixes cannot be disabled by --module-only rebuilds: https://github.com/easybuilders/easybuild-easyblocks/blob/5838f1d639566ae3e0da915f1b8a31c6d3cf04d4/easybuild/easyblocks/p/python.py#L641C26-L642

However it avoids an additional path entry and potential conflicts with software clearing $PYTHONPATH to workaround user mistakes.

I found that for Spyder (IDE): spyder-ide/spyder#22681

[boegel]

Hmm, that's probably a better suggestion than what's proposed in #3400

Downside is that ebpythonprefixes cannot be disabled by --module-only rebuilds

Why is that a problem exactly?

We could add an opt-out (based on an environment variable) to sitecustomize.py which we can trigger when --module-only is used?

[boegel]

@Micket Thoughts on this?

[Flamefire]

Downside is that ebpythonprefixes cannot be disabled by --module-only rebuilds

Why is that a problem exactly?

Because it is a) a breaking change as that worked before and b) for users to change ebpythonprefixes-handling afterwards they'd need to rebuild the whole module.

We could add an opt-out (based on an environment variable) to sitecustomize.py which we can trigger when --module-only is used?

That would work yes, but would still cause some unnecessary overhead when starting any python script/application due to Python loading the sitecustomize and then do nothing. Not sure if we should care though...

Question is then: Should we always install it and allow to disable it even when a site does never want to use it? That would be required for module-only to work both ways. I guess that is the right approach though as there are multiple downsides of using PYTHONPATH for modules (e.g. virtualenv-compatibility)

[Micket]

I don't know if placing sitecustomize separately had some strong reason behind it, so i'm passing that question on to @bartoldeman

I'm not the least bit concerned with users who change their mind on if they want to support ebpythonprefixes or not. Worst case, a rebuild, or you can fix it manually into existing installations if you can't afford the "downtime" (i.e. just delete sitecustomize.py).
That whole situation is niche^2.
I vote to keep that everything as simple as possible.

The only thing we need to keep track of if we change location is to keep the old PYTHONPATH when doing --module-only --rebuild or --skip --rebuild, which seems simple enough.
So this condition
https://github.com/easybuilders/easybuild-easyblocks/blob/5838f1d639566ae3e0da915f1b8a31c6d3cf04d4/easybuild/easyblocks/p/python.py#L641C1-L643C1
must change to checking it sitecustomize.py exists in that location, if so, add the PYTHONPATH.

[boegel]

Input from @bartoldeman would be nice too here :)