bootstrap_impl=script doesn't allow overriding libs in the runtime's site-packages directory

[rickeylev]

In #2032, it was pointed out that if you depend on a newer version of pip than what the runtime has, the newer version isn't respected.

This is because, with the bootstrap_impl=script backend, all the pypi paths are append to sys.path. This results in e.g. [stdlib, site-packages, @pypi/pip]. In the old system_python bootstrap, the ordering would be [@pypi/pip, stdlib, site-packages].

The original intent with changing the ordering as to make sure shadowing the stdlib didn't happen, which was relatively easy to do when repo's are on the path. However, a side-effect of appending is that, e.g. a newer version of pip isn't respected.

So, to fix this, we just need to put the pypi entries after the stdlib, but before site-packages.

I'm not sure if there's an API to identify which sys.path entry is the site-packages entry. Worst case, we'll just have to sniff the values and try to insert in the appropriate place.

Notes:

• There is site-packages and dist-packages. dist-packages appears to occur as part of the distribution (e.g. part of the OS python install)
• site.getsitepackages() appears to have the list of directories
• I couldn't find a simple way to divine the stdlib paths. Construction of them uses various substrings and looks non-trivial; see https://docs.python.org/3/library/sys_path_init.html#sys-path-init
• IMHO: Search forward on sys.path for {site,dist}-packages. Insert before that entry. Maybe add a sanity check to ensure it's after stdlib-looking entries