Incorporate heuristics to improve package prioritization

[charliermarsh]

See: #3078

[konstin]

I looked at the numbers of versions tried for some of our benchmark cases:

jupyter: Unchanged

Tried 100 versions: anyio 1, argon2-cffi 1, argon2-cffi-bindings 1, arrow 1, asttokens 1, async-lru 1, attrs 1, babel 1, beautifulsoup4 1, bleach 1, certifi 1, cffi 1, charset-normalizer 1, comm 1, debugpy 1, decorator 1, defusedxml 1, exceptiongroup 1, executing 1, fastjsonschema 1, fqdn 1, h11 1, httpcore 1, httpx 1, idna 1, ipykernel 1, ipython 1, ipywidgets 1, isoduration 1, jedi 1, jinja2 1, json5 1, jsonpointer 1, jsonschema 1, jsonschema-specifications 1, jupyter 1, jupyter-client 1, jupyter-console 1, jupyter-core 1, jupyter-events 1, jupyter-lsp 1, jupyter-server 1, jupyter-server-terminals 1, jupyterlab 1, jupyterlab-pygments 1, jupyterlab-server 1, jupyterlab-widgets 1, markupsafe 1, matplotlib-inline 1, mistune 1, nbclient 1, nbconvert 1, nbformat 1, nest-asyncio 1, notebook 1, notebook-shim 1, overrides 1, packaging 1, pandocfilters 1, parso 1, pexpect 1, platformdirs 1, prometheus-client 1, prompt-toolkit 1, psutil 1, ptyprocess 1, pure-eval 1, pycparser 1, pygments 1, python-dateutil 1, python-json-logger 1, pyyaml 1, pyzmq 1, qtconsole 1, qtpy 1, referencing 1, requests 1, rfc3339-validator 1, rfc3986-validator 1, root 1, rpds-py 1, send2trash 1, six 1, sniffio 1, soupsieve 1, stack-data 1, terminado 1, tinycss2 1, tomli 1, tornado 1, traitlets 1, types-python-dateutil 1, typing-extensions 1, uri-template 1, urllib3 1, wcwidth 1, webcolors 1, webencodings 1, websocket-client 1, widgetsnbextension 1

boto3: unchanged

boto3: Unchanged Tried 2553 versions: botocore 1697, boto3 849, urllib3 2, jmespath 1, python-dateutil 1, root 1, s3transfer 1, six 1

transformers-extras: improvement

Tried 1191 versions: sagemaker 152, hypothesis 67, tensorflow 21, jsonschema 19, tensorflow-cpu 18, multiprocess 10, pathos 10, tensorflow-text 10, chex 8, tf-keras 8, tf2onnx 8, aiohttp 6, aiosignal 6, alembic 6, annotated-types 6, apscheduler 6, attrs 6, backoff 6, binaryornot 6, black 6, boto3 6, click 6, coloredlogs 6, colorlog 6, dash 6, dash-bootstrap-components 6, dlinfo 6, exceptiongroup 6, execnet 6, fire 6, frozenlist 6, gitdb 6, google-auth 6, google-auth-oauthlib 6, hjson 6, iniconfig 6, jinja2-time 6, markdown 6, markdown-it-py 6, markupsafe 6, mpmath 6, namex 6, nbformat 6, ninja 6, nvidia-nvjitlink-cu12 6, onnxconverter-common 6, pandas 6, plac 6, platformdirs 6, pluggy 6, portalocker 6, poyo 6, protobuf3-to-dict 6, py-cpuinfo 6, py3nvml 6, pyarrow 6, pyarrow-hotfix 6, pydantic-core 6, pygments 6, pynvml 6, pypng 6, python-slugify 6, responses 6, smdebug-rulesconfig 6, soupsieve 6, sqlalchemy 6, tensorboard-data-server 6, tensorboard-plugin-wit 6, tensorboardx 6, threadpoolctl 6, tomli 6, wasabi 6, wcwidth 6, werkzeug 6, wheel 6, xxhash 6, zipp 6, etils 5, tensorboard 5, beautifulsoup4 4, cffi 4, clldutils 4, codecarbon 4, datasets 4, dill 4, evaluate 4, gitpython 4, hf-doc-builder 4, kenlm 4, librosa 4, llvmlite 4, nest-asyncio 4, nltk 4, optuna 4, parameterized 4, phonemizer 4, psutil 4, pyctcdecode 4, pytest 4, pytest-timeout 4, pytest-xdist 4, ray 4, rjieba 4, rouge-score 4, ruff 4, sacrebleu 4, sacremoses 4, sigopt 4, sortedcontainers 4, tensorstore 4, timeout-decorator 4, toolz 4, torchaudio 4, accelerate 3, audioread 3, cookiecutter 3, decorator 3, deepspeed 3, faiss-cpu 3, flax 3, fugashi 3, ipadic 3, isort 3, jax 3, jaxlib 3, joblib 3, keras-nlp 3, lazy-loader 3, numba 3, optax 3, pooch 3, pydantic 3, pygtrie 3, rhoknp 3, scikit-learn 3, segments 3, soundfile 3, soxr 3, sudachidict-core 3, sudachipy 3, torch 3, unidic 3, unidic-lite 3, urllib3 3, absl-py 2, arrow 2, astunparse 2, async-timeout 2, botocore 2, cachetools 2, certifi 2, chardet 2, charset-normalizer 2, csvw 2, dash-core-components 2, dash-html-components 2, dash-table 2, diffusers 2, dm-tree 2, fastjsonschema 2, flask 2, flatbuffers 2, fsspec 2, ftfy 2, gast 2, google-pasta 2, greenlet 2, grpcio 2, h5py 2, humanfriendly 2, idna 2, importlib-metadata 2, importlib-resources 2, jinja2 2, jmespath 2, jupyter-core 2, kagglehub 2, keras 2, keras-core 2, keras-preprocessing 2, libclang 2, mako 2, mdurl 2, ml-dtypes 2, msgpack 2, multidict 2, mypy-extensions 2, networkx 2, nvidia-cublas-cu12 2, nvidia-cuda-cupti-cu12 2, nvidia-cuda-nvrtc-cu12 2, nvidia-cuda-runtime-cu12 2, nvidia-cudnn-cu12 2, nvidia-cufft-cu12 2, nvidia-curand-cu12 2, nvidia-cusolver-cu12 2, nvidia-cusparse-cu12 2, nvidia-nccl-cu12 2, nvidia-nvtx-cu12 2, onnx 2, onnxruntime 2, onnxruntime-tools 2, opencv-python 2, opt-einsum 2, orbax-checkpoint 2, pathspec 2, plotly 2, pox 2, ppft 2, pyasn1-modules 2, pycparser 2, pyrsistent 2, python-dateutil 2, pytz 2, requests-oauthlib 2, retrying 2, rich 2, rsa 2, s3transfer 2, scipy 2, setuptools 2, six 2, smmap 2, sympy 2, tabulate 2, tensorflow-estimator 2, tensorflow-hub 2, tensorflow-io-gcs-filesystem 2, termcolor 2, text-unidecode 2, traitlets 2, triton 2, typing-extensions 2, tzdata 2, tzlocal 2, wrapt 2, xmltodict 2, yarl 2, Python 1, av 1, babel 1, bibtexparser 1, blinker 1, colorama 1, decord 1, filelock 1, huggingface-hub 1, isodate 1, itsdangerous 1, language-tags 1, lxml 1, numpy 1, oauthlib 1, packaging 1, pillow 1, protobuf 1, pyasn1 1, pylatexenc 1, pyparsing 1, pyyaml 1, rdflib 1, regex 1, requests 1, rfc3986 1, root 1, safetensors 1, sentencepiece 1, tenacity 1, timm 1, tokenizers 1, torchvision 1, tqdm 1, transformers 1, types-python-dateutil 1, uritemplate 1 PR: Tried 810 versions: tf2onnx 16, chex 8, multiprocess 8, pathos 8, clldutils 7, csvw 5, schema 5, tabulate 5, tensorstore 5, arrow 4, astunparse 4, binaryornot 4, black 4, certifi 4, charset-normalizer 4, click 4, cloudpickle 4, coloredlogs 4, colorlog 4, contextlib2 4, dash 4, dash-core-components 4, dash-html-components 4, dash-table 4, deepspeed 4, dill 4, dlinfo 4, docker 4, evaluate 4, fastjsonschema 4, fsspec 4, fugashi 4, gast 4, google-pasta 4, grpcio 4, hjson 4, humanfriendly 4, idna 4, iniconfig 4, ipadic 4, isort 4, jinja2 4, jinja2-time 4, joblib 4, jsonschema 4, jupyter-core 4, keras 4, libclang 4, mako 4, markdown 4, ml-dtypes 4, mypy-extensions 4, namex 4, nbformat 4, nest-asyncio 4, ninja 4, onnxconverter-common 4, onnxruntime 4, onnxruntime-tools 4, opt-einsum 4, pathspec 4, portalocker 4, pox 4, poyo 4, ppft 4, py3nvml 4, pyarrow-hotfix 4, pycparser 4, python-dateutil 4, python-slugify 4, retrying 4, rfc3986 4, rhoknp 4, rouge-score 4, sacrebleu 4, sagemaker 4, scikit-learn 4, scipy 4, segments 4, six 4, smdebug-rulesconfig 4, sudachidict-core 4, sudachipy 4, tensorboard 4, tensorflow-cpu 4, tensorflow-estimator 4, tensorflow-hub 4, tensorflow-io-gcs-filesystem 4, tf-keras 4, traitlets 4, types-python-dateutil 4, typing-extensions 4, unidic 4, unidic-lite 4, uritemplate 4, urllib3 4, wrapt 4, xxhash 4, aiohttp 3, aiosignal 3, async-timeout 3, attrs 3, blinker 3, boto3 3, botocore 3, cachetools 3, chardet 3, datasets 3, exceptiongroup 3, filelock 3, flask 3, frozenlist 3, google-auth 3, google-auth-oauthlib 3, hypothesis 3, itsdangerous 3, jmespath 3, jsonschema-specifications 3, markupsafe 3, multidict 3, oauthlib 3, onnx 3, plac 3, platformdirs 3, plotly 3, pluggy 3, pyarrow 3, pyasn1 3, pyasn1-modules 3, pyctcdecode 3, pygtrie 3, pytest 3, ray 3, referencing 3, requests-oauthlib 3, responses 3, rpds-py 3, rsa 3, s3transfer 3, sortedcontainers 3, tblib 3, tenacity 3, tensorboard-data-server 3, tensorboardx 3, text-unidecode 3, threadpoolctl 3, timm 3, tomli 3, toolz 3, torchvision 3, wasabi 3, werkzeug 3, wheel 3, yarl 3, absl-py 2, accelerate 2, alembic 2, annotated-types 2, apscheduler 2, audioread 2, backoff 2, beautifulsoup4 2, cffi 2, dash-bootstrap-components 2, decorator 2, diffusers 2, dm-tree 2, etils 2, execnet 2, faiss-cpu 2, fire 2, flatbuffers 2, flax 2, ftfy 2, gitdb 2, gitpython 2, greenlet 2, h5py 2, hf-doc-builder 2, importlib-metadata 2, jax 2, jaxlib 2, kagglehub 2, kenlm 2, keras-core 2, keras-nlp 2, lazy-loader 2, librosa 2, llvmlite 2, markdown-it-py 2, mdurl 2, mpmath 2, msgpack 2, networkx 2, nltk 2, numba 2, nvidia-cublas-cu12 2, nvidia-cuda-cupti-cu12 2, nvidia-cuda-nvrtc-cu12 2, nvidia-cuda-runtime-cu12 2, nvidia-cudnn-cu12 2, nvidia-cufft-cu12 2, nvidia-curand-cu12 2, nvidia-cusolver-cu12 2, nvidia-cusparse-cu12 2, nvidia-nccl-cu12 2, nvidia-nvjitlink-cu12 2, nvidia-nvtx-cu12 2, opencv-python 2, optax 2, optuna 2, orbax-checkpoint 2, pandas 2, parameterized 2, phonemizer 2, pooch 2, protobuf 2, psutil 2, py-cpuinfo 2, pydantic 2, pydantic-core 2, pygments 2, pynvml 2, pypng 2, pytest-timeout 2, pytest-xdist 2, pytz 2, rich 2, rjieba 2, sacremoses 2, setuptools 2, sigopt 2, smmap 2, soundfile 2, soupsieve 2, soxr 2, sqlalchemy 2, sympy 2, tensorflow-text 2, termcolor 2, timeout-decorator 2, torch 2, torchaudio 2, triton 2, tzdata 2, tzlocal 2, wcwidth 2, zipp 2, Python 1, av 1, babel 1, bibtexparser 1, codecarbon 1, colorama 1, cookiecutter 1, decord 1, huggingface-hub 1, importlib-resources 1, isodate 1, language-tags 1, lxml 1, numpy 1, packaging 1, pillow 1, pylatexenc 1, pyparsing 1, pyyaml 1, rdflib 1, regex 1, requests 1, root 1, ruff 1, safetensors 1, sentencepiece 1, tensorflow 1, tokenizers 1, tqdm 1, transformers 1, xmltodict 1

We should run a number of wall time benchmarks too.

[konstin]

This should greatly improve difficult resolution cases

[konstin]

This needs some comment about the heuristic, a reference to pip and that a lower entry means we pick it first

[zanieb]

👀 this seems like a big improvement?

[charliermarsh]

Haha. It might be somewhat random? Maybe it's not.

[zanieb]

I think we just get to the core of the incompatibility sooner

[charliermarsh]

Yeah that makes sense.

[zanieb]

This is surprisingly elegant, nice work :)

[notatallshaw]

Wow thanks for this! I won't be able to do more testing today but I checked quickly how this change affected the quite difficult requirements in #1398, performance is about the same (within error margin after three runs).

This change picks newer versions of boto3, botocore, and s3transfer, and a much older version of s3fs. This looks like it might be the more desirable user behavior! And if not, it's more obvious that s3fs is older and the user should put a minimum lower bound constraint on it if that's their preferred solution.

P.S, a lot of the difficulty in resolving in that example is the seemingly aggressive bounds (in particular upper bounds) that aiobotocore puts on botocore, it makes many user resolutions impossible, e.g. you can't have a new s3fs and a new boto3, and it's not obvious why aiobotocore is the reason to an end user. As you can see from a PR earlier this year, there were no actual functional changes required to increase the upper bound aio-libs/aiobotocore#1087. That's a tradeoff that that project might be happy with, but hopefully this can be seen as an example of why it would be bad for the ecosystem in general to take this approach.