Behavior of pin_compatible() is different for different runs?

[nehaljwani]

I have a simple recipe:

[root@f0795f5d6960 build_scripts]# cat meta.yaml 
package:
  name: pkg
  version: 1.0.0

requirements:
  host:
    - python
    - numpy
  run:
    - python
    - {{ pin_compatible('numpy') }}

I have a simple build matrix:

[root@f0795f5d6960 build_scripts]# cat conda_build_config.yaml 
numpy:
  - 1.9   # [unix]
python:
  - 2.7
  - 3.5
  - 3.6

In a fresh container of docker.io/conda/c3i-linux-64, build this recipe.

One building this package, one the first run, I get:

pkg-1.0.0-py27h24bf2e0_0.tar.bz2 -> "numpy >=1.9.3,<2.0a0"
pkg-1.0.0-py35h24bf2e0_0.tar.bz2 -> "numpy >=1.14.1,<2.0a0"
pkg-1.0.0-py36h24bf2e0_0.tar.bz2 -> "numpy >=1.9.3,<2.0a0"

Now, I run it again. This time I get:

pkg-1.0.0-py27h24bf2e0_0.tar.bz2 -> "numpy >=1.14.1,<2.0a0"
pkg-1.0.0-py35h24bf2e0_0.tar.bz2 -> "numpy >=1.14.1,<2.0a0"
pkg-1.0.0-py36h24bf2e0_0.tar.bz2 -> "numpy >=1.14.1,<2.0a0"

Now, I run it again. This time I get:

pkg-1.0.0-py27h24bf2e0_0.tar.bz2 -> "numpy >=1.9.3,<2.0a0"
pkg-1.0.0-py35h24bf2e0_0.tar.bz2 -> "numpy >=1.9.3,<2.0a0"
pkg-1.0.0-py36h24bf2e0_0.tar.bz2 -> "numpy >=1.9.3,<2.0a0"

Now, I run it again. This time I get same as above.

Now, I run it again. This time I get the same as the second run.

What's happening?

[nehaljwani]

FWIW, I caught this while skimming through the output of conda-verify, since it doesn't like >=,< run deps at the moment and kept saying:

Verifying ./linux-64/pyemd-0.4.4-py35h8619c78_0.tar.bz2...
./linux-64/pyemd-0.4.4-py35h8619c78_0.tar.bz2: C1114 Found invalid dependency "numpy >=1.9.3,<2.0a0" in info/index.json
Verifying ./linux-64/pyemd-0.4.4-py36h8619c78_0.tar.bz2...
./linux-64/pyemd-0.4.4-py36h8619c78_0.tar.bz2: C1114 Found invalid dependency "numpy >=1.14.1,<2.0a0" in info/index.json
Verifying ./linux-64/pyemd-0.4.4-py27h8619c78_0.tar.bz2...
./linux-64/pyemd-0.4.4-py27h8619c78_0.tar.bz2: C1114 Found invalid dependency "numpy >=1.14.1,<2.0a0" in info/index.json

... and my expression was:

[msarahan]

May have something to do with conda/conda#6821 (comment)

I have not been able to duplicate this issues with conda 4.3.x.

[mandeep]

@nehaljwani The conda-verify issue seems to be with the pre-release versioning rather than the relational operators. I've opened an issue on the issue tracker and hope to get it sorted out soon.

[nehaljwani]

@mandeep Thanks for looking into the conda-verify issue.

@msarahan Should we pin conda to 4.3 till this is resolved for feedstocks using pin_compatible()?

[msarahan]

@nehaljwani yes, that is the simplest answer for right now.

[kalefranz]

Does setting PYTHONHASHSEED keep the results consistent?

[nehaljwani]

@kalefranz Dumb query: where should I set it?

[kalefranz]

I guess as an environment variable before invoking conda-build. I don’t think conda-build ever subprocesses to conda. If it does, hopefully it passes the whole environment through.

[kalefranz]

As a first test, just set it to some constant value, like 1, and see if we get back to deterministic results.

[nehaljwani]

On setting that value to 1, and repeated runs, I always get the dep: "numpy >=1.14.1,<2.0a0" for all pythons, across multiple runs. On changing that value to 2, I always get the dep: "numpy >=1.9.3,<2.0a0" for all pythons, across multiple runs.

[kalefranz]

Ok. I saw similar nondetermistic behavior when working on conda/conda#6929. The solution there could solve this too. Nondetermisitic behavior like this is pretty rare, which at least hints at the same root cause.

[kalefranz]

So, some background... Conda 4.4 contains my own naive implementation of a dag data structure. Naive because I'm an electrical engineer, never having a formal data structures and algorithms class, and basically just wrote it up as I was working on development of conda/core/solve.py. It's pretty clear to me now (and probably anybody else that actually looks at it) that it has some problems. Not only is it just incorrect under certain cases, it's also way over-engineered. And, of course, non-deterministic in some cases.

So I'm reworking this whole dag data structure class. This time ripping off someone else's code as a starting point. The fix will probably be the last PR that goes into 4.5.0. I don't exactly want to put it in 4.4.x because if I don't get the handling of cyclical graphs quite right to start with, we could end up with some ugly stack traces initially. I don't think it'll take long to work those issues out, but it's my primary concern with the code I'm working with right now. This last PR for 4.5.0 should be ready to go either tomorrow or Monday.

[kalefranz]

I could not reproduce @nehaljwani's issue in a docker container, and then realized I was using python 3 instead of python 2 for the conda interpreter. While using conda 4.4.11, I switched to python 2, and with PYTHONHASHSEED=1 I got all numpys pinned to "numpy >=1.9.3,<2.0a0", and with PYTHONHASHSEED=2 I got all numpys pined to "numpy >=1.14.1,<2.0a0".

Then I updated conda to 4.5.0rc0. With PYTHONHASHSEED=[1-5], I got "numpy >=1.9.3,<2.0a0" in all cases.

[msarahan]

Excellent, that looks fixed. Thanks for testing, Kale.

[nehaljwani]

I've tested this too. 4.5.0rc0 solves this.

[msarahan]

Thanks Nehal. I'm going to go ahead and close this.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!