julia: enable all tests + stricter pinning #172

ngam · 2022-01-05T23:33:21Z

This is an issue to track progress and discussion regarding the two points below.

We ought to conduct all tests available upstream (unless they run over 6 hours).
Based on the issues we faced with libunwind and now openlibm, we should consider more exact pinning going forward.

I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Environment (conda list):

$ conda list

Details about conda and system ( conda info ):

$ conda info

The text was updated successfully, but these errors were encountered:

ngam · 2022-01-05T23:50:01Z

@mkitti would this call suffice?

Base.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),
              exit_on_error=false, revise=false, [seed])

Run the Julia unit tests listed in tests, which can be either a string or an array of strings, using ncores processors. If exit_on_error is false, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, when exit_on_error == true. If revise is true, the Revise package is used to load any modifications to Base or to the standard libraries before running the tests. If a seed is provided via the keyword argument, it is used to seed the global RNG in the context where the tests are run; otherwise the seed is chosen randomly.

https://docs.julialang.org/en/v1/stdlib/Test/#Testing-Base-Julia

mkitti · 2022-01-06T01:10:47Z

We also need to enable the doctests, the network tests, Pkg tests, and LibGit2/online

https://github.com/JuliaLang/julia/blob/98b485e0641a2e6017ae91acc9460c668d0e811a/test/choosetests.jl#L167

mkitti · 2022-01-06T01:32:20Z

We probably should talk to someone before turning all of this on. Also, it might be best to manually trigger these tests. The impression I was given earlier was that we should not do all of this all the time.

ngam · 2022-01-06T01:40:35Z

We probably should talk to someone before turning all of this on. Also, it might be best to manually trigger these tests. The impression I was given earlier was that we should not do all of this all the time.

It depends on how long it will take --- if it takes the build time from 1 hour to 2 hours, I wouldn't worry about talking to anyone else about it. But if we are going to push basically to 6 hours, then obviously, we will want to talk to people about it. Generally speaking, my personal impression is that the governance here is pretty loose and as long as you're within reasonable limits and not testing all the time with rebuilds (like we are doing now because of the activity), it should be okay.

When I enabled all the base tests, it didn't take significantly longer fwiw. I am not sure about the other tests (I don't even know how I would go about activating them)

Tagging @isuruf for a quick word, if any.

mkitti · 2022-01-06T01:44:00Z

The tests used to time out: #44 (comment) but maybe it had something to do CircleCI and the tests not being verbose.

mkitti · 2022-01-06T01:58:28Z

Here is the PR and comment when most of the tests were disabled, which is how I found it several months ago.
#50 (comment)

Anyways, let's just try it and see what happens.

ngam · 2022-01-06T02:31:54Z

See this: c2fd0f9

with results here (using half the ncores): https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=432645&view=results

can;t open, but this is a hint:

Azure Pipelines / julia-feedstock (osx osx_64_)
cancelled 11 days ago in 1h 57m 15s

I cancelled the run basically after the scoresheet came out --- there were many failures.

Point is, only 2 hours.

mkitti · 2022-01-06T03:03:31Z

We should also add downstream testing.

@valeriupredoi, I see https://github.com/ESMValGroup/ESMValTool/blob/main/tests/integration/diagnostic.jl in the ESMValTool test suite. Are there any other Julia based test files?

mkitti · 2022-01-06T03:09:55Z

When we have to patch to disable tests, we should use @test_broken. This will tell us when the tests start working.

julia-feedstock/recipe/patches/0002-disable-testing-Baz.baz-1.patch

Line 19 in bd5133e

+ # @test Base.invokelatest(Baz.baz) == 1

valeriupredoi · 2022-01-06T12:04:23Z

Hi @mkitti - that diagnostic.jl is just a script that we run part of the test suite to see if a Julia diagnostic can be run by ESMValTool - don't think of it as a test diagnostic, we use the term "diagnostic" to refer to our climate data diagnostics, and not as a system tool. This gets run in a subprocess with julia diagnostic.jl so just like any other Julia script in an environment provided with a julia executable 👍 And no, I believe this is the only functional test we run with Julia 🍺

ngam · 2022-01-07T18:14:48Z

osx-64/julia-1.7.1-h132cb31_2.tar.bz2

arpack >=3.7.0,<3.7.1.0a0, 
curl >=7.80.0,<8.0a0, 
git, 
gmp >=6.2.1,<7.0a0, 
libcxx >=11.1.0, 
libgfortran 5.*, 
libgfortran5 >=9.3.0, 
libnghttp2 >=1.43.0,<2.0a0, 
libosxunwind, 
libssh2 >=1.10.0,<2.0a0, 
libutf8proc >=2.7.0,<3.0a0, 
libzlib >=1.2.11,<1.3.0a0, 
mpfr >=4.1.0,<5.0a0, 
openblas-ilp64, 
openlibm <0.8.0, 
p7zip, pcre2 >=10.37,<10.38.0a0, 
suitesparse >=5.10.1,<6.0a0, 
zlib >=1.2.11,<1.3.0a0

linux-64/julia-1.7.1-h989b2f6_2.tar.bz2

arpack >=3.7.0,<3.7.1.0a0, 
curl >=7.80.0,<8.0a0, 
git, 
gmp >=6.2.1,<7.0a0, 
libgcc-ng >=9.4.0, 
libgfortran-ng, 
libgfortran5 >=9.4.0, 
libgit2 >=1.3.0,<1.4.0a0, 
libnghttp2 >=1.43.0,<2.0a0, 
libssh2 >=1.10.0,<2.0a0, 
libstdcxx-ng >=9.4.0, 
libunwind >=1.5.0,<1.6.0a0, 
libutf8proc >=2.7.0,<3.0a0, 
libzlib >=1.2.11,<1.3.0a0, 
mbedtls, 
mpfr >=4.1.0,<5.0a0, 
openblas-ilp64, 
openlibm <0.8.0, 
p7zip, 
pcre2 >=10.37,<10.38.0a0, 
suitesparse >=5.10.1,<6.0a0, 
zlib >=1.2.11,<1.3.0a0

ngam · 2022-01-07T18:30:45Z

mbedtls constraints: JuliaLang/julia#43624 (comment)

mkitti · 2022-01-07T19:02:35Z

I'm not sure how well you can derive constraints from that issue. Julia 1.7.1 is hard pinned to 2.24. There is significant ABI breakage from between Mbed TLS 2.24 and 2.28.

JuliaPackaging/Yggdrasil#4180

ngam · 2022-01-07T19:19:34Z

I'm not sure how well you can derive constraints from that issue

Wasn't thinking of that issue as the source. I just wanted to note that if we are to constrain mbedtls, we will need to go through all these (and your CVE-linked issue as well). Just bookmarking.

ngam · 2022-01-07T19:35:40Z

FYI:

the global pins usually appear in here: https://github.com/conda-forge/julia-feedstock/blob/master/.ci_support/linux_64_.yaml

You can edit them in here: https://github.com/conda-forge/julia-feedstock/blob/master/recipe/conda_build_config.yaml

For example, editing conda_build_config.yaml by adding:

curl: 
  - 7
...
pin_run_as_build:
...
    curl:
      max_pin: 7.80

will result in linux_64_.yaml and osx_64_.yaml:

curl:
- '7'
...
pin_run_as_build:
...
  curl:
    max_pin: '7.80'

and so on.

We should this for global pins, basically just pinning to the latest release (so that it is tested by us) and we can just add max pins in meta for the rest. The full list is in a previous comment.

ngam · 2022-01-07T19:48:17Z

@mkitti, should we pursue this max pinning sooner rather than later? We simply cannot afford to wait for the global pinning efforts and we will likely have other problems pretty soon.

ngam · 2022-01-07T19:48:20Z

conda-forge/curl-feedstock#101 (comment)

mkitti · 2022-01-07T20:23:59Z

I'm tempted to pin the configurations that we actually test. We would then periodically, weekly perhaps, relax the pins, retest, and then repin before merging again.

For official builds, perhaps someone should package juliaup:
https://github.com/JuliaLang/juliaup
We could then provide the activate and deactivate scripts with that as well.

ngam · 2022-01-07T20:32:36Z

I'm tempted to pin the configurations that we actually test. We would then periodically, weekly perhaps, relax the pins, retest, and then repin before merging again.

I understand... but if you look above, you will see some pretty decent global pinning is going on. Generally, conda/mamba will resolve to the tested version or very near it. The only problem we faced thus far is when packages upgrade, and hence we may get all we need with a simple max pinning.

I still stand by my idea of rebuilding the exact same julia (or repackaging) so that's definitely something I am interested in pursuing (see #175)

mkitti · 2022-01-08T02:14:19Z

I think juliaup offers a mechanism for systematic repackaging.

ngam · 2022-01-09T02:38:57Z

It doesn't look like we are actually using the conda-forge suitesparse (among many other deps) in the build despite the explicit instruction... a small segment of the build:

make[5]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD/Lib'
make[4]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD/Lib'
make[3]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/CHOLMOD'
make[3]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL'
make[4]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[5]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
ar: creating libldl.a
a - ldl.o
a - ldll.o
make[5]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[4]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL/Lib'
make[3]: Leaving directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/LDL'
make[3]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU'
make[4]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU/Lib'
make[5]: Entering directory '$SRC_DIR/deps/scratch/SuiteSparse-5.10.1/KLU/Lib'
ar: creating libklu.a
a - klu_free_symbolic.o
a - klu_defaults.o
a - klu_analyze_given.o
a - klu_analyze.o
a - klu_memory.o
a - klu_l_free_symbolic.o
a - klu_l_defaults.o
a - klu_l_analyze_given.o
a - klu_l_analyze.o
a - klu_l_memory.o
a - klu_d.o
a - klu_d_kernel.o
a - klu_d_dump.o
a - klu_d_factor.o
a - klu_d_free_numeric.o
a - klu_d_solve.o
a - klu_d_scale.o
a - klu_d_refactor.o
a - klu_d_tsolve.o
a - klu_d_diagnostics.o
a - klu_d_sort.o
a - klu_d_extract.o
a - klu_z.o
a - klu_z_kernel.o
a - klu_z_dump.o
a - klu_z_factor.o

and it keeps going in and out of suitesparse and other deps --- not sure if it is rebuilding stuff or doing something different.

We need to investgiate this further. Below are some of the warnings at the end:

WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
...
WARNING (julia,lib/julia/sys.so): $RPATH/libjulia-internal.so.1 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
...
WARNING (julia): run-exports library package conda-forge::pcre2-10.37-h032f7d1_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libgit2-1.3.0-haabb1ae_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::libgfortran5-11.2.0-h5c6108e_11 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::arpack-3.7.0-hdefa2d7_2 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::mpfr-4.1.0-h9202a9a_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::p7zip-16.02-he1b5a44_1000 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::zlib-1.2.11-h36c2ea0_1013 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::gmp-6.2.1-h58526e2_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::mbedtls-3.1.0-h9c3ff4c_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libnghttp2-1.43.0-ha19adfc_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): run-exports library package conda-forge::libssh2-1.10.0-ha35d2d1_2 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (julia): dso library package conda-forge::openlibm-0.7.5-h7f98852_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

@isuruf + @beckermr could you briefly check this out and let us know if you think these warnings are worth following up?

ngam · 2022-01-09T02:42:41Z

Note for this one, WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, we actually intentionally patch libLLVM-13jl.so to libLLVM.so. So it could be simply we need to patch it somewhere else too to ensure it is actually all good.

mkitti · 2022-01-09T04:07:12Z

The option should be USE_SYSTEM_LIBSUITESPARSE

ngam · 2022-01-09T04:20:02Z

Are these the legit options or are there more? https://github.com/JuliaLang/julia/blob/cd81054bae301ccf3a0fe7e0d8f60f167f1203a0/Makefile#L161-L222

# public libraries, that are installed in $(prefix)/lib
JL_TARGETS := julia
ifeq ($(BUNDLE_DEBUG_LIBS),1)
JL_TARGETS += julia-debug
endif

# private libraries, that are installed in $(prefix)/lib/julia
JL_PRIVATE_LIBS-0 := libccalltest libllvmcalltest libjulia-internal libjulia-codegen
ifeq ($(BUNDLE_DEBUG_LIBS),1)
JL_PRIVATE_LIBS-0 += libjulia-internal-debug libjulia-codegen-debug
endif
ifeq ($(USE_GPL_LIBS), 1)
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBSUITESPARSE) += libamd libbtf libcamd libccolamd libcholmod libcolamd libklu libldl librbio libspqr libsuitesparseconfig libumfpack
endif
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBBLASTRAMPOLINE) += libblastrampoline
JL_PRIVATE_LIBS-$(USE_SYSTEM_PCRE) += libpcre2-8
JL_PRIVATE_LIBS-$(USE_SYSTEM_DSFMT) += libdSFMT
JL_PRIVATE_LIBS-$(USE_SYSTEM_GMP) += libgmp libgmpxx
JL_PRIVATE_LIBS-$(USE_SYSTEM_MPFR) += libmpfr
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBSSH2) += libssh2
JL_PRIVATE_LIBS-$(USE_SYSTEM_NGHTTP2) += libnghttp2
JL_PRIVATE_LIBS-$(USE_SYSTEM_MBEDTLS) += libmbedtls libmbedcrypto libmbedx509
JL_PRIVATE_LIBS-$(USE_SYSTEM_CURL) += libcurl
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBGIT2) += libgit2
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBUV) += libuv
ifeq ($(OS),WINNT)
JL_PRIVATE_LIBS-$(USE_SYSTEM_ZLIB) += zlib
else
JL_PRIVATE_LIBS-$(USE_SYSTEM_ZLIB) += libz
endif
ifeq ($(USE_LLVM_SHLIB),1)
JL_PRIVATE_LIBS-$(USE_SYSTEM_LLVM) += libLLVM libLLVM-13jl
endif
JL_PRIVATE_LIBS-$(USE_SYSTEM_LIBUNWIND) += libunwind

ifeq ($(USE_SYSTEM_LIBM),0)
JL_PRIVATE_LIBS-$(USE_SYSTEM_OPENLIBM) += libopenlibm
endif

JL_PRIVATE_LIBS-$(USE_SYSTEM_BLAS) += $(LIBBLASNAME)
ifneq ($(LIBLAPACKNAME),$(LIBBLASNAME))
JL_PRIVATE_LIBS-$(USE_SYSTEM_LAPACK) += $(LIBLAPACKNAME)
endif

JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libgfortran libquadmath libstdc++ libgcc_s libgomp libssp libatomic
ifeq ($(OS),Darwin)
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libc++
endif
ifeq ($(OS),WINNT)
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libwinpthread
else
JL_PRIVATE_LIBS-$(USE_SYSTEM_CSL) += libpthread
endif


ifeq ($(OS),Darwin)
ifeq ($(USE_SYSTEM_BLAS),1)
ifeq ($(USE_SYSTEM_LAPACK),0)
JL_PRIVATE_LIBS-0 += libgfortblas
endif
endif
endif

ngam · 2022-01-09T04:21:06Z

Any idea why we aren't using our own libuv? I understand we have longstanding issues iwth libgit2 and mbedtls, and llvm is a hot mess. But libuv --- what's going on there? Also USE_SYSTEM_CSL: https://github.com/JuliaLang/julia/issues?q=USE_SYSTEM_CSL%3D1

export EXTRA_MAKEFLAGS="" 
if [[ "${target_platform}" == osx-* ]]; then
    export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=0 USE_SYSTEM_MBEDTLS=0"
elif [[ "${target_platform}" == linux-* ]]; then
    export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_MBEDTLS=1"
fi

...

make -j${CPU_COUNT} prefix=${PREFIX} sysconfigdir=${PREFIX}/etc \
 LIBBLAS=-lopenblas64_ LIBBLASNAME=libopenblas64_ LIBLAPACK=-lopenblas64_ LIBLAPACKNAME=libopenblas64_ \
 USE_SYSTEM_ARPACK=1 \
 USE_SYSTEM_BLAS=1 \
 USE_BLAS64=1 \
 USE_SYSTEM_CURL=1 \
 USE_SYSTEM_GMP=1 \
 USE_SYSTEM_LAPACK=1 \
 USE_SYSTEM_LIBSSH2=1 \
 USE_SYSTEM_LLVM=0 \
 USE_SYSTEM_MPFR=1 \
 USE_SYSTEM_OPENLIBM=1 \
 USE_SYSTEM_PATCHELF=1 \
 USE_SYSTEM_PCRE=1 \
 USE_SYSTEM_SUITESPARSE=1 \
 USE_SYSTEM_CSL=0 \
 USE_SYSTEM_LIBUNWIND=1 \
 USE_SYSTEM_LIBUV=0 \
 USE_SYSTEM_UTF8PROC=1 \
 USE_SYSTEM_NGHTTP2=1 \
 USE_SYSTEM_ZLIB=1 \
 USE_SYSTEM_P7ZIP=1 \
 ${EXTRA_MAKEFLAGS} \
 TAGGED_RELEASE_BANNER="https://github.com/conda-forge/julia-feedstock" \
 CC=$CC CXX=$CXX FC=$FC \
 install

ngam · 2022-01-09T04:22:07Z

Note for this one, WARNING (julia,lib/julia/libllvmcalltest.so): $RPATH/libLLVM-13jl.so not found in packages, we actually intentionally patch libLLVM-13jl.so to libLLVM.so. So it could be simply we need to patch it somewhere else too to ensure it is actually all good.

hmmmmm: JL_PRIVATE_LIBS-$(USE_SYSTEM_LLVM) += libLLVM libLLVM-13jl

ngam · 2022-01-12T15:31:37Z

One caveat is: some people want this sharing, e.g. #164 (comment)

ngam · 2022-01-12T15:37:13Z

Also, see my clueless discussion with keno and his responses here. Keno makes a decent argument why we should NOT pursue this stuff. JuliaLang/julia#43666

fingolfin · 2022-01-13T08:57:25Z

And I always thought of conda-forge as redistributing mechanism more than anything --- we are really messing up with that philosophy here. We are essentially trying to fork julia and do a julia-cf here so to speak.

+1 to that -- this is exactly the impression I have. Personally, I don't like the idea of a fork much (sounds like lots of duplicated work and also a potentially inferior experience for Julia users due to breakage that might crop up only in julia-cf). That said, if you decide "to go this way, [...] we can disagree as much as we want [...], but that's what [you] decided to do." ;-). I think it then helps to be clear and transparent about this, to avoid confusion for end users. This will also go a long way towards retaining a healthy, friendly and productive atmosphere between Conda and Julia developers.

BTW as I said, I do see why one may desire to use the exact same libgmp/libsundials/... etc. binaries in Conda and Julia (and so on...), I've been in a similar situation before: Because if one links Julia/Python/R/... code together and ends up loading two different copies of e.g. libgmp into a single process, this is a recipe for disaster. (Some people will mention disk space savings, but honestly I don't think that matters much this day and age, but I'll acknowledge that some people think it matters for them.)

However, in practice, this doesn't really affect meexactly because anything I need to ccall to is packaged in a JLL. But of course I am sure there are folks who want (need) to go beyond this, and that's reasonable to a degree. But the reality is that this is simply not possible for everything. To make an exotic example: the GAP computer algebra system (of which I am a core developer) is packaged for both Conda and Julia (as a JLL); but the two versions of libgap are not interchangeable, because they are built with different memory manager systems (the one in Julia uses the Julia GC). Of course this is far from being the norm for JLLs, I just point this out to raise awareness of this situation. There are other JLLs that carry custom Julia patches -- e.g. GMP (!).

Interestingly, this doesn't seem to be the reason @zklaus gives in #164 (comment) -- at least the way I understand the discussion there (please correct me if I misrepresent it; it definitely isn't intentional!), there isn't a single process which mixes e.g. Julia and Python code. Rather, the concern is about the same data being processed by different code, and a desire for the data processing to be consistent. There, netcdf is mention, so presumably the concern is that using two different netcdf versions may result in different "interpretations" of the same data and subsequently inconsistencies in the analysis. Ironically, this sounds very similar to my concerns that replacing a library in a JLL by another version could introduced precisely such issues for Julia packages that rely on fixing a specific version of that JLL...

All in all, I still think that trying to override the Julia JLLs on a large scale is a loosing battle: too much work for you, too much breakage for users, too little to gain. But of course in the end its your call to make! I am just trying to show some perspective "from the other side" :-)

zklaus · 2022-01-13T16:33:40Z

As a general rule in life, I've learned the hard way that it's usually a bad idea to assume (let alone voice in public) that someone else made "poor (or lazy) design choice" if I don't understand the situation fully yet.

Good advice indeed, even when the situation is fully understood, imho.

That's how we end up with NIH syndrom, "let's rewrite everything, the guys before us clearly were stupid", etc. ;-)

Ironically, NIH seems to be at the heart of Julia's packaging system. That is at least the only rationale I can see for why it doesn't make many references to nor accommodations for the decades of experience in packaging in Linux, Python's Pypi, Virtualenv or any number of other projects.

Of course, that is perfectly valid and one way to innovate. I personally think vendoring a lot of binaries is not a good idea, but this seems to be really important for many of the people in the Julia space that know and care about it. For me personally, that means Julia is not a viable development platform.

giordano · 2022-01-13T16:43:12Z

That is at least the only rationale I can see for why it doesn't make many references to nor accommodations for the decades of experience in packaging in Linux, Python's Pypi, Virtualenv or any number of other projects.

You may have totally missed Conda.jl and BinDeps.jl, which do exactly that and had been developed way before this insanity with JLLs.

Sadly, users and developers had plenty of problems with installing binaries from Linux distributions (root rights not always available) or Conda (I think it didn't support all platforms needed by Julia, e.g. FreeBSD, but maybe the situation improved today)

zklaus · 2022-01-13T17:23:06Z

Thanks for you comments, @giordano. I was loosely aware of Conda.jl, though not of BinDeps.jl. I mostly focused on JLLs because that seems to be the method preferred by the people advancing the package in this repo and the Julia parts of conda-forge.

But my comment was also an expression of disappointment about the lack of discussion of these aspects in the Pkg documentation; basically the prior work and state-of-the-art discussion one would expect at the beginning of a scientific paper.

Anyways, you seem to suggest that JLLs might not be the best way to add binary packages and Julia packages with binary dependencies to conda-forge. Perhaps also CondaBinDeps.jl should be taken into account.

Would you be interested to contribute a comment on #14 or #161/#164?

giordano · 2022-01-13T17:30:39Z

Anyways, you seem to suggest that JLLs might not be the best way to add binary packages and Julia packages with binary dependencies to conda-forge. Perhaps also CondaBinDeps.jl should be taken into account.

Did I? I merely pointed out the prior art, not the current state of affairs. Anything related to BinDeps.jl is completely unmaintained and deprecated.

zklaus · 2022-01-13T17:44:18Z

Oh sorry, I didn't want to put words in your mouth. I took this from you bringing up those projects and referring to JLLs as "insanity". But I guess then you agree that at this point in time the method to deal with binary dependencies in Julia is JLLs?

giordano · 2022-01-13T17:55:51Z

I took this from you bringing up those projects and referring to JLLs as "insanity".

Sorry, that was a tongue in cheek comment 🙂

But I guess then you agree that at this point in time the method to deal with binary dependencies in Julia is JLLs?

Yes, this is the method currently being used by the vast majority of packages in the Julia ecosystem.

mkitti · 2022-01-13T18:30:38Z

There's several levels of discussion here, and a lot of misunderstanding. For the most part we are talking about the management of binary dependencies rather than the distribution of Julia code.

Pkg.jl mostly addresses the distribution of Julia code, including the versioning of packages. There was a component of it, Artifacts, that did deal with the automatically downloading and management of tarballs, but as of Julia 1.6 that's been separated as a distinct standard library. Much of it's network based functionality can be replicated by cloning git repositories. There is not much intrinsic to Pkg.jl that dictates that you must use BinaryBuilder. In fact, one could use the build step to download source and build it:
https://pkgdocs.julialang.org/v1/creating-packages/#Adding-a-build-step-to-the-package
In fact, it is possible to bypass Pkg.jl entirely by just manipulating the Julia load path:
https://docs.julialang.org/en/v1/manual/code-loading/#code-loading

As Mose, pointed out above there were several iterations of how to address binary dependencies, but this led to a rather inconsistent experience.

The JLL packages separate the interface to binaries into a distinct package. All the JLL packages do is point to the location of the binaries and loads them. In many cases the Julia packages which depend on them, just depend on the paths. While the JLL packages themselves are used by BinaryBuilder, they do not have to involve BinaryBuilder. They can literally point to binaries by defining a few paths. For example, see my mock package here, which implements the minimal interface needed by Cxx.jl:
https://github.com/mkitti/libcxxwrap_julia_jll_mock/blob/cf/src/libcxxwrap_julia_jll.jl

BinaryBuilder automates the creation of JLL packages from the build recipe source tree, https://github.com/JuliaPackaging/Yggdrasil, and their associated artifacts using a cross compilation framework.

It is perfectly possible for conda-forge to create their own JLL packages if needed. There's potential here for a mess if done without coordination, but it is possible. Another way would be to increase the configurability of the existing JLL packages. Most of the JLLs created by BinaryBuilder rely on a single package, JLLWrappers.jl, that defines a template. This template has a common override mechanism:
https://github.com/JuliaPackaging/JLLWrappers.jl/blob/79226fad1b220ba7a3516a66e76922eb84f293ae/src/wrapper_generators.jl#L8-L19

As it is right now, one might be able to use that override mechanism by creating a symbolic link from an overrides directory in the artifact to $CONDA_PREFIX. At the moment, this still results in the download of the BinaryBuilder artifact, but that is something that could change. For example, there has been discussion of making the JLL packages using JLLWrappers.jl more configurable via a project (environment) specific preferences system: https://github.com/JuliaPackaging/Preferences.jl.

To summarize, there a several options available to conda-forge:

Fork and create an alternate JLLWrappers.jl with the same UUID as the BinaryBuilder one.

As I mentioned above, this does allow a global override for the packages created by BinaryBuilder, but this is probably a bad idea. This would force packages to look for artifacts in $CONDA_PREFIX even if the binaries are not installed by conda-forge.

Create alternate JLL packages. These could be delivered to the user completely via conda if desired.

This is much more targeted to specific packages. For example, with Sundials_jll one could create a fork of https://github.com/JuliaBinaryWrappers/Sundials_jll.jl with the same UUID. This package would contain an intrinsic override to point the artifact directory to $CONDA_PREFIX. I would imagine this would be installed by conda and have a dependency on the sundials package in conda-forge such that the binaries are guaranteed to exist.

This could be supported by another template package, CondaForgeJLLWrappers.jl, that has a separate UUID from JLLWrappers.jl. In theory, one could convert a BinaryBuilder JLL package to a conda-forge based one by changing a two lines.

This is a decent intermediate solution that can be implemented at the moment. This also could be implemented in a fashion that is not specific to the conda-forge distribution of Julia such that it can be used by users of the official Julia binaries. It is however difficult to scale this, but it is on the same order of effort as the R packages in conda-forge. In fact it's easier since there is usually a clear separation of the pure-Julia package and its corresponding binary dependency interface.

Get involved with the addition of a Preferences.jl based way to extend the overrides mechanism of a BinaryBuilder JLLWrappers.jl based JLL packages. Perhaps this could include a mechanism to check a global override directory, such as $CONDA_PREFIX, for existing binaries as part of a "build" step before activating the Artifacts system to download BinaryBuilder provided binaries.

ngam · 2022-01-13T18:30:41Z

Thank you all, @zklaus + @giordano + @fingolfin, this is extremely important discussion. Just one point:

But of course in the end its your call to make! I am just trying to show some perspective "from the other side" :-)

HELL NO! We really want to accommodate here, and we want people like you all to be making the calls (or helping us make the correct calls). In fact, I am more than happy to add you all as maintainers of this repo so that you have equal rights (simply merge/push rights) as the rest of maintainers here in deciding what we do.

Also, @mkitti is the vastly more knowledgeable person of the julia stuff here. I am just an enthusiastic beginner :)

mkitti · 2022-01-13T19:19:23Z

While they are more than welcome to join, I suspect you'll have to make do with me. Mosè Giordano almost single-handedly runs BinaryBuilder and Yggdrasil from day-to-day by moonlighting, so he's quite preoccupied with that. I suspect he would greatly appreciate if we do not make a mess out of the JLLs.

https://www.youtube.com/watch?v=S__x3K31qnE

We are essentially trying to fork julia and do a julia-cf here so to speak

I think this is avoidable. While not all package configurations will be completely compatible with existing binaries, we should be able to get close and reduce the difference over time by making sure the dependencies are available at the same versions and synchronizing necessary patches.

If we do this properly, we should be able to expand the number of JLLs available. In some cases, we are already pulling conda-forge binaries when cross-compiling:
https://github.com/JuliaBinaryWrappers/HDF5_jll.jl

zklaus · 2022-01-13T20:45:09Z

Let me see if I get this right. For the typical case we are talking about, there is the following chain of dependencies on the Julia side:

UserPackage -> JuliaPackage -> JLLPackage -> binary library

where UserPackage is some code that wants to use the library, JuliaPackage is essentially the Julia binding for the library, JLLPackage is an almost empty package in the Julia space that finds out or knows how to load the actual binary library. In principle, the binary library could come from anywhere, but in practice, it is usually downloaded by the JLL as a pre-built artifact from the release artifacts of the JLL on github.

What we would want to do for conda-forge, is having the binary library be provided by the (often already existing) conda-forge package. That means we need to provide a JLL package that replaces the one from JuliaBinaryWrappers but has the same UUID and points to the conda-forge binary library.
In terms of conda-forge packages, I would certainly expect one conda-forge package each for UserPackage, JuliaPackage and binary library(usually already present).
The question is where to fit in JLLPackage. Intuitively, there are three places (numbering only for future reference, no ordering or preference implied):

Integrate it into JuliaPackage
Integrate it into binary library
Package it as a separate package

Option 1 is nice because it keeps the number of packages down and makes sure that JuliaPackage comes complete. The challenge with this approach is that the version number of binary library must be integrated into the package which in practice might mean very hard pinning. That's not nice, but perhaps doable; it would imply re-building JuliaPackage everytime a new version of binary library comes out, something that conda-forge's migrators might help with.

Option 2 is nice because it would provide a relatively small overhead on part of the binary library and make sure that the version numbers are always in sync. JuliaPackage can be more flexible in its pinning in this scenario.
The challenge will be firstly to convince binary library's conda-forge feedstock to ship Julia infrastructure, and secondly to deal with the JLL depending on the version of Julia.

Option 3 gives the most freedom, but also creates the most overhead in terms of packages. Even with this option, I don't think we need to fork the JLLs, or at least not separately from the conda-forge feedstocks. What little code might be needed outside of the upstream JLL repository can comfortably live within the feedstock repo.

In any case, it would probably be best to start by building a few pure Julia packages that have no binary dependencies at all. I feel like I am largely responsible for dragging us away from that and into these binary weeds, so apologies for that.

Do you have a few (lets say 3) suggestions for pure Julia packages that would be nice, easy, and useful to start with?

ngam · 2022-01-13T21:01:37Z

What we would want to do for conda-forge, is having the binary library be provided by the (often already existing) conda-forge package. That means we need to provide a JLL package that replaces the one from JuliaBinaryWrappers but has the same UUID and points to the conda-forge binary library.

Just a side note on this for reference: If we move to do this wholesale, we are likely going to make any conda env with julia in it unusable due to the strict/exact dependencies required by julia. In other words, there is an implicit assumption here that a conda env with julia in it only has julia, in which case, that's fine, but that again takes us back to: Why not just repackage the damn thing? I know your specific case (a package that aims to utilize different backends smoothly) will benefit from this, but it may eventually break the other backends... or do you not anticipate such an issue?

zklaus · 2022-01-13T21:05:24Z

[...] If we move to do this wholesale, we are likely going to make any conda env with julia in it unusable due to the strict/exact dependencies required by julia. [...]

This is not necessarily the case. Yes, every build of the conda-forge package that contains the JLL package will depend strictly on one version of the binary library, but there may be many builds available that accrue over time, giving us back the freedom to install different versions of the binary library---each with its corresponding JLL providing package.

mkitti · 2022-01-13T23:10:59Z

I think @SylvainCorlay is lining up a test case here via Xtensor.jl as discussed in JuliaInterop/CxxWrap.jl#309, and there's a chance here for a bidirectional exchange.

Xtensor.jl depends on CxxWrap.jl. and xtensor-julia (Julia bindings for the C++ library)
CxxWrap.jl depends on libcxxwrap_julia_jll.jl (currently in BinaryBuilder)
xtensor-julia depends on xtensor

Currently, libcxxwrap_julia_jll.jl will grab artifacts from BinaryBuilder. However, they would like to use binaries installed by https://github.com/conda-forge/libcxxwrap-julia-feedstock .

In this case, we need an alternate version of libcxxwrap_julia_jll.jl for conda-forge. My proof-of-concept is
https://github.com/mkitti/libcxxwrap_julia_jll_mock/tree/cf
(This needs to be generalized beyond Linux and expanded to cover the entire JLL interface)

In this case, we have a Julia specific binary package, https://github.com/conda-forge/libcxxwrap-julia-feedstock . In this case, I think we could embed the alternate conda-forge specific JLL directly into the feedstock as a subdirectory package. The subdirectory package could either be installed directly from the Github repository or from a local path installed by the conda package. I imagine, this would occur via post-link.sh: julia -e 'import Pkg; Pkg.add(url="https://github.com/conda-forge/libcxxwrap-julia-feedstock/libcxxwrap_julia_jll.jl")'

As long as we install the conda-forge version of libcxxwrap_julia_jll.jl before CxxWrap.jl, then the BinaryBuilder artifacts are never accessed. CxxWrap.jl's Project.toml is happy because there is a package called libcxxwrap_julia_jll installed with UUID 3eaa8342-bff7-56a5-9981-c04077f7cee7 within the version bounds specified [0.9, 1.0). The conda-forge libcxxwrap_julia_jll points to libraries within $CONDA_PREFIX.

Perhaps later we make a Xtensor_jll.jl package and perhaps there end up being conda-forge and BinaryBuilder options for that package as well.

While Julia packages can be completely installed via git protocol, often the use of a registry and package server can make installation easier. Julia's registries are roughly analogous to conda's channels. Discussions on the Julia side have yielded a proposal for conda-forge based registry. That said, there are also ways to deliver Julia packages via conda or mamba.

valeriupredoi · 2022-01-14T12:50:42Z

hey guys, this is a terrific discussion, and am sorry I've not participated much to it - I'll leave it to the Julia specialists and enthusiast users (like @zklaus ) - have to admit, I am neither 😁 One idea about testing if the whole ecosystem works: would nightly/weekly/fortnightly (you'd have to decide on the frequency) conda lock file generation and verification (the conda lock file that gets generated is used to create the env and install Julia, and test it) be an option to detect dependency variability and breakages? You can even set an automatic PR that updates the lock file everytime there is a change in deps and that works well, then chain pipe that to a new automatic build of the Julia conda package/container. Anyways, keep the good discussion up, let me know if you want me to test anything from the package user point! Good weekend 🍺

mkitti · 2022-01-14T13:27:20Z

That sounds interesting @valeriupredoi and more on topic than the recent discussions. Could you elaborate on how one may do that?

valeriupredoi · 2022-01-17T15:37:01Z

hey @mkitti apols for the tardy reply, had to figure out a few things on the condalock/auto PR stuff at my end, but I have had it fully implemented and working now: have a look at this PR ESMValGroup/ESMValCore#1164 and GA workflow that generates a conda lock file, and then uses it to create an env, install a package (our package), test it, then generate an auto (bot) PR if the lock file has changed during (re-)generation https://github.com/ESMValGroup/ESMValCore/pull/1164/files - you may have seen these sort of tools before, not sure, sorry if it's a road you've already driven down. The good thing about the lock file is that it can be generated once and then used nightly, or weekly, or whatever frequency you chose to run it as a cron job, so you can test your recipe YAML file this way (and the pins inside the file), then install Julia in that env and see if it works - run your preferred test suite, if things go south you can always turn the automated generation on (like we have it) as a cron job, and again you test the newly created file and env, and if all goes fine you have an auto PR that updates your lock file, and from there the conda recipe. What do you think?

mkitti · 2022-01-17T22:56:57Z

That's interesting. I'm not sure I completely understand it. It seems like Julia's Manifest.toml perhaps? Maybe @ngam has a better idea.

ngam · 2022-01-18T00:01:36Z

Yes, pretty similar. A conda lockfile is basically ~~to ensure~~ something like what julia does by default with all these crazy exact deps 😆

You can try it locally by

ngam@mp15:~$ mamba create -n lock_julia julia conda-lock -c conda-forge/label/julia_dev -c conda-forge

Then you will see stuff like

(lock_julia) ngam@mp15:~$ mamba list
# packages in environment at /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/lock_julia:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
arpack                    3.7.0                hefb7bc6_2    conda-forge
...
julia                     1.8.0.dev.1328       h8742b08_1    conda-forge/label/julia_dev
...

(lock_julia) ngam@mp15:~$ mamba env export > lock_julia.env
(lock_julia) ngam@mp15:~$ cat lock_julia.env 
name: lock_julia
channels:
  - conda-forge/label/julia_dev
  - conda-forge
dependencies:
  - appdirs=1.4.4=pyh9f0ad1d_0
  - arpack=3.7.0=hefb7bc6_2
  - brotlipy=0.7.0=py310he24745e_1003
  - bzip2=1.0.8=h0d85af4_4
  - c-ares=1.18.1=h0d85af4_0
  - ca-certificates=2021.10.8=h033912b_0
  - certifi=2021.10.8=py310h2ec42d9_1
  - cffi=1.15.0=py310hcc37b68_0
  - charset-normalizer=2.0.10=pyhd8ed1ab_0
  - click=8.0.3=py310h2ec42d9_1
  - click-default-group=1.2.2=pyhd8ed1ab_1
  - conda-lock=0.13.2=pyhd8ed1ab_0
  - cryptography=36.0.1=py310ha82f1d4_0
  - curl=7.81.0=hf45b732_0
  - ensureconda=1.4.1=pyhd8ed1ab_0
  - expat=2.4.2=he49afe7_0
  - filelock=3.4.2=pyhd8ed1ab_1
  - gettext=0.19.8.1=hd1a6beb_1008
  - git=2.34.1=pl5321h9a53687_0
  - gmp=6.2.1=h2e338ed_0
  - idna=3.3=pyhd8ed1ab_0
  - jinja2=3.0.3=pyhd8ed1ab_0
  - julia=1.8.0.dev.1328=h8742b08_1
  - krb5=1.19.2=hcfbf3a7_3
  - libblas=3.9.0=12_osx64_openblas
  - libcblas=3.9.0=12_osx64_openblas
  - libcurl=7.81.0=hf45b732_0
  - libcxx=12.0.1=habf9029_1
  - libedit=3.1.20191231=h0678c8f_2
  - libev=4.33=haf1e3a3_1
  - libffi=3.4.2=h0d85af4_5
  - libgfortran=5.0.0=9_3_0_h6c81a4c_23
  - libgfortran5=9.3.0=h6c81a4c_23
  - libiconv=1.16=haf1e3a3_0
  - liblapack=3.9.0=12_osx64_openblas
  - libnghttp2=1.43.0=h6f36284_1
  - libopenblas=0.3.18=openmp_h3351f45_0
  - libopenblas-ilp64=0.3.18=openmp_h470725b_0
  - libosxunwind=0.0.6=h940c156_0
  - libssh2=1.10.0=h52ee1ee_2
  - libutf8proc=2.7.0=h0d85af4_0
  - libzlib=1.2.11=h9173be1_1013
  - llvm-openmp=12.0.1=hda6cdc1_1
  - markupsafe=2.0.1=py310he24745e_1
  - metis=5.1.0=h2e338ed_1006
  - mpfr=4.1.0=h0f52abe_1
  - ncurses=6.2=h2e338ed_4
  - openblas-ilp64=0.3.18=openmp_ha601604_0
  - openlibm=0.7.5=h0d85af4_0
  - openssl=1.1.1l=h0d85af4_0
  - p7zip=16.02=he49afe7_1000
  - pcre2=10.37=ha16e1b2_0
  - perl=5.32.1=1_h0d85af4_perl5
  - pip=21.3.1=pyhd8ed1ab_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pydantic=1.9.0=py310he24745e_0
  - pyopenssl=21.0.0=pyhd8ed1ab_0
  - pysocks=1.7.1=py310h2ec42d9_4
  - python=3.10.2=h1248fe1_0_cpython
  - python_abi=3.10=2_cp310
  - pyyaml=6.0=py310he24745e_3
  - readline=8.1=h05e3726_0
  - requests=2.27.1=pyhd8ed1ab_0
  - setuptools=60.5.0=py310h2ec42d9_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.37.0=h23a322b_0
  - suitesparse=5.10.1=h7aff33d_1
  - tbb=2021.5.0=h940c156_0
  - tk=8.6.11=h5dbffcc_1
  - toml=0.10.2=pyhd8ed1ab_0
  - typing-extensions=4.0.1=hd8ed1ab_0
  - typing_extensions=4.0.1=pyha770c72_0
  - tzdata=2021e=he74cb21_0
  - urllib3=1.26.8=pyhd8ed1ab_1
  - wheel=0.37.1=pyhd8ed1ab_0
  - xz=5.2.5=haf1e3a3_1
  - yaml=0.2.5=h0d85af4_2
  - zlib=1.2.11=h9173be1_1013
prefix: /Users/ngam/.Mambaforge-MacOSX-x86_64/envs/lock_julia

(lock_julia) ngam@mp15:~$ conda-lock lock -f lock_julia.env --platform osx-64 --mamba
Generating lockfile(s) for osx-64...
 - Install lock using : conda create --name YOURENV --file conda-osx-64.lock
(lock_julia) ngam@mp15:~$ cat conda-osx-64.lock 
# Generated by conda-lock.
# platform: osx-64
# input_hash: c4f6f32fd370260278b07d479f2cf54aabfbf99b9c64ea1dc35efb810bbb4a0a
@EXPLICIT
https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.8-h0d85af4_4.tar.bz2#37edc4e6304ca87316e160f5ca0bd1b5
https://conda.anaconda.org/conda-forge/osx-64/c-ares-1.18.1-h0d85af4_0.tar.bz2#00b3e98a61e6430808fe7a2534681f28
https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2021.10.8-h033912b_0.tar.bz2#bb82d0243db9882b509702ecb69e38f0
https://conda.anaconda.org/conda-forge/osx-64/libcxx-12.0.1-habf9029_1.tar.bz2#444dfb8ed0bf2a8864f73eb6436b29e0
https://conda.anaconda.org/conda-forge/osx-64/libev-4.33-haf1e3a3_1.tar.bz2#79dc2be110b2a3d1e97ec21f691c50ad
https://conda.anaconda.org/conda-forge/osx-64/libffi-3.4.2-h0d85af4_5.tar.bz2#ccb34fb14960ad8b125962d3d79b31a9
https://conda.anaconda.org/conda-forge/osx-64/libiconv-1.16-haf1e3a3_0.tar.bz2#c5fab167412a52e491c8e11453ae016f
https://conda.anaconda.org/conda-forge/osx-64/libutf8proc-2.7.0-h0d85af4_0.tar.bz2#d11cd97b8c35f9c803a1f45491d373a5
https://conda.anaconda.org/conda-forge/osx-64/libzlib-1.2.11-h9173be1_1013.tar.bz2#a3a6a53beaa92c5cfe52ee3a198e78cc
https://conda.anaconda.org/conda-forge/osx-64/llvm-openmp-12.0.1-hda6cdc1_1.tar.bz2#35b1fd6ed9f0f1c68583dc9c9f672513
https://conda.anaconda.org/conda-forge/osx-64/metis-5.1.0-h2e338ed_1006.tar.bz2#ed90f7784864e7c0580624ec3ed5534e
https://conda.anaconda.org/conda-forge/osx-64/ncurses-6.2-h2e338ed_4.tar.bz2#9cef1910395d1543527583e73dba30f1
https://conda.anaconda.org/conda-forge/osx-64/openlibm-0.7.5-h0d85af4_0.tar.bz2#c6640fda983acc73ec8ea934a7e14fad
https://conda.anaconda.org/conda-forge/osx-64/perl-5.32.1-1_h0d85af4_perl5.tar.bz2#500b89467a589a52ba33a64095aea9c2
https://conda.anaconda.org/conda-forge/noarch/tzdata-2021e-he74cb21_0.tar.bz2#a751ec502589ebdc2eceb183ff602569
https://conda.anaconda.org/conda-forge/osx-64/xz-5.2.5-haf1e3a3_1.tar.bz2#41116deb499e9bc58048c297d6403ce6
https://conda.anaconda.org/conda-forge/osx-64/yaml-0.2.5-h0d85af4_2.tar.bz2#d7e08fcf8259d742156188e8762b4d20
https://conda.anaconda.org/conda-forge/osx-64/expat-2.4.2-he49afe7_0.tar.bz2#f2676d01827827c3ec42b7c1a0e53232
https://conda.anaconda.org/conda-forge/osx-64/gettext-0.19.8.1-hd1a6beb_1008.tar.bz2#28c370fc39becf486601d9e491a5e184
https://conda.anaconda.org/conda-forge/osx-64/gmp-6.2.1-h2e338ed_0.tar.bz2#dedc96914428dae572a39e69ee2a392f
https://conda.anaconda.org/conda-forge/osx-64/libedit-3.1.20191231-h0678c8f_2.tar.bz2#6016a8a1d0e63cac3de2c352cd40208b
https://conda.anaconda.org/conda-forge/osx-64/libgfortran5-9.3.0-h6c81a4c_23.tar.bz2#a6956ceb628b14594613cefee5127a7a
https://conda.anaconda.org/conda-forge/osx-64/libosxunwind-0.0.6-h940c156_0.tar.bz2#01662161c29ec05885b4d5a284345e21
https://conda.anaconda.org/conda-forge/osx-64/openssl-1.1.1l-h0d85af4_0.tar.bz2#f7bcdbb7a5dae97030eee20b884b1f8a
https://conda.anaconda.org/conda-forge/osx-64/p7zip-16.02-he49afe7_1000.tar.bz2#d51ed3d63d47f4c0c305c83d9990db5c
https://conda.anaconda.org/conda-forge/osx-64/readline-8.1-h05e3726_0.tar.bz2#2832e9b6a7caa7cb192fcda6cfcd8871
https://conda.anaconda.org/conda-forge/osx-64/tbb-2021.5.0-h940c156_0.tar.bz2#bee824901d92a242d12f7d192c297063
https://conda.anaconda.org/conda-forge/osx-64/zlib-1.2.11-h9173be1_1013.tar.bz2#cf985617d679990418c380099620b01a
https://conda.anaconda.org/conda-forge/osx-64/libgfortran-5.0.0-9_3_0_h6c81a4c_23.tar.bz2#60f48cef2d50674e0428c5579b6c3f66
https://conda.anaconda.org/conda-forge/osx-64/libnghttp2-1.43.0-h6f36284_1.tar.bz2#8489d081927ce544b21a64c9e1c254ee
https://conda.anaconda.org/conda-forge/osx-64/libssh2-1.10.0-h52ee1ee_2.tar.bz2#8c8f3804e8e252b47443cfe8e40eddf9
https://conda.anaconda.org/conda-forge/osx-64/mpfr-4.1.0-h0f52abe_1.tar.bz2#afe26b08c2d2265b4d663d199000e5da
https://conda.anaconda.org/conda-forge/osx-64/pcre2-10.37-ha16e1b2_0.tar.bz2#a3be43be1fcee194d1e82e5ca2ce47bc
https://conda.anaconda.org/conda-forge/osx-64/sqlite-3.37.0-h23a322b_0.tar.bz2#68f13bbe00ac288ffc1c1b868bc4da23
https://conda.anaconda.org/conda-forge/osx-64/tk-8.6.11-h5dbffcc_1.tar.bz2#e35b11155bd70facee39dfdec7368ad0
https://conda.anaconda.org/conda-forge/osx-64/krb5-1.19.2-hcfbf3a7_3.tar.bz2#22a613e8000ad9c0d8a14c5ca1e8e40b
https://conda.anaconda.org/conda-forge/osx-64/libopenblas-0.3.18-openmp_h3351f45_0.tar.bz2#a0636ed965924977a848fd0d0258cf17
https://conda.anaconda.org/conda-forge/osx-64/libopenblas-ilp64-0.3.18-openmp_h470725b_0.tar.bz2#269bdb874aec510b5cc82722e7278c18
https://conda.anaconda.org/conda-forge/osx-64/python-3.10.2-h1248fe1_0_cpython.tar.bz2#216f49028a895f6f33dd6a81489bb06f
https://conda.anaconda.org/conda-forge/noarch/appdirs-1.4.4-pyh9f0ad1d_0.tar.bz2#5f095bc6454094e96f146491fd03633b
https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-2.0.10-pyhd8ed1ab_0.tar.bz2#ea77236c8031cfa821720b21b4cb0ceb
https://conda.anaconda.org/conda-forge/noarch/filelock-3.4.2-pyhd8ed1ab_1.tar.bz2#d3f5797d3f9625c64860c93fc4359e64
https://conda.anaconda.org/conda-forge/noarch/idna-3.3-pyhd8ed1ab_0.tar.bz2#40b50b8b030f5f2f22085c062ed013dd
https://conda.anaconda.org/conda-forge/osx-64/libblas-3.9.0-12_osx64_openblas.tar.bz2#3d38a706f06849a4b08f8a3f642bec93
https://conda.anaconda.org/conda-forge/osx-64/libcurl-7.81.0-hf45b732_0.tar.bz2#d0396298c85013ee3b186cb982f9f0dd
https://conda.anaconda.org/conda-forge/osx-64/openblas-ilp64-0.3.18-openmp_ha601604_0.tar.bz2#caf1a724e272aa2fbd3c9fdd7ccf02ab
https://conda.anaconda.org/conda-forge/noarch/pycparser-2.21-pyhd8ed1ab_0.tar.bz2#076becd9e05608f8dc72757d5f3a91ff
https://conda.anaconda.org/conda-forge/osx-64/python_abi-3.10-2_cp310.tar.bz2#502ca4a8b43b424316f380d864832877
https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2#e5f25f8dbc060e9a8d912e432202afc2
https://conda.anaconda.org/conda-forge/noarch/toml-0.10.2-pyhd8ed1ab_0.tar.bz2#f832c45a477c78bebd107098db465095
https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.0.1-pyha770c72_0.tar.bz2#1fc03816925d3cb7fdab9ab234e7fea7
https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.tar.bz2#1ca02aaf78d9c70d9a81a3bed5752022
https://conda.anaconda.org/conda-forge/osx-64/certifi-2021.10.8-py310h2ec42d9_1.tar.bz2#5e78bd701d1fb88bb110859ac7278774
https://conda.anaconda.org/conda-forge/osx-64/cffi-1.15.0-py310hcc37b68_0.tar.bz2#bca69dbd12cea5ae0dc298fc58a3c4ee
https://conda.anaconda.org/conda-forge/osx-64/click-8.0.3-py310h2ec42d9_1.tar.bz2#bb30d1eaa4b1ff47986a9eb4d0f9e9dc
https://conda.anaconda.org/conda-forge/osx-64/curl-7.81.0-hf45b732_0.tar.bz2#11c453fee87eb4bd10ab92c8f5354a82
https://conda.anaconda.org/conda-forge/osx-64/libcblas-3.9.0-12_osx64_openblas.tar.bz2#0cb9afc4cfdc6e95aa38328fc1befa05
https://conda.anaconda.org/conda-forge/osx-64/liblapack-3.9.0-12_osx64_openblas.tar.bz2#ca35c8b85ced3a53059c7a001518cbae
https://conda.anaconda.org/conda-forge/osx-64/markupsafe-2.0.1-py310he24745e_1.tar.bz2#9ee2b4f56f8db5f65551bf5fe674861e
https://conda.anaconda.org/conda-forge/osx-64/pysocks-1.7.1-py310h2ec42d9_4.tar.bz2#9892cbd606e255650871f9c21f90aa20
https://conda.anaconda.org/conda-forge/osx-64/pyyaml-6.0-py310he24745e_3.tar.bz2#69c8931611aa6ce88602b3ed951ce545
https://conda.anaconda.org/conda-forge/osx-64/setuptools-60.5.0-py310h2ec42d9_0.tar.bz2#0c6533076466687781b1966abbd7f149
https://conda.anaconda.org/conda-forge/noarch/typing-extensions-4.0.1-hd8ed1ab_0.tar.bz2#c0d4ec4bcbceb927bff1103a997410d3
https://conda.anaconda.org/conda-forge/osx-64/arpack-3.7.0-hefb7bc6_2.tar.bz2#9d5a86841fa68d17f7129707af11136a
https://conda.anaconda.org/conda-forge/osx-64/brotlipy-0.7.0-py310he24745e_1003.tar.bz2#fd46e059e9a7dcb8b5a517b49813e72b
https://conda.anaconda.org/conda-forge/noarch/click-default-group-1.2.2-pyhd8ed1ab_1.tar.bz2#72a46ffc25701c173932fd55cf0965d3
https://conda.anaconda.org/conda-forge/osx-64/cryptography-36.0.1-py310ha82f1d4_0.tar.bz2#463fb2dda6729ce5be2b8113a4f47a92
https://conda.anaconda.org/conda-forge/osx-64/git-2.34.1-pl5321h9a53687_0.tar.bz2#cfb51c97c3967661be38d7238ab7b986
https://conda.anaconda.org/conda-forge/noarch/jinja2-3.0.3-pyhd8ed1ab_0.tar.bz2#036d872c653780cb26e797e2e2f61b4c
https://conda.anaconda.org/conda-forge/noarch/pip-21.3.1-pyhd8ed1ab_0.tar.bz2#e4fe2a9af78ff11f1aced7e62128c6a8
https://conda.anaconda.org/conda-forge/osx-64/pydantic-1.9.0-py310he24745e_0.tar.bz2#50b8b7c7b1f36d5a13205810e8a421a7
https://conda.anaconda.org/conda-forge/osx-64/suitesparse-5.10.1-h7aff33d_1.tar.bz2#d30185298e8ba70c8bc17121f837730d
https://conda.anaconda.org/conda-forge/label/julia_dev/osx-64/julia-1.8.0.dev.1328-h8742b08_1.tar.bz2#f1f876fee1d35e4c132b61b375b14c16
https://conda.anaconda.org/conda-forge/noarch/pyopenssl-21.0.0-pyhd8ed1ab_0.tar.bz2#8c49efecb7dca466e18b06015e8c88ce
https://conda.anaconda.org/conda-forge/noarch/urllib3-1.26.8-pyhd8ed1ab_1.tar.bz2#53f1387c68c21cecb386e2cde51b3f7c
https://conda.anaconda.org/conda-forge/noarch/requests-2.27.1-pyhd8ed1ab_0.tar.bz2#7c1c427246b057b8fa97200ecdb2ed62
https://conda.anaconda.org/conda-forge/noarch/ensureconda-1.4.1-pyhd8ed1ab_0.tar.bz2#52a7f7cc9076e2c8a25e15e19ad42821
https://conda.anaconda.org/conda-forge/noarch/conda-lock-0.13.2-pyhd8ed1ab_0.tar.bz2#4fa525b4257e4938eaeb2269fdf33d1f

I usually don't these types of things much in day-to-day stuff, but this could be helpful here if we were to automate some testing as @valeriupredoi helpfully explains.

fingolfin · 2022-01-18T00:18:17Z

@ngam not sure what you mean by

what julia does by default with all these crazy exact deps

actually a Manifest.toml is exactly a lockfile. It is not the primary list of dependencies (that is stored in Project.toml).

So I guess conda (and ruby and ...) lockfiles are also crazy?! :-)

ngam · 2022-01-18T00:20:01Z

So I guess conda (and ruby and ...) lockfiles are also crazy?! :-)

yes ;)

actually a Manifest.toml is exactly a lockfile. It is not the primary list of dependencies (that is stored in Project.toml).

Thanks for stepping in and explaining it!

ngam · 2022-01-18T00:21:46Z

I was actually referring to building julia, but I am all over the place --- sorry, couldn't help but throw a cheap hit at julia lol

ngam · 2022-01-18T00:26:01Z

Also, @mkitti you may wonder what the hell: mamba env export and conda-lock kinda do similar things, yep, I have no idea --- but usually, the mamba env export trick is an after-thought; rather, people usually start from a env.yaml file of sort. Anyway, I think a big advantage of the conda-lock utility is the updating, etc. (also as explained by @valeriupredoi)

ngam · 2022-01-18T00:27:49Z

(@fingolfin please fact-check me)

ngam · 2022-01-18T00:40:07Z

But overall --- let's pause for a second. Our primary concern isn't even this level of sophistication at this stage. Our problems came from straightforward breaking changes (openlibm, curl, etc.) and hopefully we are now better positioned to find these and resolve them more speedily with the dev branch activity. What we are trying to do here in this feedstock isn't to have a centralized feedstock (we don't want to repeat what julia did) and instead we should rely on downstream feedstocks to keep track of these things. Does that make sense? Some coordination is good, a lot of coordination and we end up recreating the julia syndrome (sorry @fingolfin lol) of reinventing the wheel all over the again

I don't think it would be reasonable for us to introduce a juliano bot. And we really have to rely on the global pinning for a lot of these deps anyway. Oh my daaaze, we really are far from the title of this issue now: "julia: enable all tests + stricter pinning" (nope not really gonna happen, we should do a julia-cf vs julia proper tho, that's my conclusion)

valeriupredoi · 2022-01-19T13:10:16Z

Also, @mkitti you may wonder what the hell: mamba env export and conda-lock kinda do similar things, yep, I have no idea --- but usually, the mamba env export trick is an after-thought; rather, people usually start from a env.yaml file of sort. Anyway, I think a big advantage of the conda-lock utility is the updating, etc. (also as explained by @valeriupredoi)

indeed! Here's a short overview that touches upon the difference between conda env export and conda-lock as well as upon pinning https://pythonspeed.com/articles/conda-dependency-management/ - note that there are a few issues with conda lock files, among them md5sum checks support etc. Anyways, let you guys discuss the matter at hand, let me know if you have any q's about other conda stuff, or if you'd like me to test things for you related to Julia, have some spare time on and off 👍

ngam · 2022-04-03T18:36:07Z

This was a good discussion! I am going to close this as we have accomplished the stuff in the title. If you disagree please open a new issue :)

See here where all tests passed in our CI: #200 (comment)

ngam assigned mkitti and ngam Jan 5, 2022

ngam mentioned this issue Jan 7, 2022

julia=1.7.1 currently needs openlibm<0.8.0 therefore latest conda env builds with deps from conda-forge fail to solve JuliaLang/julia#43666

Closed

ngam mentioned this issue Jan 8, 2022

update openlibm dep for julia versions conda-forge/conda-forge-repodata-patches-feedstock#208

Merged

5 tasks

ngam closed this as completed Apr 3, 2022

julia: enable all tests + stricter pinning #172

julia: enable all tests + stricter pinning #172

Comments

ngam commented Jan 5, 2022 • edited Loading

ngam commented Jan 5, 2022

mkitti commented Jan 6, 2022

mkitti commented Jan 6, 2022

ngam commented Jan 6, 2022

mkitti commented Jan 6, 2022 • edited Loading

mkitti commented Jan 6, 2022

ngam commented Jan 6, 2022 • edited Loading

mkitti commented Jan 6, 2022 • edited Loading

mkitti commented Jan 6, 2022

valeriupredoi commented Jan 6, 2022

ngam commented Jan 7, 2022

ngam commented Jan 7, 2022 • edited Loading

mkitti commented Jan 7, 2022

ngam commented Jan 7, 2022

ngam commented Jan 7, 2022 • edited Loading

ngam commented Jan 7, 2022

ngam commented Jan 7, 2022

mkitti commented Jan 7, 2022

ngam commented Jan 7, 2022 • edited Loading

mkitti commented Jan 8, 2022

ngam commented Jan 9, 2022 • edited Loading

ngam commented Jan 9, 2022

mkitti commented Jan 9, 2022

ngam commented Jan 9, 2022

ngam commented Jan 9, 2022 • edited Loading

ngam commented Jan 9, 2022

ngam commented Jan 12, 2022

ngam commented Jan 12, 2022

fingolfin commented Jan 13, 2022

zklaus commented Jan 13, 2022

giordano commented Jan 13, 2022

zklaus commented Jan 13, 2022

giordano commented Jan 13, 2022 • edited Loading

zklaus commented Jan 13, 2022

giordano commented Jan 13, 2022

mkitti commented Jan 13, 2022 • edited Loading

ngam commented Jan 13, 2022

mkitti commented Jan 13, 2022

zklaus commented Jan 13, 2022

ngam commented Jan 13, 2022 • edited Loading

zklaus commented Jan 13, 2022

mkitti commented Jan 13, 2022 • edited Loading

valeriupredoi commented Jan 14, 2022

mkitti commented Jan 14, 2022

valeriupredoi commented Jan 17, 2022

mkitti commented Jan 17, 2022

ngam commented Jan 18, 2022 • edited Loading

fingolfin commented Jan 18, 2022

ngam commented Jan 18, 2022

ngam commented Jan 18, 2022 • edited Loading

ngam commented Jan 18, 2022 • edited Loading

ngam commented Jan 18, 2022

ngam commented Jan 18, 2022 • edited Loading

valeriupredoi commented Jan 19, 2022

ngam commented Apr 3, 2022

ngam commented Jan 5, 2022 •

edited

Loading

mkitti commented Jan 6, 2022 •

edited

Loading

ngam commented Jan 6, 2022 •

edited

Loading

mkitti commented Jan 6, 2022 •

edited

Loading

ngam commented Jan 7, 2022 •

edited

Loading

ngam commented Jan 7, 2022 •

edited

Loading

ngam commented Jan 7, 2022 •

edited

Loading

ngam commented Jan 9, 2022 •

edited

Loading

ngam commented Jan 9, 2022 •

edited

Loading

giordano commented Jan 13, 2022 •

edited

Loading

mkitti commented Jan 13, 2022 •

edited

Loading

ngam commented Jan 13, 2022 •

edited

Loading

mkitti commented Jan 13, 2022 •

edited

Loading

ngam commented Jan 18, 2022 •

edited

Loading

ngam commented Jan 18, 2022 •

edited

Loading

ngam commented Jan 18, 2022 •

edited

Loading

ngam commented Jan 18, 2022 •

edited

Loading