Build Coin-OR and JuMP solvers

[giordano]

To address jump-dev/Clp.jl#73. CC@ @Yuya-Furusawa @juan-pablo-vielma

The original builder mixes several packages in a single builder:

• clp
• osi
• coinutils
• ASL

Note by @ViralBShah: I updated this issue for Coin-OR libraries and solvers JuMP needs:

• cbc
• glpk
• scs
• ipopt
• ecos
• sdpa
• csdp

Instead of CoinBLAS and CoinLAPACK, we want to link CoinUtils etc. to Julia's openblas (#544), which may need some patching for 64-bit indices. Similarly for CoinMetis and CoinMUMPS, we want to be able to link to the libraries we build as JLLs here.

[giordano]

Ok, let's start with the first issue. We already have a builder for a package called ASL. @juan-pablo-vielma could you please confirm it's the same thing?

[dpo]

Yes it's the same thing. You can also eliminate COINLapack and COINBLAS and replace them with OpenBLAS_jll. COINMetis can be replaced with METIS@4_jll (#530). I'm working on MUMPS as time allows.

[giordano]

You can also eliminate COINLapack and COINBLAS and replace them with OpenBLAS_jll.

Oh, good to know that COINBLAS was a complete waste of time. Was fun patching a dead project

[dpo]

I'm not saying COINBLAS, etc., are dead projects. But COINBLAS and COINLapack will install generic (i.e., non optimized) BLAS/Lapack libraries and you're almost certainly better off with the somewhat optimized OpenBLAS (which ships both).

[giordano]

I'm not saying COINBLAS, etc., are dead projects.

The maintainer of the project that closed my issue asking to fix the configure script told me so 🙂

[dpo]

Ah ok. I've never used them for the reason above. I didn't know. Sorry you spent time on it.

[ViralBShah]

Is it possible to get COINMetis updated to use METIS 5?

[dpo]

What for though? There's already the METIS builder here, which builds METIS 5.

[ViralBShah]

Ah - I see. I thought COINMetis had a dependency on METIS 4 perhaps. Sorry about the noise.

[dpo]

As far as I can tell, it only builds METIS 4. But #530 should make it unnecessary.

[ViralBShah]

The problem we will face is that METIS 5 will start shipping with SuiteSparse in Julia base. My worry is that any use of METIS 4 will then conflict with SuiteSparse.

[giordano]

My worry is that any use of METIS 4 will then conflict with SuiteSparse.

That's very true 😕

[ViralBShah]

I wonder if METIS 4 can be linked into clp directly as libmetis.a. That might work fine.

[dpo]

We need METIS 4 for other packages than CLP though. This is the case of any application that uses SuiteSparse, and not just Julia, right? How is the situation different here than anywhere else? A user should be able to use METIS 4 if they need it.

[ViralBShah]

I am guessing it would work if you don't use cholmod in the same process as metis 4 libraries. Do we know if the two libraries have the same APIs or global state? If they won't stomp on each other, may be we can get away with simultaneously shipping two versions.

[ViralBShah]

Actually, I don't think we have any other case where we have two versions of the same library in simultaneous use. Usually it is part of a transition - where older releases of a julia package use the older version of a library, and newer releases of that package use a newer version.

[dpo]

Wouldn't this be a good case for renaming the METIS 5 symbols inside SuiteSparse?

[ViralBShah]

I would have thought that more packages will adopt METIS 5 given its improvements and how old METIS 4 is. At that point we'd have to then make changes in every package that uses METIS 5. We would also have to update METIS.jl (which is much easier to do).

I wonder if @fredrikekre knows about the APIs of both versions to say if they can co-exist.

[ViralBShah]

The METIS manual has quite a bit of information on the differences between the 2 versions: http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/manual.pdf

From what I can gather - I don't think both versions can co-exist in the same process.

[ericphanson]

If Clp gets built, maybe it'll be easy to build SDPA too, since they share a lot of the dependencies (https://github.com/JuliaOpt/SDPABuilder/blob/master/build_tarballs.jl).

[giordano]

I think I'll give up, all these packages have super broken build systems, wasting my time on fixing all of them is not a fun activity

[ViralBShah]

Yeah, many of these packages have really weird build systems. It is also not scalable for you to fix every single one of them!

[odow]

FWIW, the COIN build system is exceedingly difficult to work with. We bundled everything together to avoid this dependency mess. It's not the best solution, but it works.

[ViralBShah]

FWIW, the COIN build system is exceedingly difficult to work with. We bundled everything together to avoid this dependency mess. It's not the best solution, but it works.

That is definitely experience speaking!

[ViralBShah]

@mlubin Moving the discussion here.

Basically, BinaryBuilder is staying as it is. We used to have old builders in project specific repos. Those are now centralized into https://github.com/JuliaPackaging/Yggdrasil and the CI system builds all the architectures and registers the produced binaries as artifacts. For example, Clp binaries would be uploaded as the Clp_jll.jl Julia package. Basically, BinaryProvider and deps/build.jl goes away, and you then add a dependency for Clp_jll in the Julia package that needs to ccall that library.

While this issue is specifically for Clp, perhaps we should use it to discuss all the various JuMP dependencies?

[ViralBShah]

@odow In response to your message in #653 (comment), the current BinaryProvider stuff will keep working for a while. The artifacts system does provide lot of benefits in terms of simplifying the user experience.

I suspect it should be possible to take the monolithic builders as they are and move them to the artifacts system. Are the monolithic pieces in the ClpBuilder only used within Clp or do we need to reuse them elsewhere? I'm happy to help out as much as I can.

[giordano]

The problem of the monolithic builder of Clp is also that it uses static libraries in several places, hiding a lot of complexity.

Having a monolithic builder is very bad under many aspects:

• increases code/library duplication reducing possibility of reusing the same library in other contexts
• updating a single component requires updating the entire monolithic package

We followed the route of a monolithic builder for Xorg, it turned out to be extremely wrong: #92.

[ViralBShah]

I'll give Clp a shot without being monolithic just to see what it takes.

[odow]

I have a PR for Clp: jump-dev/Clp.jl#77. Can confirm that it is passing all tests on

julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

Outstanding work.

[ViralBShah]

Wow that was a bit unexpected! Do we not need the sparse solver to be plugged into Clp?

[odow]

I guess there is a non-sparse fallback somewhere? All tests pass on the PR across all platforms.

The tests are all very small problems. So we won't notice any performance issues.

On a larger instance (linked to in jump-dev/Clp.jl#53 about testing the LAPACK fallback), the Clp_jll takes 70 seconds, compared to 27 seconds for the BinaryBuilder version.

[mlubin]

Based on my experience from 10 years ago, Clp doesn't use MUMPS. It has its own sparse linear algebra implemented in CoinUtils.

[ViralBShah]

Huh - why does it then offer to link to mumps in the build? I would imagine it would be faster if you use external solvers. In Clp 1.17, they are also offering the ability to use suitesparse.

[ViralBShah]

@mlubin @odow are there large benchmark problems sense and sparse I can easily try to see if the external direct linear solvers help? If it does in experiments, I can provide the BB updates.

[mlubin]

No idea. Maybe it's for Clp's barrier code (that's mostly unused).

The standard benchmark instances to run are at http://plato.asu.edu/ftp/lpsimp.html.

[odow]

Here's the script I used before:

using Clp
const MOI = Clp.MOI
download(
    "ftp://ftp.numerical.rl.ac.uk/pub/cuter/netlib/QAP12.SIF",
    "QAP12.mps"
)
mps = MOI.FileFormats.MPS.Model()
MOI.read_from_file(mps, "QAP12.mps")
model = Clp.Optimizer()
MOI.copy_to(model, mps)
MOI.optimize!(model)

Clp0000I Optimal - objective value 522.89435
Clp0032I Optimal objective 522.8943506 - 25514 iterations time 28.962

julia> 

(clp) pkg> st
    Status `/private/tmp/clp/Project.toml`
  [e2554f3b] Clp v0.7.1

Compared to

Clp0000I Optimal - objective value 522.89435
Clp0032I Optimal objective 522.8943506 - 25951 iterations time 66.952

(clp2) pkg> st
    Status `/private/tmp/clp2/Project.toml`
  [e2554f3b] Clp v0.7.1 #od/jll (https://github.com/JuliaOpt/Clp.jl.git)

[ViralBShah]

@odow Does this mean we are slower now than before?

[odow]

Yes, which I take to mean that Juan-Pablo's BinaryBuilder version of Clp (v0.7.1) is using mumps.

[ViralBShah]

Ok, I'll prepare a new one with mumps.

@dpo It looks like MUMPS_seq uses METIS 5, which will conflict with METIS 4 that Clp needs. Also, I am thinking it may be better to standardize on 32-bit BLAS and LAPACK in the optimization ecosystem for now, until it can all be upgraded to 64-bit BLAS. MUMPS_seq is currently using 64-bit BLAS, but for now I am willing to test it out.

[ViralBShah]

I have the first binaries for Cbc as well. You'll have to update Clp_jll to 1.17.5 as well (I hope it works). Basically Clp and Cbc were using different versions of the Coin-OR dependencies, and we want to use a consistent set across all Coin-OR solvers.

@odow Can you let me know how Clp 1.17.5 does? And if Cbc 2.10.5 works? None of these are using Mumps just yet, but once we have the basic set up all working, I'll add Mumps across the board.

[ViralBShah]

@dpo Note I am cosolidating all the Coin-OR solvers in one place: #775

Perhaps we can get Ipopt there too. Should ASL move there? Or stay where it is (since it doesn't seem to technically be part of Coin-Or, but a dependency)?

[odow]

@ViralBShah: can confirm that Clp_jll 1.17.5 passes tests locally (Mac). I've updated the PR.

I can also confirm that Cbc 2.10.5 passes tests locally. Here's the PR: jump-dev/Cbc.jl#133

Now to see how it does on Windows and linux...

[ViralBShah]

That's comforting!

[odow]

Spoke too soon. Clp errors on linux: https://travis-ci.org/github/JuliaOpt/Clp.jl/jobs/671375887#L307 (Fine on Mac though.)

[ViralBShah]

Seems like the tests all pass, but there's a crash afterwards. Very similar to a crash we saw with sundials just a few mins back.

[ViralBShah]

Does it happen on Julia 1.4 too?

[odow]

Yes, identical failure: https://travis-ci.org/github/JuliaOpt/Clp.jl/jobs/671375888

The first block of tests pass, but there is still a second @testset that didn't finish running:
https://github.com/JuliaOpt/Clp.jl/blob/c6121a52c6c980afa231d88944b71d90884494c0/test/runtests.jl#L6-L12

[ViralBShah]

Is it possible to do ENV["OPENBLAS_NUM_THREADS"]=1 before running the tests?

[blegat]

Another Coin-OR solver used in JuMP is CSDP: https://github.com/JuliaOpt/CSDP.jl
It is currently not working for windows so switching to a jll might be a noticeable improvements for windows users.

[ViralBShah]

@dpo has now published Ipopt_jll. So that's one more! I suspect that Ipopt_jll's dependency on MUMPS 5 and Metis 5 is going to conflict with the dependencies needed by Clp and Cbc.

[amontoison]

@blegat, @odow CSDP builder is working on all platforms.
#795

[blegat]

@amontoison Thank you so much ! Would it be possible to build the version of CSDP of this PR:
coin-or/Csdp#12
The CSDP.jl Julia wrapper uses this version but the CSDP maintainers have not merged it yet upstream.

[ViralBShah]

I feel comfortable closing this one. There'll still be issues to work through in individual packages, but by and large I feel we are in a good place.