Cbc is not sending settings to Clp for the LP solve (with interactive or C++ API)

[ianfiske]

With a file called file.lp having contents:

Minimize
obj: - 2 x3

Subject To
c1: x2 - x1 <= 10
c2: x1 + x2 + x3 <= 20

Bounds
x1 <= 30

End

If I run Cbc 2.10.8 interactively and try to set logLevel to 0, it does not mute logging in Clp:

Welcome to the CBC MILP Solver 
Version: 2.10.8 
Build Date: May  7 2022 

CoinSolver takes input from arguments ( - switches to stdin)
Enter ? for list of commands or help
Coin:import file.lp
Coin:logLevel 0
Coin:logLevel     
logLevel has value 0
Coin:solve
Presolve 0 (-2) rows, 0 (-3) columns and 0 (-5) elements
Empty problem - 0 rows, 0 columns and 0 elements
Optimal - objective value -40
After Postsolve, objective -40, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective -40 - 0 iterations time 0.002, Presolve 0.00

This was noted as an issue that affects the Julia interface to Cbc: jump-dev/Cbc.jl#168.

My true issue is that the settings for PrimalTolerance and DualTolerance also are not being passed down to Clp for the LP, but it was a little tougher to make an MWE to show that, so I stuck with the logLevel example. This is causing a problem for a problem I have where tolerances end up making a big difference.

[tkralphs]

There are separate parameters for the log level of Cbc itself and the log level of the LP solver. To change the log level for the LP solver, use the parameter slogLevel. It's probably not the most intuitive parameter name, but it is at least documented:

Coin:slog?
slog(Level) : Level of detail in (LP) Solver output

So I guess the issue you are having with passing tolerance parameters is something unrelated. If you give me some idea of what you're doing (what parameters are being set and not passed, etc.), I can at least try to point to where those parameters should be passed in the code (or perhaps see that they are not).

[ianfiske]

Thank you for the quick response! That's good to know regarding the log level, though as you noted that's not my real issue.

The parameters that I'm trying to pass to Clp are PrimalTolerance and DualTolerance. In slightly older versions of Cbc, I was able to set these 2 parameters and they had the desired effect on the LP (Clp). However, in the latest version, this stopped working.

[tkralphs]

Could you specify a particular version of Cbc where this worked? Are you suggesting that it worked for previous releases in the 2.10 series or in previous stable versions altogether (2.9 or earlier)?

[ianfiske]

Yes, this worked fine back in v2.10.3.

[odow]

this worked fine back in v2.10.3

FWIW, Cbc.jl used a patched version of 2.10.3 that meant we never went through Clp for LPs. We're now back on mainline in 2.10.5, which is why these issues cropped up.

The slogLevel didn't seem to work either: jump-dev/Cbc.jl#201.

[ianfiske]

Thank you for the explanation @odow ! This makes so much more sense now. It sounds like the solution is to find the name of the setting in Cbc, if it exists, to actually change the LP primal and dual tolerances since these are no longer affected by the names I sent before, which were presumably Clp attributes, not Cbc attributes.

@tkralphs Do you know what the setting names would be for Cbc to change the primal and dual tolerances for the inner LP problem solver?

[ianfiske]

I have also come across a similar problem where the "seconds" attribute appears to be ignored in the inner LP. I don't have an MWE yet.

[SBuercklin]

This still seems to be an issue on 2.10.8.

To reproduce Ian's seconds behavior, I have two MPS files, one which constrains variables to be integers and one which lets them take on any real number. Running these through Cbc, the integer MPS respects attributes which are passed in (in this case, seconds) while the real number MPS actual completes the optimization and ignores the seconds attribute.

I put the MPS files, CLI instances, and solutions in gists here (failing to pass seconds) and here (successfully passing seconds in)

Reproducing the resulting solutions here to show the difference:

Without constraining to integers, the optimization ignores the time limit and goes through the optimization

Optimal - objective value -10.00000000
      0 x1                     4                       0
      1 x2                     6                    -0.5

Constraining the optimization to integers, the optimization obeys the time limit and terminates immediately

Stopped on time (no integer solution - continuous used) - objective value -10.00000000
      0 x1                     4                       0
      1 x2                     6                    -0.5

[tkralphs]

@SBuercklin, thanks for the thorough bug report. I guess this is more of a feature than a bug. I believe that Cbc doesn't ever interrupt the LP solver (Clp), which is part of the reason why the solution process doesn't always exit at exactly the time limit. It exits after the last LP solve that results in exceeding the time limit. In other words, the code can only exit from within the Cbc part of the code, not within Clp.

When there are no integer variables, Cbc detects that the problem is an LP and simply passes it to Clp, which never has a time limit for its computations when called from Cbc. In both cases, the LP relaxation is being solved to optimality before exiting, the difference is probably just that in the case of an integer program, the computation is actually terminated early because it would otherwise move on to the branch-and-bound, but in the case of being passed to Clp, the computation is completed after exiting from the LP solve, so the status isn't that the problem is being stopped on time---it was actually solved to optimality.

But this is all without really digging into the code.

[SBuercklin]

Thanks @tkralphs, that makes sense. Is there a list of attributes or options that are forwarded to Clp? We're still encountering an issue where dualTolerance and primalTolerance do not seem to be forwarded to Clp.

I will work on reproducing a MWE for that directly in the meantime

[tkralphs]

I don't think there is a list, but it should not be too difficult to figure out by looking through CbcSolver.cpp for every occurrence of setting a Clp parameter. That should be the only place where they are set. In the master branch, this is probably easier to parse, as I have partially refactored both Cbc and Clp to completely separate their parameters sets. So it is easy to identify which parameters are Cbc versus Clp in the code. In stable/2.10, it is not as obvious because there is one set of parameters for both solvers. Even in master CbcSolver.cpp still needs some refactoring. Let me see if I can easily grep for Clp parameters and get a list.

[tkralphs]

If you just search CbcSolver.cpp in master for clpParameters, you can get an idea. Indeed, it seems that primal tolerance and dual tolerance are set to default values here:

Cbc/src/CbcSolver.cpp

Lines 980 to 992 in 0713c30

	clpParameters[ClpParam::DUALBOUND]->setVal(lpSolver->dualBound());
	clpParameters[ClpParam::DUALTOLERANCE]->setVal(lpSolver->dualTolerance());
	clpParameters[ClpParam::IDIOT]->setVal(doIdiot);
	clpParameters[ClpParam::PRESOLVETOLERANCE]->setVal(1.0e-8);
	clpParameters[ClpParam::MAXFACTOR]->setVal(lpSolver->factorizationFrequency());
	clpParameters[ClpParam::MAXITERATION]->setVal(lpSolver->maximumIterations());
	clpParameters[ClpParam::PRESOLVEPASS]->setVal(preSolve);
	clpParameters[ClpParam::PERTVALUE]->setVal(lpSolver->perturbation());
	clpParameters[ClpParam::PRIMALTOLERANCE]->setVal(lpSolver->primalTolerance());
	clpParameters[ClpParam::PRIMALWEIGHT]->setVal(lpSolver->infeasibilityCost());
	clpParameters[ClpParam::SPRINT]->setVal(doSprint);
	clpParameters[ClpParam::SUBSTITUTION]->setVal(substitution);
	clpParameters[ClpParam::DUALIZE]->setVal(dualize);

But they don't seem to be mentioned anywhere else. It should be easy to allow.

[tkralphs]

Actually, I spoke too quickly. I had put aside the refactoring effort and it's been long enough that I've probably forgotten a few things. Both primalTolerance and dualTolerance are in fact listed as parameters that Cbc understands and they do seem to be accepted if you set them from the interactive shell. I'll have to find some time to dig into this. If they don't seem to be having an effect, something deeper is going on. An MWE would certainly be helpful.

[SBuercklin]

@tkralphs, I appreciate your help. This was cropping up in the Julia interface and we've been trying to reduce it to a MWE for a while. We finally got it down by exporting a larger model that we giving us issues, and it appears that the issue we're running into is either a problem with Cbc.jl itself, or a problem with the C interface which sets the parameters.

I've opened an issue with a MWE here, and the MWE seems to definitively show it's not happening in the CLI, but someone else will need to determine exactly where the bug is coming from.

[SBuercklin]

Cbc.jl maintainers believe it's an upstream issue, so I'll point to the C-interface calls from Julia to try to help pin it down. The .mps file + CLI example in the linked MWE might help for reproduction.

The call to the C interface happens here, where the parameter name and value (converted to a string) are passed into the C wraper

The actual C call happens here to Cbc_setParameter. AFAICT the issue seems to be that function and how it handles "{double,primal}Tolerance".

[odow]

I confirmed that this is a problem in the C API, not the Julia package: jump-dev/Cbc.jl#211 (comment)

But @SBuercklin has a bigger problem: their model is poorly scaled, which I guess is why they're wanting to tighten the primal/dual tolerances.