Problem on windows platform with full paths in interfaces of Cbc, CbcSolve, Clp

[svigerske]

Issue created by migration from Trac.

Original creator: mserg

Original creation time: 2007-09-21 11:07:39

Assignee: somebody

Version:

Example:

---

...
Result - Finished objective 25562 after 14 nodes and 3613 iterations - took 31.6
57 seconds
Coin:solu c:\result.txt
Unable to open file .\c:\result.txt
Coin:

---

I use version 1.1.1 ...

[svigerske]

Comment by mserg created at 2007-10-14 14:18:10

Dear colleague,

This problem is model-independent. You can use any model, for example "hello.mps" from cbc-collection problems. See the follow log:
******** problem ":" **************************************
Coin Cbc and Clp Solver version 1.02.00, build Oct 12 2007
CoinSolver takes input from arguments ( - switches to stdin)
Enter ? for list of commands or help
Coin:import hello.mps
At line 1 NAME Hello
At line 2 ROWS
At line 25 COLUMNS
At line 185 RHS
At line 197 RANGES
At line 209 BOUNDS
At line 263 ENDATA
Problem Hello has 21 rows, 53 columns and 224 elements
Model was imported from .\hello.mps in 0.01 seconds
Coin:solve
Presolve 14 (-7) rows, 23 (-30) columns and 113 (-111) elements
Optimal - objective value 0
After Postsolve, objective 0, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0 - 0 iterations time 0.032, Presolve 0.03
Coin:solu c:\Result.txt
Unable to open file .\c:\Result.txt
Coin:
********* problem ":" ***************************************
The problem is absolute path with symbol ":".

I have found other problems in user's interface. The fatal error happens when the "pret" parameter is setting:
********* problem "PreT" *************************************
Coin:pret 1
preTolerance was changed from 1e-008 to 1

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
********* problem "PreT" *************************************

The suspected initial value is displayed when change "sec" parameter:
********* problem "sec" **************************************
Coin:sec??
sec(onds) : maximum seconds
After this many seconds coin solver will act as if maximum nodes had
been reached.
<Range of values is -1 to 1e+012;
current 1e+008>
Coin:sec 9999
seconds was changed from 1e+100 to 9999
Coin:Coin:
********* problem "sec" **************************************
Has the time limit the problems without integer variables? I am not yet sure...

-1 can be assigned to parameters "log" and "slog". What is this mean? There is no help... To parameter "passC" can be negative. What is this mean?
********* strange negative **************************************
Coin:log??
log(Level) : Level of detail in Coin branch and Cut output
If 0 then there should be no output in normal circumstances. 1 is
probably the best value for most uses, while 2 and 3 give more information.
<Range of values is -1 to 63;
current 1>
Coin:slog??
slog(Level) : Level of detail in Solver output
If 0 then there should be no output in normal circumstances. 1 is
probably the best value for most uses, while 2 and 3 give more information.
<Range of values is -1 to 63;
current 0>
Coin:passC??
passC(uts) : Number of cut passes at root node
The default is 100 passes if less than 500 columns, 100 passes (but
stop if drop small if less than 5000 columns, 20 otherwise
<Range of values is -999999 to 999999;
current -1>
********* strange negative **************************************

The strange value is displayed when the "cutoff" parameter change. The branch and cut algorithm is ignores "cutoff" changes... I set 1e55, but B&C use 1e50. See the follow log (the example "p0033.mps" was taken from cbc-collection):
********* cutoff problems ***************************************
Coin Cbc and Clp Solver version 1.02.00, build Oct 12 2007
CoinSolver takes input from arguments ( - switches to stdin)
Enter ? for list of commands or help
Coin:cutoff??
cuto(ff) : All solutions must be better than this
All solutions must be better than this value (in a minimization sense).
This is also set by code whenever it obtains a solution and is set
to value of objective for solution minus cutoff increment.
<Range of values is -1e+060 to 1e+060;
current 1e+050>
Coin:cutoff 1e55
cutoff was changed from 1e+100 to 1e+055
Coin:heur off
Option for heuristicsOnOff changed from on to off
Coin:cuts off
Option for cutsOnOff changed from on to off
Coin:import p0033.mps
At line 15 NAME P0033
At line 16 ROWS
At line 34 COLUMNS
At line 109 RHS
At line 118 BOUNDS
At line 152 ENDATA
Problem P0033 has 16 rows, 33 columns and 98 elements
Model was imported from .\p0033.mps in 0 seconds
Coin:solve
Optimal - objective value 2520.57
Cgl0006I 1 SOS (5 members out of 33) with 0 overlaps - too much overlap or too many others
Cgl0004I processed model has 15 rows, 32 columns (32 integer) and 97 elements
processed model has 15 rows, 32 columns and 97 elements
Optimal - objective value 2819.36
32 integer variables
Cbc0009I Objective coefficients multiple of 1
cutoff increment increased from 1e-005 to 0.999
Cbc0010I After 0 nodes, 1 on tree, 1e+050 best solution, best possible 2819.36 (0.01 seconds)
Cbc0016I Integer solution of 3302 found by strong branching after 21 iterations and 9 nodes (0.06 seconds)
Cbc0016I Integer solution of 3089 found by strong branching after 205 iterations and 89 nodes (0.29 seconds)
Cbc0001I Search completed - best objective 3089, took 278 iterations and 121 nodes (0.36 seconds)
Cbc0032I Strong branching done 528 times (921 iterations), fathomed 53 nodes and fixed 29 variables
Cuts at root node changed objective from 2819.36 to 2819.36
Result - Finished objective 3089 after 121 nodes and 278 iterations - took 0.38 seconds
********* cutoff problems ***************************************

Maybe my English is too bad, but the "inc??" says "plus increment" against "cutoff??" "minus increment"... Seems too strange...
********* cutoff increment direction ****************************
Coin:inc??
inc(rement) : A valid solution must be at least this much better than last integer solution
Whenever a solution is found the bound on solutions is set to solution
(in a minimizationsense) plus this. If it is not set then the code
will try and work one out e.g. if all objective coefficients are multiples
of 0.01 and only integer variables have entries in objective then
this can be set to 0.01. Be careful if you set this negative!
<Range of values is -1e+020 to 1e+020;
current 1e-005>
Coin:cutoff??
cuto(ff) : All solutions must be better than this
All solutions must be better than this value (in a minimization sense).
This is also set by code whenever it obtains a solution and is set
to value of objective for solution minus cutoff increment.
<Range of values is -1e+060 to 1e+060;
current 1e+050>
********* cutoff increment direction ****************************

I can't understand the help of "PrimalP" without drinking bottle of vodka... Which options is workable? Why the "primal exact" help is equivalent "dual steepest" help?
******** primalP & dualP help ***********************************
Coin:primalP??
primalP(ivot) : Primal pivot choice algorithm
Clp can use any pivot selection algorithm which the user codes as
long as it implements the features in the abstract pivot base class.
The Dantzig method is implemented to show a simple method but its
use is deprecated. Exact devex is the method of choice and there
are two variants which keep all weights updated but only scan a subset
each iteration. Partial switches this on while change initially does
dantzig until the factorization becomes denser. This is still a work
in progress.
<Possible options for primalPivot are: auto(matic) exa(ct) dant(zig) part(ial) steep(est) change sprint;
current auto(matic)>
Coin:dualP??
dualP(ivot) : Dual pivot choice algorithm
Clp can use any pivot selection algorithm which the user codes as
long as it implements the features in the abstract pivot base class.
The Dantzig method is implemented to show a simple method but its
use is deprecated. Steepest is the method of choice and there are
two variants which keep all weights updated but only scan a subset
each iteration. Partial switches this on while automatic decides at
each iteration based on information about the factorization.
<Possible options for dualPivot are: auto(matic) dant(zig) partial steep(est);
current auto(matic)>
******** primalP & dualP help ***********************************

-----Original Message-----
From: coin-trac@coin-or.org [mailto:coin-trac@coin-or.org]
Sent: Monday, September 24, 2007 1:32 AM
Cc: cbc-tickets@list.coin-or.org
Subject: Re: [COIN-OR Branch-and-Cut MIP Solver] #32: Problem on windows platform with full paths in interfaces of Cbc, CbcSolve, Clp

#32: Problem on windows platform with full paths in interfaces of Cbc, CbcSolve,
Clp
-------------------------+--------------------------------------------------
Reporter: mserg | Owner: somebody
Type: defect | Status: new
Priority: major | Milestone:
Component: component1 | Version:
Resolution: | Keywords:
-------------------------+--------------------------------------------------
Comment (by jpfasano):

What Version of Cbc are you using?
What do you do to create the problem?
How can the problem be recreated?

--
Ticket URL: https://projects.coin-or.org/Cbc/ticket/32#comment:1
COIN-OR Branch-and-Cut MIP Solver http://projects.coin-or.org/Cbc
An LP-based branch-and-cut MIP solver.

[svigerske]

Comment by ladanyi created at 2008-04-11 13:18:56

fixed it in trunk. Will soon make a new stable and release out of it.

[svigerske]

Comment by ladanyi created at 2008-04-11 13:18:56

Resolution: fixed

[svigerske]

Comment by marotig created at 2008-11-25 13:26:46

The problem with full paths appears in the windows version of Cbc-2.2.2: calling
$ cbcSolve.exe -import d:\lp.lp -solve -solu d:\sol.sol
triggers:
Unable to open file .\d:\sol.sol
The same happens to clp.exe.

Regards,
G.

[svigerske]

Comment by marotig created at 2008-11-25 13:26:46

Resolution changed from fixed to

[svigerske]

Comment by marotig created at 2008-11-25 13:26:46

Changing status from closed to reopened.

[svigerske]

Comment by marotig created at 2009-06-19 15:24:46

The absolute paths are still broken in Cbc-2.3.0 (on Windows). It looks like I am the only one who encounters this problem...

Regards,
Gabor

[joakimkejser]

Hi,

I still have issues with absolute paths on windows, but only for the mipstart file:
opening mipstart file .\C:\Users\JOAKIM~1.KEJ\AppData\Local\Temp\eee606dd399540daa6bd17fcabbb10b3-pulp_init.sol

Reading .mps files and specifying the solution file works fine with absolute paths.

I'm on 2.10.3

[gastald88]

Hi,

I still have issues with absolute paths on windows, but only for the mipstart file:
opening mipstart file .\C:\Users\JOAKIM~1.KEJ\AppData\Local\Temp\eee606dd399540daa6bd17fcabbb10b3-pulp_init.sol

Reading .mps files and specifying the solution file works fine with absolute paths.

I'm on 2.10.3

Hello @joakimkejser, I have the same problem. Did you somehow overcome such issue?

[rahulbanerjee26]

@joakimkejser @gastald88
Facing a similar issue when trying to warm start the solver. Adding a parameter keepFiles = True changed the directory to relative and warm start was working fine.

[gastald88]

@rahulbanerjee26 thank you very much for the answer

[jsiirola]

I just hit this as well. The problematic code:

Cbc/src/CbcSolver.cpp

Lines 9445 to 9459 in 3dcedb2

	std::string fileName;
	if (field[0] == '/' || field[0] == '\\') {
	fileName = field;
	} else if (field[0] == '~') {
	char *environVar = getenv("HOME");
	if (environVar) {
	std::string home(environVar);
	field = field.erase(0, 1);
	fileName = home + field;
	} else {
	fileName = field;
	}
	} else {
	fileName = directory + field;
	}

needs an additional check for something like

else if (field.length() >= 2 && field[1] == ':' &&
         (field[0] >= 'a' && field[0] <= 'z' || field[0] >= 'A' && field[0] <= 'Z')) {
    fieldName = field
}

[tkralphs]

Sorry that this has been outstanding for so long. This and many other related things are cleaned up in the refactor branch, which has been the focus of recent development. I've moved all of the processing of file names from Cbc and Clp into some common functions in CoinUtils, so bugs like this will be much easier to fix. Detection of absolute paths is encapsulated there (although I now realize that it is slightly buggy even there).

Since I hope to merge the refactor branch soonish, I would prefer not to patch this in master. However, since 2.10 is so widely used, it seems worth fixing this there and including the fix in a final 2.10.6 release that will patch up a few things. It looks to me like merging your check with the first one (which is also checking whether the path is absolute) would be the easiest fix. @jsiirola, if you agree, I'll make the change. This bug arises in a couple other places where this same logic is used, but I'm not sure if I want to try to fix all occurrences, since the others are unlikely to be an issue.

Incidentally, in the refactor branch, CoinUtils is currently using

if (field[0] == '/' || field[0] == '\\' || fileName.find(':') != std::string::npos) {

But now I realize that since ':' is legal in Linux filenames, I guess this is technically not quite correct. Thanks for alerting me to this!

[MLopez-Ibanez]

Is there a reason to not use the standard library functions for this type of basic functionality?

https://en.cppreference.com/w/cpp/filesystem/absolute

The same functions exist in boost:

https://www.boost.org/doc/libs/1_58_0/libs/filesystem/doc/reference.html#Operational-functions

Or the gnulib function canonicalize: https://www.gnu.org/software/gnulib/MODULES.html#module=canonicalize

(There are lots of very useful functions in gnulib implemented in the most portable way possible).

[tkralphs]

Partly, the answer is that, while it is clear there's a lot of stuff that could be improved by using modern C++, it's just a lot of work and there's nobody to do it. Pull requests are always welcome (but please check the refactor branches first)!

In Cbc and Clp currently, there are quite a few different places where filename processing occurs, so the first natural step is to move all of this processing back into CoinUtils so that fixes don't need to be propogated out in dozens of places. This is one of the things being done in the refactor branches. Then the final fix will only need to be applied in one place.

As for your suggestions, std::filesystem::absolute requires C++17, which we don't want to assume. Requiring Boost also seems problematic. Gnulib looks nice, but is probably infeasible due to license incompatibility. Perhaps there is some liberally licensed code out there to do this, I haven't looked very hard. Further suggestions are welcome.

[MLopez-Ibanez]

As for your suggestions, std::filesystem::absolute requires C++17, which we don't want to assume. Requiring Boost also seems problematic.

Just out of curiosity, may I ask the reasons for both decisions? The C++17 filesystem library is supported since a few years by all major compilers: https://en.cppreference.com/w/cpp/compiler_support

Boost is a library that is widely used. Yes, it is another dependency, but one that should not be too hard to install. Of course, I'm not asking anyone to do it, but I'm wondering if the project would accept it if someone decided to spend the effort (or sponsor the work) to do it and what would be the rationale to decline.

[tkralphs]

Just out of curiosity, may I ask the reasons for both decisions? The C++17 filesystem library is supported since a few years by all major compilers: https://en.cppreference.com/w/cpp/compiler_support

That is a nice reference that I will definitely study a bit. One issue arising over time has been not breaking the build with older compilers, which we would still like to support for a variety of reasons. For example, building Python extensions used to require older versions of the Visual Studio compiler. It's possible this situation has improved, but it's been a moving target over time and with developer bandwidth so limited, the path of least resistance to ensure portability for the widest range of users is not to bring in things like C++17. I think there's also something to be said for uniformity. Right now, we still have quite a bit of even C-style string handling, which would require a big effort to factor out. Having older C-style string handling in some places and modern C++17 in others doesn't seem ideal. Some of the string-handling is being improved in the refactor branches.

So I guess the short answer is a (severe) lack of bandwidth to even consider whether starting to incorporate C++17 into the code base would break anything important. Cbc is embedded in so many different places that have their own build requirements, I would be hesitant to even start down the road at this point.

Boost is a library that is widely used. Yes, it is another dependency, but one that should not be too hard to install. Of course, I'm not asking anyone to do it, but I'm wondering if the project would accept it if someone decided to spend the effort (or sponsor the work) to do it and what would be the rationale to decline.

It's been a while but Boost used to be a pretty heavy install on Windows and a bit difficult to work with in certain build configurations. We need to support builds in conda, Msys, Visual Studio, Julia, R, etc. Sounds like something that would require some work to make seamless across all builds. It might be easy enough to detect whether Boost is present, but there's no point is supporting it as an optional dependency, since we would then need a fallback. This again mostly goes back to the severe lack of developer bandwidth and just a huge range of build environments that we're currently managing to support. Adding Boost as a dependency seems likely to upset the apple cart.