Initial merge of the refactor branch

[tkralphs]

TL;DR: The refactor branch will be merged on 1/1/2022. There are some changes to various interfaces. For a limited time, compatibility files that allow switching back to the "old style" are provided in the src/Attic subdirectory. Details below.

Cbc is has been undergoing a major refactoring for some time. This has been moving slowly due to a lack of developer bandwidth. This refactoring is not expected to affect the performance of the code in general, but it is highly likely that there will be some random side effects. We would be happy to hear about any serious regressions, but please keep in mind that while performance is expected to be similar on the whole, performance on individual instances could vary (as one would expect with any change). The goals of the refactoring are only to address readability, maintainability, and usability.

Overall goals

As a reminder (see also #374), here are some of the main goals of this refactoring.

• Make code thread-safe and eliminate global variables.
• Make structure of code easier to understand and maintain.
• Simplify and unify the parameters setting mechanism and separate it completely from Clp.
• Convert to C++ strings/streams everywhere to unify handling of input.
• Unify all messaging through message handlers and eliminate back-door methods of printing and controlling print levels.
• Unify error handling
• Combine separate libraries into a single library.
• Further develop the CbcSolver class to contain what is currently ClpMain0() and ClpMain1().
• Modularize the implementation of CbcMain1(), splitting the while loop into separate functions.

Here is what has been done so far and that will be merged from the current refactor branch.

• Build out the already existing parameter mechanism in CoinUtils.
• Build out the already existing utilities in CoinUtils for reading input from command-line, file, or environment.
• Create a CbcParam derived from CoinParam.
• Create a CbcParameters class with methods to set up parameters and contain the vector of parameters.
• Convert Cbc to use the new parameter and input mechanisms (thus separating it completely from Clp).
• Convert to using string keywords rather than integer modes to the extent possible for code readability and also eliminate other "magic numbers" where possible.
• Create new parameter types for file names and directories so that these can be parsed and validated correctly to ensure they represent valid paths, addressing existing bugs.
• Create constructors in the CbcParameters class representing different possible parameters settings (e.g., emphasize optimality versus feasibility).
• Ensure that default parameter values make sense.
• Create enums for the keyword parameters.
• Eliminate global variables and ensure thread safety
• Use modern C++ string manipulation
• Eliminate direct printing to std::out and make sure all output (at least in CbcSolver) is controlled by the log parameter through the proper message handler.
• Eliminate the separate libCbcSolver, which didn't seem to serve a purpose.

Changes to the interface

In this first step of the refactoring, we have endeavored not to change the interfaces too dramatically, but there are some differences, summarized here. Note that we may have missed some things, so feel free to let us know in the comments about differences you run into that trip you up in the conversion. Also, if there are any differences that you feel make the interface worse, let us know that. The changes were intended to be improvements, but we may have missed the mark.

Command line interface

Most of the changes to the command-line interface that users will notice come from changes made to allow for better parsing and validation of parameter values. There are two additional parameter types, Directory and File, so that values assigned to parameters that are expected to be file or directory names can be properly validated to ensure they specify valid paths. This avoids later errors when trying to open files and bugs that have been reported by users over time. The parameter types are as follows.

• Double: Parameters whose value is a double
• Integer: Parameters whose value is an integer
• String: Parameters whose value is a string
• Directory: For storing directory names, such as default locations for file types
• File: For storing the locations/names of files
• Keyword: Parameters that have one of several enumerated values identified by either strings or corresponding integer modes.
• Action: Not parameters in the traditional sense, but used to trigger actions, such as solving an instance.

The main changes to how parameters are specified both on the command line and in the interactive console are as follows.

• (Almost) all parameters that end in File (e.g., exportF(ile), mipStartF(ile)) are used to set the default file location/name for certain input/output files. Similarly, parameters that begin with dir specify a directory that is the default location for some file type.

• File name parameters are all associated with actions that involve reading or writing from the files whose names they specify. For actions involving writing/reading to a file, a $ was previously used to indicate that the action should be performed using "the same file as the last time the action was performed," but this is no longer supported. Instead, the previous (or initial default) file name will be used automatically if no argument is provided to the action parameter.

  As described above, there is a separate command for setting the default file name. Alternatively, if a file name is given as an argument, then this also automatically sets the file name. To exemplify what all this means, here are some basic commands as they would be given in the interactive console, with explanations

~> cbc
Welcome to the CBC MILP Solver
Version: Devel (unstable)
Build Date: Dec 30 2021
Cbc takes input from arguments ( - switches to stdin)
Enter ? for list of commands or help
Cbc: import mip1.mps
Cbc: solve
Cbc: writeSol                  # write solution to file named opt.sol (initial default name)
Cbc: writeSol my.sol           # write solution to file named my.sol
Cbc: import mip2.mps
Cbc: writeSol                  # write solution to file my.sol
Cbc: solFile my2.sol           # set solution file name to my2.sol
Cbc: writeSol                  # write solution to file my2.sol

• Parameter names for some action parameters have been changed to try to make the naming more consistent overall. In particular, a parameter beginning with read is something that reads a file and a parameters beginning with write is something that writes a file.
• Similarly, an action beginning with print should usually write to the standard output. This is to replace the old mechanism, which was to use the special file name -.
• Parameters names for setting the log levels of the LP solver and the MIP solver are now named log(Level) and lplog(Level), which is hopefully more memorable.

Changes to the API

Cbc is still invoked by calling CbcMain0() and CbcMain1(), as before, but the signatures of these functions have changed to reflect changes to the underlying implementation (the old signatures continue to work through wrapper functions). The signature of CbcMain0 is now

void CbcMain0(CbcModel &model, CbcParameters &parameters)

where model is a CbcModel object, as before, and parameters is an object containing parameter settings. To obtain a CbcParameters object pre-populated with default parameter settings, simply use its default constructor.

p = CbcParameters()

Parameters can then be customized individually using the new mechanism for setting parameter values described in more detail below. The signature of CbcMain1 has also changed. It is now

int CbcMain1(std::deque<std::string> inputQueue, CbcModel &model,
             CbcParameters &parameters,
             int callBack(CbcModel *currentSolver, int whereFrom) = dummyCallback, 
             ampl_info *info = NULL);

The inputQueue is a standard deque containing the strings making up the command line, which is in turn just a list of commands for setting parameters. parameters names are preceded by a -. A convenience method for building an input queue from strings is provided, so that a simple sequence of commands for solving a MIP would then be.

OsiClpSolverInterface solver;
CbcModel model(solver);
CbcParameters parameters;
std::deque<std::string> inputQueue;
CoinParamUtils::formInputQueue(inputQueue, "cbc", "p0033.mps", "-solve", "-printSol");
CbcMain0(model, parameters);
CbcMain1(inputQueue, model, parameters);

Note that the callBack and ampl_info arguments to CbcMain1() have default values, so these are not ordinarily needed in calling CbcMain1.

More Details on Parameter mechanism

There are two main classes and some utilities.

• The CbcParam class is derived from CoinParam and hold information for a single parameter (value, type, upper and lower limits, help strings, etc.).
• The CbcParameters contains a vector of CbcParam objects representing the current settings.

CbcParam

Each parameter has a unique code, which is also its index in the parameter vector stored in the CbcParameters class (described below). The codes are specified in the CbcParamCode enum within the CbcParam class.

Parameters also have a type, which can be queried and is one of

• Double: Parameters whose value is a double
• Integer: Parameters whose value is an integer
• String: Parameters whose value is a string
• Directory: For storing directory names, such as default locations for file types
• Filenames: For storing the locations/names of files
• Keyword: Parameters that have one of several enumerated values identified by either strings or corresponding integer modes.
• Action: Not parameters in the traditional sense, but used to trigger actions, such as solving an instance.

There are different constructors for each type, as well as setup functions for populating an existing parameter object with

• Name
• Short help string
• Long help string
• Limits on values (for error checking)
• Defaults values
• Display priority (to limit which parameters get displayed when users ask for help)

For keyword parameters, there is also a mechanism for creating a mapping between keyword value strings and the so-called "mode" value, which is an integer. The value of a keyword parameter can be set or retrieved either as a keyword string or as an integer mode. The modes may also be specified in enums.

There are separate get/set methods for each parameter type, but for convenience, there is also a single setVal() and getVal() method that is overloaded and can set the value of any parameter (the input/output is interpreted based on the known parameter type), e.g.,

param.setVal(0);

Each parameter object also has optional "push" (and "pull") function that can perform actions or populate related data structures upon the setting of a parameter value. The push/pull functions are defined with the CbcParamUtils name space, described below.

None of the methods in the class produce output directly. The get/set methods can optionally populate a string object that is passed in as an optional second argument. This is to allow the calling function to control output by piping the string to a message handler as appropriate or simply ignoring it if printing is not desired. This obviates the need to control printing internal to the functions themselves, which would require a separate parameter.

std::string message;
if (param->setKwdVal(value, &message)){
   messageHandler->message(CBC_GENERAL, generalMessages) << message << CoinMessageEol;
}

CbcParameters

The CbcParameters class serves primarily as a container for a vector of parameters, but can (and will) be used to store other auxiliary information that needs to be accessed by the methods of (soon-to-be) CbcSolver class. The class has methods for setting parameter values, which are pass-throughs to the methods of the CbcParam class. It also defines a [] operator, so that parameter objects contained in the parameter vector, which is a class member, can be directly accessed, e.g.,

  CbcParameters parameters;
  parameters[CbcParam::CLIQUECUTS]->setVal("ifmove");;

The constructor of the class calls a method, which sets up all the parameters (as described above). It is envisioned that different constructors will eventually be used to obtain different sets of parameters for different purposes.

CbcParamUtils

There are a number of utilities contained in the CbcParamUtils namespace. Mostly, these are the push functions that can extend the functionality of the set methods of the CbcParam class.

More Details on Input/Output mechanism

The input/output mechanism is used to pass a sequence of parameters to Cbc, including action parameters. This parameter sequence can be set equivalently passed in five different ways.

• Command line argument
• Interactive prompt
• Environment variable
• Parameter file
• Programatically from a driver

Regardless of the method, the parameter sequence is stored and accessed as a FIFO queue of strings (inputQueue). This can be passed as an input to what is currently CbcMain1 or constructed within the main while loop by parsing either interactive input, the contents of an environment variable, or the contents of a parameter file.

The main while loop then pops strings off the input queue, looks them up in the parameters list and deals with the result.

Restoring the old interface

To restore the old interface, simply copy files from src/Attic to src in both Clp and Cbc. To do so, invoke the following commands from the root of a coinbrew checkout.

echo "copying Clp files from Clp/src/Attic"
cp Clp/src/Attic/ClpMain.cpp Clp/src
cp Clp/src/Attic/ClpSolver.cpp Clp/src
echo "copying Cbc files from Cbc/src/Attic"
cp Cbc/src/Attic/Cbc_C_Interface.cpp Cbc/src
cp Cbc/src/Attic/CbcSolver.hpp Cbc/src
cp Cbc/src/Attic/CbcSolverHeuristics.cpp Cbc/src
cp Cbc/src/Attic/CoinSolve.cpp Cbc/src
cp Cbc/src/Attic/CbcSolverHeuristics.hpp Cbc/src
cp Cbc/src/Attic/unitTestClp.cpp Cbc/src
cp Cbc/src/Attic/CbcSolver.cpp Cbc/src

To restore the new version just do git reset --hard HEAD in both the Cbc and Clp subdirectories.

Obtaining Cbc master branch prior to refactoring

coinbrew fetch Cbc@before-refactor

[svigerske]

I'm trying to update some Cbc interface code that was processing the Cbc parameters.
There are two options that I had before and seem to have vanished:

1. costStrategy: I see it mentioned in the CbcParamCode enum in CbcParam.hpp, but it does not seem to appear in the vector of parameters I get from CbcParameters::paramVec().
2. pivotAndComplement: I see it mentioned in the CbcParamCode enum in CbcParam.hpp, but it does not seem to appear in the vector of parameters I get from CbcParameters::paramVec(). There is code about it in CbcParamUtils.cpp that is commented out with an #if 0.

Are these options going to be completely removed, or will they come back?

[jjhforrest]

Stefan,

On 18/01/2022 05:33, Stefan Vigerske wrote: I'm trying to update some Cbc interface code that was processing the Cbc parameters. There are two options that I had before and seem to have vanished: 1. |costStrategy|: I see it mentioned in the |CbcParamCode| enum in |CbcParam.hpp|, but it does not seem to appear in the vector of parameters I get from |CbcParameters::paramVec()|.

This seems to have been renamed as branchStrategy.

1. |pivotAndComplement|: I see it mentioned in the |CbcParamCode| enum in |CbcParam.hpp|, but it does not seem to appear in the vector of parameters I get from |CbcParameters::paramVec()|. There is code about it in |CbcParamUtils.cpp| that is commented out with an |#if 0|.

This does not seem to be in stable either - I can't find the code anywhere in last few years.  Also looking at pivotAndFix, that exists and can be called but does not seem to do anything with default defines - and if you modify defines there is coding such as -   std::cout << "Lucky you! You're in the Pivot-and-Fix Heuristic" << std::endl; which would imply that this is not finished code - I will look further.

Are these options going to be completely removed, or will they come back?

John

[svigerske]

OK, thanks.

Looks like the Pivot-and-Complement heur only has (or had) a parameter:
https://github.com/coin-or/Clp/blob/stable/1.17/Clp/src/CbcOrClpParam.cpp#L3207-L3217

But the parameter has no effect. There is a disabled include of CbcHeuristicPivotAndComplement.hpp and some code that only compiles if JJF_ZERO is defined.

If Pivot-and-Fix does not actually exist, it may make sense to remove its parameter, too.