Adding Bayesian Optimization

[AnthoneyGriffith]

---

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

Closes #2124

What are the significant changes in functionality due to this change request?

This change request adds a new optimizers node 'BayesianOptimizer', which allows for the specification of a Bayesian
Optimization algorithm. This method also requires the specification of a GPR ROM under the Models node. The GPR ROM node is also extended to include custom kernel specification and anisotropic kernels. These changes are intended to improve the use of the Bayesian Optimizer.

---

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

• 1. Review all computer code.
• 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
• 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
• 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
• 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
• 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
• 7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
• 8. If an analytic test is changed/added is the the analytic documentation updated/added?
• 9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

[moosebuild]

All jobs on 15eb19c : canceled by @wangcj05

There are some libraries conflicts right now that will causes test failure, try it latter.

[wangcj05]

Test failed on some linux machine:

@AnthoneyGriffith Could you take a look?

[wangcj05]

@joshua-cogliati-inl @PaulTalbot-INL @dylanjm This PR is ready for review. Are you available to review this PR?

[moosebuild]

Job Test Ubuntu 16 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test Fedora 31 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test CentOS 8 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test CentOS 7 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test Fedora 32 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test Ubuntu 20-2 Optional on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test Ubuntu 18-2 Python 3 on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[moosebuild]

Job Test qsubs sawtooth on 26fbeb0 : canceled by @joshua-cogliati-inl

stuck running HERON tests

[joshua-cogliati-inl]

The following tests fail the XSD check ( ./developer_tools/validate_xml.sh ):

-----------------------------------------------------------------------
tests/framework/Optimizers/BayesianOptimizer/Basic/BayesianBasic.xml
tests/framework/Optimizers/BayesianOptimizer/MishraBirdConstrained/BayesianMishraBirdConstrained.xml
tests/framework/Optimizers/BayesianOptimizer/Matyas/BayesianMatyas.xml

developer_tools/XSDSchemas/Optimizers.xsd will probably need to be modified.

[dylanjm]

Reviewed most of it, will review again after some changes are made. I'm open to discussion on any suggested change.

[moosebuild]

Job Test qsubs sawtooth on 52dec74 : invalidated by @joshua-cogliati-inl

failed in fetch

[moosebuild]

Job Test CentOS 7 on cca7e06 : invalidated by @joshua-cogliati-inl

raven updated.

[moosebuild]

Job Test Ubuntu 18-2 Python 3 on 8f948d8 : invalidated by @joshua-cogliati-inl

killed long running ipopt

[moosebuild]

Job Test qsubs sawtooth on 8f948d8 : invalidated by @joshua-cogliati-inl

Caused by HPC problem: [HPC #18342]

Comments Addressed