GPMSA Example: Brian Williams' linear verification #580

briadam · 2017-05-05T22:42:50Z

@roystgnr @dmcdougall @brianw525 This starts to address #579 by implementing Brian's test problem in a QUESO example. It runs and doesn't crash (issues a bunch of warnings from one particular spot though.)

In terms of your development workflow, I wasn't sure if you want this to be managed on a branch until the test is verified vs. merged to devel. Feel free to guide me as to how to work.

Add initial version of Bayes linear verification problem due to Brian
Williams

[1] Adams, B.M., Coleman, Kayla, Hooper, Russell W., Khuwaileh, Bassam
A., Lewis, Allison, Smith, Ralph C., Swiler, Laura P., Turinsky, Paul
J., Williams, Brian J., "User Guidelines and Best Practices for CASL
VUQ Analysis Using Dakota, Sandia Technical Report 2016-11614 (also
CASL-U-2016-1233-000), September 2016.

Add initial version of Bayes linear verification problem due to Brian Williams [1] Adams, B.M., Coleman, Kayla, Hooper, Russell W., Khuwaileh, Bassam A., Lewis, Allison, Smith, Ralph C., Swiler, Laura P., Turinsky, Paul J., Williams, Brian J., "User Guidelines and Best Practices for CASL VUQ Analysis Using Dakota, Sandia Technical Report 2016-11614 (also CASL-U-2016-1233-000), September 2016.

codecov · 2017-05-05T23:02:56Z

Codecov Report

Merging #580 into dev will increase coverage by 0.17%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev     #580      +/-   ##
==========================================
+ Coverage   71.15%   71.33%   +0.17%     
==========================================
  Files         167      167              
  Lines       14457    14453       -4     
==========================================
+ Hits        10287    10310      +23     
+ Misses       4170     4143      -27

Impacted Files	Coverage Δ
src/core/src/GslVector.C	`80.48% <0%> (-0.08%)`	⬇️
src/core/src/GslMatrix.C	`85.33% <0%> (-0.04%)`	⬇️
...c/core/src/InfiniteDimensionalMCMCSamplerOptions.C	`3.44% <0%> (ø)`	⬆️
src/stats/src/MLSamplingOptions.C	`56.41% <0%> (ø)`	⬆️
src/basic/src/ScalarFunction.C	`76.59% <0%> (ø)`	⬆️
src/stats/src/MLSamplingLevelOptions.C	`89.55% <0%> (ø)`	⬆️
src/misc/src/Miscellaneous.C	`85.71% <0%> (+12.44%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e77509d...d1fadab. Read the comment docs.

briadam · 2017-05-15T17:28:09Z

Spoke with @roystgnr who suggested if possible we make some or all of this a test, with XFAIL status if needed. While the ultimate test @brianw525 has in mind likely won't be feasible for this (requires 100k or 250k samples) and is why I targeted examples/, we can make a scaled down version to use as at least a regression test.

dmcdougall · 2017-05-26T22:24:15Z

Do you want me to merge this example or wait for a test?

briadam · 2017-05-31T19:25:35Z

Yes, but I defer to you all as to what's appropriate in the project. One of my goals prior to our next in-person meeting is to implement a few of Brian Williams' verification cases so we can actually iterate on working code in the meeting and run cases. Since the cases have to run many samples, perhaps for now they belong as examples and not tests and that's just okay? Also, since I'll be adding a few more cases similar to this, I am keenly interested in feedback on my implementation before I copy/paste and make other cases, but that's not required.

dmcdougall

You can add your example to this file if you want it to be easily executed from our run_all.sh script.

Caveat other remarks, this looks fine to me. Thanks!

dmcdougall · 2017-06-06T18:41:33Z

examples/gp/linear_verif/gpmsa_input.txt

+###############################################
+#ip_mh_help                 = anything
+ip_mh_dataOutputFileName   = outputData/sipOutput
+ip_mh_dataOutputAllowedSet = 0 1


Vectors need single quotes around them ('0 1') becausedev uses GetPot to parse input files, and not boost. You can still use boost if you want, but the default is GetPot.

dmcdougall · 2017-06-06T18:41:56Z

examples/gp/linear_verif/gpmsa_input.txt

+ip_mh_rawChain_displayPeriod        = 2000
+ip_mh_rawChain_measureRunTimes      = 1
+ip_mh_rawChain_dataOutputFileName   = outputData/ip_raw_chain
+ip_mh_rawChain_dataOutputAllowedSet = 0 1


Needs quotes.

dmcdougall · 2017-06-06T18:42:07Z

examples/gp/linear_verif/gpmsa_input.txt

+ip_mh_tk_useLocalHessian            = 0
+ip_mh_tk_useNewtonComponent         = 0
+ip_mh_dr_maxNumExtraStages          = 0
+ip_mh_dr_listOfScalesForExtraStages = 5. 10. 20.


Needs quotes.

dmcdougall · 2017-06-06T18:44:38Z

examples/gp/linear_verif/gpmsa_linear_verif.C

@@ -0,0 +1,345 @@
+// Bayes linear verification problem due to Brian Williams


Can you put the QUESO licence before this? You can copy it from one of the other examples.

briadam · 2017-06-06T19:19:34Z

Thanks for the helpful feedback. Brian Williams updated this example and gave me several others, so I'm thinking I'll also reorganize the directories and do some cleanup, maybe even build some utilities that can be shared among the cases, before we merge...

dmcdougall · 2017-06-06T21:30:54Z

No worries. Let me know when you're ready to pull the trigger.

@dmcdougall

Address feedback from @dmcdougall in libqueso#580 * Add license * Quote vectors in text input file

Refactor linear verification data read to work for both scalar and multivariate cases (number of responses > 1). Rename input and data files in preparation for adding multivariate case.

Minor rework of main to drive scalar vs. multivariate cases.

@brianw525

Add multivariate five-response linear verification case (example toggles based on input filename for now), including observation error covariance matrix. Case is almost same as scalar, but just copied the code for now to make debugging more straightforward. Also add comments with @brianw525 recommended changes to priors and initial point (tried changing them, but then scalar case doesn't run)

dmcdougall · 2017-06-13T17:54:54Z

examples/gp/linear_verif/gpmsa_linear_verif.C

@@ -11,7 +11,7 @@
 // Public License as published by the Free Software Foundation.
 //
 // This library is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// but WITHOUT ANY WARRANT Y; without even the implied warranty of


please fix thx

dmcdougall · 2017-06-13T18:23:16Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+	   outputVecs, experimentScenarios, experimentVecs);
+
+  // Experiment observation error
+  for (unsigned int i = 0; i < numExperiments; i++)


numExperiments should be numExperiments * experimentSize in the general case.

dmcdougall · 2017-06-13T18:26:21Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+
+  // Experiment observation error
+  for (unsigned int i = 0; i < numExperiments; i++)
+    (*experimentMat)(i, i) = std::pow(0.05, 2.0);


If we're normalising the experiments by the mean and variance of the simulations, we also need to normalise this matrix by the simulation variance.

Since this is a matrix, we'd pre- and post-mulitply by 1/sqrt(diag(S)) where division is done component-wise.

dmcdougall · 2017-06-13T18:37:10Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+  // paramInitials[6]  = std::exp(-0.025);  // emul corr strength (scenario/beta?)
+  // paramInitials[7]  = std::exp(-0.025);  // emul corr strength (scenario/beta?)
+  // paramInitials[8]  = std::exp(-0.025);  // emul corr strength (scenario/beta?)
+  // paramInitials[9]  = std::exp(-0.025);  // emul corr strength (scenario/beta?)


The commented out code is correct. Please replace elements 4 through 9 above.

dmcdougall · 2017-06-13T18:37:42Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+  // paramInitials[9]  = std::exp(-0.025);  // emul corr strength (scenario/beta?)
+
+  paramInitials[10] = 1000.0;   // discrepancy precision
+  // paramInitials[10] = 20.0;        // discrepancy precision


Ditto. 20 should be the default and for the verification.

dmcdougall · 2017-06-13T18:37:55Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+  paramInitials[13] = 0.1;      // discrepancy corr strength x3
+  // paramInitials[11] = std::exp(-0.025);      // discrepancy corr strength x1
+  // paramInitials[12] = std::exp(-0.025);      // discrepancy corr strength x2
+  // paramInitials[13] = std::exp(-0.025);      // discrepancy corr strength x3


dmcdougall · 2017-06-13T18:38:31Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+  // paramInitials[13] = std::exp(-0.025);      // discrepancy corr strength x3
+
+  paramInitials[14] = 10000.0;  // emul data precision
+  // paramInitials[14] = 1000.0;  // emul data precision


dmcdougall · 2017-06-13T18:52:05Z

examples/gp/linear_verif/gpmsa_linear_verif.C

+  paramInitials[5]  = 1.0;      // weights0 precision
+  paramInitials[6]  = 1.0;      // weights0 precision
+  paramInitials[7]  = 1.0;      // weights0 precision
+  paramInitials[8]  = 1.0;      // weights0 precision


By this point we already know how many basis vectors there are. QUESO just needs to expose enough to let the user query it.

@brianw525

* Use @brianw525 recommended prior and initial point settings. * Scale experiment observation error covariance by simulation variance.

roystgnr · 2017-06-13T20:24:20Z

examples/Makefile.am

+gpmsa_linverif_CPPFLAGS = -I$(top_srcdir)/examples/gp/linear_verif $(AM_CPPFLAGS)
+
+dist_gpmsa_linverif_DATA =
+dist_gpmsa_linverif_DATA += ${gpmsa_scalar_SOURCES}


This installs gpmsa_scalar.C, not gpmsa_linear_verif.C

Minor edits per reviewer feedback

dmcdougall requested changes Jun 6, 2017

View reviewed changes

briadam added 4 commits June 9, 2017 18:09

GPMSA linear verification (scalar): address feedback

2d8d9d1

Address feedback from @dmcdougall in libqueso#580 * Add license * Quote vectors in text input file

GPMSA linear verification: refactor for multi-response

bd48d48

Refactor linear verification data read to work for both scalar and multivariate cases (number of responses > 1). Rename input and data files in preparation for adding multivariate case.

GPMSA linear verification: setup driver for scalar vs MV

8fac568

Minor rework of main to drive scalar vs. multivariate cases.

dmcdougall requested changes Jun 13, 2017

View reviewed changes

GPMSA verification: address review @dmcdougall comments

ca12433

* Use @brianw525 recommended prior and initial point settings. * Scale experiment observation error covariance by simulation variance.

roystgnr reviewed Jun 13, 2017

View reviewed changes

GPMSA verification: fix install rule, loop ordering

d1fadab

Minor edits per reviewer feedback

dmcdougall merged commit f63f6bd into libqueso:dev Jun 13, 2017

roystgnr mentioned this pull request Aug 1, 2017

Adding large GPMSA verification example #615

Merged

briadam deleted the example_linear_verif branch September 29, 2017 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPMSA Example: Brian Williams' linear verification #580

GPMSA Example: Brian Williams' linear verification #580

briadam commented May 5, 2017

codecov bot commented May 5, 2017 •

edited

Loading

briadam commented May 15, 2017

dmcdougall commented May 26, 2017

briadam commented May 31, 2017

dmcdougall left a comment

dmcdougall Jun 6, 2017

dmcdougall Jun 6, 2017

dmcdougall Jun 6, 2017

dmcdougall Jun 6, 2017

briadam commented Jun 6, 2017

dmcdougall commented Jun 6, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

dmcdougall Jun 13, 2017

roystgnr Jun 13, 2017

		@@ -0,0 +1,345 @@
		// Bayes linear verification problem due to Brian Williams

GPMSA Example: Brian Williams' linear verification #580

GPMSA Example: Brian Williams' linear verification #580

Conversation

briadam commented May 5, 2017

codecov bot commented May 5, 2017 • edited Loading

Codecov Report

briadam commented May 15, 2017

dmcdougall commented May 26, 2017

briadam commented May 31, 2017

dmcdougall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

briadam commented Jun 6, 2017

dmcdougall commented Jun 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 5, 2017 •

edited

Loading