Rw correlated arma #650

PaulTalbot-INL · 2018-05-16T16:37:50Z

This PR can wait until the rework branch is merged into devel, assuming that doesn't take too long.

Splits the superVisedLearning module into its constituent components in separate files
Allows the ARMA to train multiple independent targets.
Implements a VARMA for training correlated targets.
Adds the "statsmodels" python library to our standard package.

Tests for all of the above.

Closes #703.

* convert metric pp to use new data objects * clean up

* edits * edits * comments review-1 * edits * asDictionary works * added full loop cycle test

…into alfoa/dataobject-rework_poly_sum_exponential

* adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * added cluster labeling to outstream print, will be tested with optimizer * optimizers working, but need to fix reading from CSV before merging * stash * fixed prefix as numpy array * loading CSV correctly now, so optimizer can find optimal solns * cleanup * now handling infs and nans * cleanup * added the option to avoid to reprint the files in case their content did not change * reverted 101 tests modified by the conversion scripts (by mistake) * reverted other tests * reverted tests user guide * reverted all the other tests * Update beale.xml removed ```"``` * removed " * removed abstract method that has been removed from the XDataSet * fixed with respect to the new prefix strategy * fixed loading of dataset in case no metadata are found in the xml (finally statement in a try except is always executed) * fixed type in ROM causing a failure * fixed another typo that was making the Dummy.py model to fail * removed whitespace that was added from another commit * updated to use pd.isnull instead of np.isnan in unordered CSV differ * test files for differ

…//github.com/idaholab/raven into alfoa/dataobject-rework_poly_sum_exponential

* fix typo in rom * modify tests * convert cross validation pp * convert tests for cross validation pp * clean up * keep the options to dump the cross validation results for each fold * update conversion scripts for cross validation

* fixed history set reading from CSV if no XML present, or if present for that matter * fixed up PointSet and DataSet versions, too.

* typing is now working, but if more samples are taken after asDataset is called, and asDataset is called again, integers will be converted into floats. I submitted a stackOverflow question about this, and we will see. * fixed integer preservation using concat instead of merge

* added hierarchical methods * fixed multiline in databases * added sanity assertion * stash * mechanics in place for hierarchical * hierarchal working with collector OR data * a little more testing

* stash * stash * stash * topologicals working * cleanup * review comments

* convert importance rank pp to use the new data objects * fix comments

…olab#469) * fixed return code * starting reworking database * ok * ok * ok * ok * ok * moving ahead * speeded up group search and allgroup list * fixed subgroup * almost done * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * fixed time dep ROMs * addressed Paul's comments * fixed plot * fixed other 2 tests * fixed other 3 tests * regolded another test * initial edits * fixed write...but the history set does not work yet * fixed return code * Update README.md (idaholab#439) just removed few sentences * unit tests passing again * slicing now works * added abstract methods to object base * starting reworking database * histories working: * switched to comprehension * fname to fName * removing variables works * added addReal check * Correct setting of default pivots for history set (idaholab#448) * fixed default pivot param setting to be in order * put 'time' back in since we handle that now * ok * ok * ok * ok * edits * ok * moving ahead * speeded up group search and allgroup list * fixed subgroup * progressive CSV writing with appends instead of rewriting each iteration for multiruns * cleanup * almost done * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * fixed time dep ROMs * check data alignment works (idaholab#452) * fixed no-scalar bug, albeit in a non-traditional way. (idaholab#455) * addressed Paul's comments * pylint problems addressed (idaholab#451) * pylint problems addressed * Update TestXPointSet.py * malformed comment blocks in 2 tests * new unordered CSV differ, tested on test_Lorentz * convert code to use new data objects (idaholab#457) * initial convert of code for new data object * fix bugs in dataset and continue convert code for accepting the new data object * additional converting * clean up * Sampler Restarts (idaholab#458) * started work * sampler restarts working * Adaptive Samplers (plus Dummy IO fix) (idaholab#459) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * fixed prefix as numpy array * convert basic statistics pp to use the new data object (idaholab#460) * convert basicStatistics to use the new data objects * convert tests of basicStatisitics * convert more tests * clean up * move addMetaKeys to localInputAndChecks * resolve comments * fix checkIndexAlignment in DataSet * add unit test for checkIndexAlignment * Closes idaholab#464 * added test * convert metric pp to use new data objects (idaholab#462) * convert metric pp to use new data objects * clean up * fixed the tests * added the new gold file for the new test * Mods for new dataObject (idaholab#463) * edits * edits * comments review-1 * edits * asDictionary works * added full loop cycle test * fixed plot * fixed other 2 tests * fixed other 3 tests * Update TestXDataSet.py * Update Dummy.py * other tests converted * readded deleted file * almost done * ok * fixed tests * Fixed LimitSurface Postprocessors, added conversion script, added documentation * ok * adapted LimitSurfaceSearch sampler and moved directory for more clearity * almost finished safest point * ok * fixed safest point PP * fixed adaptive batch * modified documentation for SafestPoint * removed Outputhold for Optimizer since is not used and not working * fixed metadata fro samples * removed all * modified unit test * addressed Diego's comments * convert tests (idaholab#480) * removed commented part in user manual * addrresed Conjiang's * remove .DS_Store * convert external post processor to use new DataObjects (idaholab#479) * convert external pp to use the new data objects * regold tests * fix the DataObjects * address comments * starting reworking database * ok * ok * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * addressed Paul's comments * fixed plot * fixed other 2 tests * fixed other 3 tests * regolded another test * unit tests passing again * slicing now works * starting reworking database * histories working: * removing variables works * ok * ok * progressive CSV writing with appends instead of rewriting each iteration for multiruns * cleanup * finished h5py_interface * working on IOStep * dfloat * ok finished database * finished not hirearchical HDF5 and fixed static ROMs * check data alignment works (idaholab#452) * fixed no-scalar bug, albeit in a non-traditional way. (idaholab#455) * pylint problems addressed (idaholab#451) * pylint problems addressed * Update TestXPointSet.py * malformed comment blocks in 2 tests * new unordered CSV differ, tested on test_Lorentz * Adaptive Samplers (plus Dummy IO fix) (idaholab#459) * adaptive sparse grid, sobol working * Adaptive sampling, plus Dummy-based rlz updates from input, output * cleanup * fixed prefix as numpy array * convert basic statistics pp to use the new data object (idaholab#460) * convert basicStatistics to use the new data objects * convert tests of basicStatisitics * convert more tests * clean up * move addMetaKeys to localInputAndChecks * resolve comments * fix checkIndexAlignment in DataSet * add unit test for checkIndexAlignment * Closes idaholab#464 * convert metric pp to use new data objects (idaholab#462) * convert metric pp to use new data objects * clean up * fixed the tests * fixed other 2 tests * other tests converted * readded deleted file * almost done * ok * fixed tests * Fixed LimitSurface Postprocessors, added conversion script, added documentation * ok * adapted LimitSurfaceSearch sampler and moved directory for more clearity * almost finished safest point * ok * fixed safest point PP * fixed adaptive batch * modified documentation for SafestPoint * removed Outputhold for Optimizer since is not used and not working * fixed metadata fro samples * addressed Diego's comments * removed commented part in user manual * addrresed Conjiang's * remove .DS_Store * convert external post processor to use new DataObjects (idaholab#479) * convert external pp to use the new data objects * regold tests * fix the DataObjects * address comments

* merged Diego's commits for InterfacePostProcessor * Fixed OutStreams printing and regolder foulty ROMs results * modified ARMA tests * fixed reading from CSV of HistorySet (correctly considering the pivotParameter) * fixed * TEMPORARY fix of printing of HISTORY SET * fixed ROM, ARMA, DATABASE (loader), Multiple usage of database * fixed multi target rom and rom trainer test * added the possibility to ask for unfolded XArrays in the realization(...) method * ok * modified other two tests * addressed Paul and Congjiang's comments (no interface PP related) * removed comment that does no apply anymore in HistorySet * fixed another test database * addressed comments * addressed comments 2 * edits * fixed test file

moosebuild · 2018-07-16T20:00:22Z

Job Test mac on 2cc6653 : invalidated by @PaulTalbot-INL

unusual build error

PaulTalbot-INL · 2018-07-17T18:53:06Z

Due to changes needed referenced in #597, the VARMA tests only the creation of the outputs; the difference even between statistics on two different operating system VARMAs are significant, and need to be controlled through independent RNG.

UPDATE: Controlling the measurement and state shocks, we can check the statistics but not the individual samples, which is better than just outputs.

…arma noise generation I think

alfoa

Minimal changes requested

alfoa · 2018-07-23T18:04:19Z

framework/Distributions.py

    """
      Function to get random numbers
-      @ In, args, dict, args
+      @ In, size, int, number of entries to return


@ In, size, int, optional, number of entries to return. If None, only a single random number is returned

alfoa · 2018-07-23T18:04:46Z

framework/Distributions.py

      rvsValue = self.ppf(random())
    else:
-      rvsValue = [self.rvs() for _ in range(args[0])]
+      # TODO to speed up, do this on the C side instead of in python


If you remind me this, I will add this in the next crow touch

I thought about this; I'll keep it in mind.

alfoa · 2018-07-23T18:08:30Z

framework/SupervisedLearning/DynamicModeDecomposition.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+  Created on May 8, 2018


Hey Paul,
Can you report here that these files comes from the split of the original SupervisedLearning.py module and author and date (if retrievable) of the initial implementation? ( just to keep track of the "maturity" of the "ROM")

alfoa · 2018-07-23T18:09:27Z

few changes...not big deal

alfoa

good to go

alfoa · 2018-07-23T22:54:25Z

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code. done
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts). done
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details. ok
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass. they pass
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True. added tests
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync. n/a
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done. Correlated ARMA #703
8. If an analytic test is changed/added is the the analytic documentation updated/added? n/a

alfoa · 2018-07-23T22:55:19Z

Checklist passed... Issue closure checklist passed...

Good to go.

Merging...

alfoa and others added 30 commits December 7, 2017 16:49

first draft

b12cbdf

convert metric pp to use new data objects (idaholab#462)

65f877c

* convert metric pp to use new data objects * clean up

Mods for new dataObject (idaholab#463)

8e4f5d3

* edits * edits * comments review-1 * edits * asDictionary works * added full loop cycle test

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

61ef9c5

…into alfoa/dataobject-rework_poly_sum_exponential

initial test

be55016

Merge branch 'devel' into dataobject-rework

a720f02

Merge branch 'dataobject-rework' of https://github.com/idaholab/raven …

39cd71a

…into alfoa/dataobject-rework_poly_sum_exponential

pk

08883f3

diff algorithm

9d5b335

ok

594bb3e

merge

5af1977

new input

86de7ec

fix

d37fade

ok

a4bc33f

Merge branch 'alfoa/dataobject-rework_poly_sum_exponential' of https:…

0b54106

…//github.com/idaholab/raven into alfoa/dataobject-rework_poly_sum_exponential

Fixing no-meta CSV reading (idaholab#473)

fc7a4bc

* fixed history set reading from CSV if no XML present, or if present for that matter * fixed up PointSet and DataSet versions, too.

finished

52eefee

added tests

dcd20a8

ok

5db3a47

ok

31f6acc

added hierarchical methods (idaholab#474)

002944b

* added hierarchical methods * fixed multiline in databases * added sanity assertion * stash * mechanics in place for hierarchical * hierarchal working with collector OR data * a little more testing

D.O. Rework: Topological PP (idaholab#472)

cce9e67

* stash * stash * stash * topologicals working * cleanup * review comments

convert importance rank pp to use the new data objects (idaholab#477)

5f10826

* convert importance rank pp to use the new data objects * fix comments

convert tests (idaholab#480)

e0071a7

addressed Congjiang's comments

f4a0014

Merge branch 'devel' into rw-correlated-arma

8dbe747

PaulTalbot-INL requested a review from alfoa June 14, 2018 21:03

PaulTalbot-INL added 3 commits July 11, 2018 13:49

removed default pivot

b6699ab

Merge branch 'devel' of github.com:PaulTalbot-INL/raven into devel

0def39f

Merge branch 'devel' into rw-correlated-arma

2cc6653

PaulTalbot-INL added 3 commits July 17, 2018 09:10

Merge branch 'devel' of github.com:PaulTalbot-INL/raven into devel

6193057

varma to statistical test

de64f14

mergefix

34f8089

PaulTalbot-INL added 12 commits July 17, 2018 12:53

varma tests outputs only, needs independent rng

50e7604

updated test to use controlled correlated data, and now controlling v…

a8aea55

…arma noise generation I think

gold file

5ebab37

fixed RNG for initial state

ae98469

updated for increased bins

fb9b7ff

stats test for varma

f1daf19

more realistic test metric

a99947d

longer history for better statistics

0bd1fa7

changes to split supervised learning to keep up with PR 674

f5be577

Merge branch 'devel' of github.com:idaholab/raven into devel

22e9e17

merged in devel after idaholab#674

5e9d90a

typo

749ce9c

alfoa requested changes Jul 23, 2018

View reviewed changes

updated module docstrings

cd8fd30

alfoa approved these changes Jul 23, 2018

View reviewed changes

alfoa merged commit 0878010 into idaholab:devel Jul 23, 2018

PaulTalbot-INL deleted the rw-correlated-arma branch April 8, 2021 20:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rw correlated arma #650

Rw correlated arma #650

PaulTalbot-INL commented May 16, 2018 •

edited

Loading

moosebuild commented Jul 16, 2018

PaulTalbot-INL commented Jul 17, 2018 •

edited

Loading

alfoa left a comment

alfoa Jul 23, 2018

PaulTalbot-INL Jul 23, 2018

alfoa Jul 23, 2018

PaulTalbot-INL Jul 23, 2018

alfoa Jul 23, 2018

PaulTalbot-INL Jul 23, 2018

alfoa commented Jul 23, 2018

alfoa left a comment

alfoa commented Jul 23, 2018 •

edited

Loading

alfoa commented Jul 23, 2018

Rw correlated arma #650

Rw correlated arma #650

Conversation

PaulTalbot-INL commented May 16, 2018 • edited Loading

moosebuild commented Jul 16, 2018

PaulTalbot-INL commented Jul 17, 2018 • edited Loading

alfoa left a comment

Choose a reason for hiding this comment

alfoa Jul 23, 2018

Choose a reason for hiding this comment

PaulTalbot-INL Jul 23, 2018

Choose a reason for hiding this comment

alfoa Jul 23, 2018

Choose a reason for hiding this comment

PaulTalbot-INL Jul 23, 2018

Choose a reason for hiding this comment

alfoa Jul 23, 2018

Choose a reason for hiding this comment

PaulTalbot-INL Jul 23, 2018

Choose a reason for hiding this comment

alfoa commented Jul 23, 2018

alfoa left a comment

Choose a reason for hiding this comment

alfoa commented Jul 23, 2018 • edited Loading

For Change Control Board: Change Request Review

alfoa commented Jul 23, 2018

PaulTalbot-INL commented May 16, 2018 •

edited

Loading

PaulTalbot-INL commented Jul 17, 2018 •

edited

Loading

alfoa commented Jul 23, 2018 •

edited

Loading