Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work on making a pip installable HERON package. #216

Merged
merged 19 commits into from
Oct 13, 2022

Conversation

joshua-cogliati-inl
Copy link
Collaborator

@joshua-cogliati-inl joshua-cogliati-inl commented Oct 11, 2022


Pull Request Description

What issue does this change request address?

idaholab/raven#1764

What are the significant changes in functionality due to this change request?

Allow running parts of HERON in pip installed mode.


For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

  • 1. Review all computer code.
  • 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
  • 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
  • 4. Automated Tests should pass.
  • 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large tes.
  • 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
  • 7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
  • 8. If an analytic test is changed/added, the the analytic documentation must be updated/added.
  • 9. If any test used as a basis for documentation examples have been changed, the associated documentation must be reviewed and assured the text matches the example.

@joshua-cogliati-inl
Copy link
Collaborator Author

Requires idaholab/raven#1994

with warnings.catch_warnings():
warnings.simplefilter('ignore')
rom = ravenROM(fpath, raven_loc).rom
try:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that this method gets called repeatedly in the DispatchManager.py for each ARMA realization. I was able to reduce quite a significant amount of memory consumption by memoizing this function. Instead of making a whole new PR, it might be worth to just add here. All you need to do is:

at the top in the imports:

from functools import cache

and then add the decorator to the function:

@cache
get_synthhist_structure

Copy link
Collaborator

@PaulTalbot-INL PaulTalbot-INL Oct 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhh, because RAVEN runs this each time instead of once ever ... maybe Mohammad's work will fix this to some degree. Good catch.

Does cacheing fix this if parallelized?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed: 764ce1c

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty should this cache is per process, so non-shared memory parallelism would have to cache it multiple times.

@moosebuild
Copy link
Collaborator

Job CentOS 8 on 764ce1c : invalidated by @joshua-cogliati-inl

Updated RAVEN to add ravenframework.ROMExternal

@@ -21,7 +21,8 @@
# allows pyomo to solve on threaded processes
import pyutilib.subprocess.GlobalData
pyutilib.subprocess.GlobalData.DEFINE_SIGNAL_HANDLERS_DEFAULT = False
from pyutilib.common._exceptions import ApplicationError
#from pyutilib.common._exceptions import ApplicationError
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented code can be removed.

Copy link
Collaborator

@dylanjm dylanjm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one change and one question. Otherwise changes look good to me.

@@ -106,7 +106,7 @@
</Components>

<DataGenerators>
<ARMA name='Price' variable="Signal">%HERON%/tests/integration_tests/ARMA/Sine/arma.pk</ARMA>
<ARMA name='Price' variable="Signal">%BASE_WORKING_DIR%/../../../ARMA/Sine/arma.pk</ARMA>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious about the purpose this change.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, this is for testing heron from a pip install, and in a pip install, %HERON% may not be anywhere near the tests directory, so I changed it to be relative to the base working directory. At some point all the tests will have to be changed to do this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So after HERON is pip installed, we can:

cd tests/integration_tests/workflows/production_flex
heron heron_input.xml
raven_framework outer.xml

Copy link
Collaborator

@dylanjm dylanjm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good and don't affect major functionality.

@dylanjm dylanjm merged commit 4189682 into idaholab:devel Oct 13, 2022
@joshua-cogliati-inl joshua-cogliati-inl deleted the package_changes branch October 13, 2022 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants