RMG Improvement Proposal [RIP]: Make ReactionMechanismSimulator.jl Optional [COMMENTS WELCOME!]

[JacksonBurns]

Important

Please read and comment on this issue if you are a developer or user of RMG - we need lots of input!

The purpose of this issue is to centralize discussion and hopefully reach a compromise surrounding a topic which has been discussed throughout many issues, pull requests, and offline meetings - whether ReactionMechanismSimulator.jl and its associated Julia dependencies should be required to run RMG.

This issue is styled after the Python Enhancement Proposal (see PEP 733 for an example), thus the name 'RMG Improvement Proposal', or RIP for short. This RIP will attempt to:

• provide a brief summary of all of the below points
• introduce the perceived issue
• discuss the drawbacks of the proposed change
• discuss the benefits of the proposed change
• discuss the drawbacks of the proposed change

Summary

• RMS and its associated Julia dependencies are currently required to install and run RMG
• This requirements has created installation issues and runtime issues for both users and developers
• A pull request is in place which would make RMS optional and resolve these
• There are drawbacks to this approach and a suggested alternative which would merely avoid some Julia operations but not allow RMS to be optional

Introduction

As of writing, RMG-Py currently requires RMS and its associated dependencies to be installed in order to run anything in RMG. This is because (a) RMG can optionally interface with RMS to use its reactors and (b) Arkane optionally uses Julia's differential equations solver.

Reasons to Require RMS

I'm attempting to summarize comments from #2631 (especially here) and elsewhere. Please leave comments replying to any of these points, or add your own, if you think I missed or misrepresented anything.

The 'Batteries Included' Ideology

In general Python adheres to a "Batteries Included" thinking - everything that you could possibly need should come included without additional installation. By including RMS with RMG by requirement, we are avoiding the questions of "what features are available if I add RMS?" and the like.

Counterpoint

We can continue to ship the Docker container with 'batteries included' (all dependencies) but allow for RMS-less source or binary installs. The CI will also continue to test RMS and future PRs will require RMS tests to pass, so there is no risk of incompatible code being merged.

Promised Features in Papers are only in RMS

Some recent papers coming out of our groups describe features which are only available via RMS. It makes sense that the default installation of RMG should therefore provide these features. It's also an additional layer of 'protection' for these features to make them required by default - by making them required and continuing to interact with them, we stand a better chance of them continuing to work into the future.

Simplifying Debugging by Forcing Everyone to Install the Same Way

Offering multiple installation setups can lead to more complicated bug reports. If users install with our without Julia they may end up running a different set of unit tests, for example. This is a similar line of thinking to why we moved to Docker in the first place - unify the user experience.

Reasons to Make RMS Optional

Resolves Outstanding Issues

There are issues scattered across the @ReactionMechanismGenerator organization which are related to these dependencies that would be resolved by making them optional:

• Missing Riedel Equation Parameters for Methanol RMG-database#634 (comment) - user reported that missing reidel equation parameters would crash RMG because RMS required them even though RMS was not being used
• Arkane takes a long time to run due to loading Julia dependencies #2547 - loading Julia dependencies takes minutes to then run a few-second Arkane job that does not use Julia
• Reinstalled from source and it is very slow.  #2617 - installing Julia dependencies required "scary" manual steps beyond what we want for our users and RMG was also slower than previously

Counterpoint

These issues would also be resolved by merging part of #2631 - only the "avoid unnecessary Julia operations".

Enables Conda Binary Building

Currently, our conda binaries are significantly (~3 years) out of date because we have not been able to successfully build the rmg package since the integration of RMS. Users must either do a source install or use the Docker image.

This would be resolved by #2641 already - by making a pure-python RMG package and then having users install RMS on top of it, the conda build for RMG is massively simplified and shown to work.
The alternative w/o making RMS optional in #2636 was not successful.

Health of the RMG Software

As far back as 2021 in this discussion (#2247) we have had requests to make installing the Julia dependencies optional, citing that the features it provides, when unused, amount to bloat.
Along the same lines of thinking, extending RMG is prohibitively difficult because of the huge number of dependencies besides RMS - taking chances to make dependencies optional will help alleviate this.

This is opinionated, but the separation between RMG and RMS is clean, amounting to ~300 lines of code in https://github.com/ReactionMechanismGenerator/RMG-Py/pull/2631/files. To quote PEP 20: "Simple is better than complex", and separating these two halves is simple.

Highly Opinionated Point - Julia can be a bit of a headache

Julia is very new and consequently suffers a lot of ecosystem problems - a selection of some reported here:

• Julia restriction #2609 - we are currently limited to a specific patch of Julia because others do not work
• Revert libstdcxx-ng Patches in CI, Docs, and Dockerfile #2469 - a previous specific patch release of Julia which also did not work
• Continuous Integration, Documentation Build, and Dockerfile Fixes #2608 - Dockerfile would not build on Windows without a special Julia environment variable being thrown in
• CI for Mac frequently timing out #2453 - we can't run our CI on MacOS because the Julia installation times out for no obvious reason (almost fixed in Run Complete CI on MacOS and Ubuntu #2648)

Developing RMG with Julia suffers from problems as well:

• typo prevents writing input files #2531 - attempting to debug a problem takes extra minutes on every run because of loading Julia dependencies
• Conversion from nose to pytest #2516 (comment) - you cannot run the RMG tests in parallel because of Julia
• There are no conda binaries for Julia for the latest Apple CPUs (M1+) requiring users to instead use a compatibility layer and run the x86 code: e0e2601#diff-155d3a2dcb81a38f68cd8cda2bcebe6368b9c0677c0513541e1140dfcebd7fd7R76-R79

Acknowledgements

Thanks to @rwest for the suggestion to formalize this discussion into this format.

cc

I'd specifically like to make sure that the following people see this issue and have an opportunity to respond: @mjohnson541 @hwpang @alongd @calvinp0 @rwest @sevyharris @oscarwumit @jonwzheng @xiaoruiDong @ChrisBNEU - please see the Summary above, consider reading the full issue, and let us know your thoughts!

[rwest]

Thanks for starting, Jackson. I was hoping for a more balanced summary, that might have been written by either "side" of the debate. Would you like to try role playing someone with the other opinion? I was hoping the pros and cons wouldn't be so tied to people

[JacksonBurns]

Happy to - I need some help brainstorming though. What are some other pros of keeping RMS required?

[calvinp0]

I am not very well versed in the technical details of Python like everyone else is that are mentioned. I use RMG from time to time but uncertain whether my RMG runs are engaging RMS. However, I can say that the installation of RMS has become a little more tedious for our team. Since we run RMG on our university servers and they are CentOS (7.9.2009) based, they do not come with Zlib 1.9.2 installed. I came across the issue back in 30 October 2023 and eventually resolved it through forum searching. I know that Docker is preferred when using RMG as it will come with all the required packages and codecs but our servers don't support Docker.

Sorry, I realise that my response does not add anything to the conversion. I guess my position is that optional is preferred as users who require RMG may have to jump through hoops in order to get RMS to work. On the other hand, if there was a simpler way to install RMS and connect it to RMG that would also be good (although, I have no idea what that would look like or if there is a simpler way).

[jonwzheng]

In addition to the points mentioned above, here are some thoughts (full disclosure, I don't have experience with using RMS):

Additional points for making RMS Optional

• There have been moments in RMG's lifetime when something was broken with the Julia install and precluded me from installing RMG since I had no idea how to troubleshoot in Julia. Adding a required Julia dependency adds an additional troubleshooting step that most users won't be familiar with.
• Philosophically, RMG and RMS serve different functionalities and should be separate modules (though they are currently slightly coupled). RMG outputs can be used in other simulation software and many of our users do use Chemkin/Cantera instead of RMS...

Additional points against making RMS Optional

• ...however, we would shut off any future additional core RMG functionality from using Julia (at least without partially reverting this change). As Matt pointed out in Avoid Unnecessary Julia Operations + Optional RMS Installation #2631 this would currently include additional reactor types and RMG electrochemistry, among others. However I am not sure of the extent to which anyone is/plans on adding any Julia code to the core RMG functionality, maybe the other devs can comment on this?
• The proposed requirement of requiring the user to un-comment the required installs in the .yml is awkward. Ideally the optional install can be a oneliner from the CLI like with pip. I did see that there is some recent discussion in the conda dev team to make this a feature (see Optional groups of dependencies conda/conda#7502) so maybe this will be alleviated soon.
• Is there potential to affect unit testing? You say "there is no risk of incompatible code being merged", but I could imagine an oddball case where we add a Julia/RMS feature that somehow is erroneously called in the RMG core code, so we would need CI to test both the Julia-less and Julia-including installs?

Overall thoughts

I lean toward making the RMS install optional to separate the Julia components as much as possible, but I would like to ensure that (1) our messaging of this separation to our users is very clear, (2) we have error catching early on so users will quickly realize they need to upgrade their install for their desired use cases (rather than having to wait to the end of a mechanism generation cycle), (3) the optional install process is easy and its inclusion does not overly complicate the testing and development process.

[sevyharris]

Thanks, Jackson, for getting the discussion started/organized.

This is mostly a repeat of what's already been said, but here are my thoughts on the issue:

For RMS Optional

• It's a huge pain to debug any part of RMG that loads Julia/RMS. I don't have the patience to wait 10 minutes every time I want to modify the code and see how it changes things. This has been a huge headache in general.
• I think including Julia dependencies adds a lot of complexity to development and maintenance and that shouldn't be underestimated. It's not just one more thing to install. It's one more potential point of failure and a whole new language to learn if you want to take a stab at debugging it. I'm not familiar enough with the RMS features to judge whether they're worth that extra price of complexity, but it is a price.

Against RMS Optional

• If we could optionally turn off the RMS imports, that would alleviate most of the pain in debugging, and I believe Jackson's Avoid Unnecessary Julia Operations + Optional RMS Installation #2631 takes care of that.
• I like the "batteries included" philosophy and I think any feature we've mentioned in the latest RMG release paper really should come standard in the latest RMG install. Correct me if I'm wrong but I think the liquid reactors use RMS?

[JacksonBurns]

@calvinp0 thank you for the input! I had not thought about installation on platforms other than plain debian linux, which I use daily. I think the challenges of getting RMG to work on its own on different systems might be another compelling reason to separate these installs - at least that way, users would only need to do one slightly tedious installation and an additional slightly tedious installation if needed, rather than a mandatory very difficult one.

@jonwzheng thank you for the input also! You have some great points here.

• for the possible CI issue, I will mention that I have previously dealt with packages that offer multiple installation options, and the testing process to avoid incompatible code changes is as simple as what is shown here: https://github.com/JacksonBurns/astartes/blob/09aa54bbba7063ff8a94f1dfa8f71adc7c07688f/.github/workflows/ci.yml#L63-L75
• as far as other core RMG functionality to move over to Julia/RMS, @mjohnson541 mentioned in this comment a few years ago that there were plans to port all of RMG's reactors to Julia: to_rms issue #2214 (comment) though I'm not sure where that stands today.
• I agree that the given installation process is awkward. I agree providing a better one-liner would be a great path forward, too.

@sevyharris and thank you for your input as well! Towards your last point, I really like the "batteries included"-ness we have right now as well.

RMG has a python liquid reactor: https://github.com/ReactionMechanismGenerator/RMG-Py/blob/36bceb35bc6b0d58beb2f2fa53faa12cf35c7034/rmgpy/solver/liquid.pyx
and a Julia liquid reactor:

RMG-Py/rmgpy/rmg/reactors.py

Line 515 in 36bceb3

class ConstantTVLiquidReactor(Reactor):

and users can access both in their input file according to these names:

RMG-Py/rmgpy/rmg/input.py

Lines 1536 to 1537 in 36bceb3

	'constantTVLiquidReactor': constant_T_V_liquid_reactor,
	'liquidReactor': liquid_reactor,

[ChrisBNEU]

I don't think I have much more to add to the discussion, but I am wondering if there is an option where we keep the RMS attached to rmg, we use something like PackageCompiler to turn rms into a c library. Most of the issues I have had with RMS have come from the Julia to Python tie in, and not the package itself. Just a thought. I didn't have time to go back and read all the related issues so it's possible this has already been discussed.

Overall, I am in favor of not having to use both python and julia in the same environment for my own overall quality of life/sanity. For the reasons specified above, it sounds like that involves decoupling rmg and rms. but if there is an alternative to that I am all ears.

[JacksonBurns]

Thanks for that suggestion! I imagine that using that repackager will be easier if we first separate RMG and RMS, too.

[mjohnson541]

How we can eat our pie and have two pies leftover

I feel inclined to note that there are number of things in the big initial description that are at least misrepresented, but I think going over them individually would be counterproductive.

I think a lot of this really comes down to some understandable misconceptions about: what the point of having the RMG-RMS integration is, why we’ve been stuck in what I consider to be a worst of both worlds state and due to a new generation of tools how close we are to resolving these issues.

I will articulate:

1. How we got here
2. What the dream was
3. Why we couldn’t achieve the dream at the time and why some things have gotten worse since then
4. How to achieve the dream now and make our Julia problems vanish
5. Logistics of how we get there

Why did we make RMS?

We present a number of good reasons to have RMS in the manuscript, but most of them were not why we built it in the first place. The first reason that’s less relevant here was to be able to use more modern numerical methods for sensitivity analysis such as adjoint and threaded sensitivities. The other reason that’s much more relevant here was to have a general simulation software that was speed competitive with existing software, but easy enough for students to add features relevant to their project without adding the feature needing to be a whole project in itself.

What was the original dream for the RMG-RMS interface?

What makes RMG unique from most other automatic mechanism generation algorithms is the use of species selection techniques, most notably the flux algorithm where we simulate the system and calculate fluxes to the edge allowing us to follow the flux to trace out a mechanism that accurately represents the chemistry in that system.

However, this has an inherent assumption that the system we’re running RMG on is the same system we’re ultimately simulating to get our results. …for taste in https://doi.org/10.1002/kin.21489 I simulated four different pyrolysis/combustion experiments each with distinct reaction systems only one of those four was reasonably represented by RMG’s simpleReactor. Even in catalysis where constant T is a bit more common ideally we would be simulating multiple facets instead of just one.

Enabling users to use reactors that better match their experiments of interest should significantly improve species selection, reduce model truncation errors and make writing RMG inputs files less art and more science. Which overall, should be a very significant improvement for RMG.

Achieving this and getting users on board isn’t reasonably feasible without the RMS reactors eventually being the default reactors.

Having RMS be RMG’s primary simulator also comes with additional benefits:
-Faster simulations/species selection => faster RMG can generate bigger models
-Can get rid of old reactors and dependencies we maintain
-Unification of the reactor interface, all options are available for all reactors

How’d we end up in the state we are now?

1. The Julia tools at the time were not up to the task

• pyjulia isn’t very compatible with conda and the python-jl hack I used that gets around this was never intended to be a long-term fix…
• Precompile time was still a major issue in the Julia community that didn’t have a great solution yet, but it was big enough that they were working on solving it
• Reliable automatic installation of Julia was bad enough I was building my own Julia conda binaries

2. I was graduating:
   To build this dream a number of things needed done:
   (1) Implement the core-edge algorithms in RMS
   (2) Manage install and calling RMS from RMG
   (3) Enable efficient launching of RMS core-edge from RMG
   (4) Implement set of new reactors
   (5) Teach people to use the new reactors and how customize them

However, it turns out (1) and (3) were a whole lot harder than I expected and I was graduating and leaving for my postdoc. I realized this was going to be my pdep/uncertainties module that was going to get passed down.

So a couple tricky decisions got made:

The dev team decided to merge the system for a few reasons:

1. It was functional
2. It was needed for future work in a couple different projects
3. We had some good reasons to optimistic about the near future of pyjulia even if that optimism was ultimately misplaced
4. Letting the feature die on the vine seemed like a huge waste

I decided to focus what time I had on cleaning up the aspects that I thought would be most challenging for anyone after me to deal with, which was (1) and (3). So (4) and (5) mostly didn’t happen and (2) that you guys are most familiar with was not great.

Since then I think the only other relevant change to note is the change from the Julia binaries I compiled myself to conda-forge Julia. This sounded like a good idea at the time, but this created two new problems: 1) conda-forge Julia is a large constraining dependency in environment/binary construction 2) binaries from places like conda-forge have become notorious enough that Julia actively warns against using them on its installation website. I strongly suspect the Zlib issues are associated with use of conda-forge Julia.

How do we achieve the dream now?

While at the original time I was one of the only people trying to interface Julia and python, it’s now quite popular and there are now a number of tools that weren’t available when I did this originally:

• Juliaup: Julia official version manager/installer…this allows us to install juliaup (a minimal dependency) instead of conda-forge Julia (a complex dependency that is known to cause problems)

• PackageCompiler.jl: Enables us to compile RMS into the julia system image which eliminates the RMS precompile time. Note that this system-image should be particular only to Julia version (not Julia install) and architecture so recompiling should be rare. Theoretically, we can distribute the system-image to particular architectures (removing nearly all of the installation procedure) as long as we are careful to keep our julia source and version constant.

• juliacall/PythonCall.jl: python-Julia interface designed to work with a conda environment (unlike pyjulia)…removes the need to install diffeqpy and DifferentialEquations.jl…removes need for python-jl…actively developed…easier to use and better than pyjulia in pretty much every aspect except some syntax stuff…

Integration of RMG with Juliaup and juliacall/PythonCall.jl is already available in #2640

I have successfully used PackageCompiler.jl to eliminate precompile time, although I haven’t tested system image transferability yet.

I compiled a list of Julia issues from most of you that I think can be summarized as:
• Julia is a heavy dependency in the conda environment
• Some issues such as missing libraries that seem to be associated with use of conda-forge julia
• General concerns about RMS installation maintainability
• Due to modifications of dependencies like in conda environments and occasionally the RMS environment won’t build (it’s a similar failure time window to conda environments, but it is an additional point of failure)
• When RMS is loaded it precompiles taking a very appreciable amount of time that makes debugging much slower and is frustrating when running very short RMG runs

All of these can be resolved or at least mostly mitigated relatively using the tools listed above:
• Julia heaviness in conda env => install Juliaup instead and installing Julia through it virtually eliminates this weight
• Conda-forge Julia issues => Switch to Juliaup
• General install maintainability concerns => switch to juliacall that is designed for our configuration unlike pyjulia
• RMS environment challenges
- => have users load a particular Julia version, Julia environment manifest and RMS version that we know works…these are easy to update by RMS developers if we need newer features
- => Create and distribute system images with RMS compiled on them for each CPU target, this should entirely eliminate any Julia env construction, precompilation etc and make Julia related installation time virtually instant…
• RMS precompile time: Use PackageCompiler.jl to compile RMS system image during install eliminating this time

Solutions are great, but how do we get there?

• We merge Jackson’s Upgrading to Python 3.11 #2635 once ready which is already in the works to enable python >3.7
• We merge Hao-Wei’s PR that switches us to juliaup and juliacall: Overhaul RMG-RMS Python-Julia dependencies & Add tests for many testless Arkane network algorithms #2640
• I’ll create PRs for:
  • Enable easy system image compilation “make rms” type command
  • Freeze us at a particular julia version, manifest and RMS version
• Decide how we want to handle transport/distribution of system-images
• Hackathon on creating new RMS reactors in RMG
• Eventually deprecate the old RMG reactors and remove the associated dependencies

None of this precludes us from merging the part of #2631 that avoids importing pyjulia when not necessary in the meantime.

[JacksonBurns]

An incomplete follow up on some offline discussion at the MIT RMG subgroup meeting - we will push on the Python upgrade and Julia dependency overhaul, which all parties seem interested in at some stage of development.