`CI.yml` and `environment.yml` updates to run CI on MacOS and Ubuntu #2415

JacksonBurns · 2023-04-16T22:59:22Z

This PR runs the Continuous Integration on a mac runner in addition to the Ubuntu runner, the idea being that we can catch issues that might crop up on the mac platform independently of debian linux. The results from the ubuntu runner are still considered to be the "baseline" for regression testing.

See #2355 and #2392 for issues related to running on Macs

JacksonBurns · 2023-04-17T00:26:18Z

Previous CI failed due to a limitation of macos runners, see byrnedo/docker-reg-tool#9. Updating now

JacksonBurns · 2023-05-08T01:55:19Z

Switching to official Julia binaries, which may resolve the code signing problem, is blocked by the same libgfortran update that is impacting the upgrade to OpenMOPAC (i.e. potential fix is blocked by #2417). See failed CI run when attempting to use them.

JacksonBurns · 2023-05-08T01:58:43Z

Potential bug in PATH setup for setup-julia action: julia-actions/setup-julia#89

codecov · 2023-05-11T18:22:27Z

Codecov Report

Merging #2415 (25cbbc7) into main (2978f1a) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2415      +/-   ##
==========================================
+ Coverage   48.11%   48.13%   +0.02%     
==========================================
  Files         110      110              
  Lines       30626    30626              
  Branches     7988     7988              
==========================================
+ Hits        14735    14743       +8     
+ Misses      14362    14353       -9     
- Partials     1529     1530       +1

see 2 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

.github/workflows/CI.yml

JacksonBurns · 2023-05-11T23:00:14Z

@calvinp0 could you review my updated installation instructions, which include warnings for apple silicon users?

JacksonBurns · 2023-05-12T00:13:56Z

Merging this PR also:

resolves Nosetests on Mac fail #2392
resolves RMG test run error on MacOS #2355
resolves RMS error after commit 437739 #2290

calvinp0 · 2023-05-12T13:41:25Z

environment.yml

  - rmg::pydqed >=1.0.3
-  - rmg::pyjulia


Is pyjulia installed through conda-forge::julia=1.8.5?

It's not, and the rmg:pyjulia does not either. The conda-recipes repo has the build script and you can see that it only provides Julia (as far as I can tell, I'm new to building conda packages). I think pyjulia is an implicit dependency of something else in the environments file? So it is added when the environment is solved?

So, correct if I am misunderstanding what you are saying, but we will need to keep pyjulia as it is the connection between Julia and Python. So I guess we can either keep getting from the rmg channel or from conda-forge channel.

However, the CI appears to be passing? I am a bit confused though how it is doing that when pyjulia isn't installed in the rmg_env (I am assuming here or really misunderstanding something)

So, correct if I am misunderstanding what you are saying, but we will need to keep pyjulia as it is the connection between Julia and Python. So I guess we can either keep getting from the rmg channel or from conda-forge channel.

However, the CI appears to be passing? I am a bit confused though how it is doing that when pyjulia isn't installed in the rmg_env (I am assuming here or really misunderstanding something)

My mistake - conda still ends up retrieving pyjulia from the rmg channel. See this line in the latest CI run.

CC'ing @mjohnson541 - pyjulia, how does it work? The conda-recipe doesn't make sense to us.

So my understanding of PyJulia is that, assuming you have Julia installed and set in the PATH (or set in our conda env path for our case when rmg_env is active), it is imported in python like so:

import julia

Then we use PyJulia for the julia.install() which does the whole PyCall adding (if not present) and building (unless it is M1, then it appears to have an issue with that

We can also use PyJulia to create our system images for precompiling

Apologies if that was already understood

It is a dependency of pyrms and diffeqpy which causes it to be installed as well.

calvinp0 · 2023-05-12T13:42:56Z

documentation/source/users/rmg/installation/anacondaDeveloper.rst

+    conda create -n rmg_env
+    conda activate rmg_env
+    conda config --env --set subdir osx-64
+    conda install python=3.7


A bit pedantic and not really a concern or issue, but I figure we could remove the install of python=3.7 here as it will be install in the conda env update part of the script

Yes that seems like a wasted instruction. We should try the install from scratch on a mac without it just to be sure, though.

I did it last night on my Mac, so it installed fine without that command

JacksonBurns · 2023-05-16T03:28:56Z

I have updated environment.yml to make it explicit which packages which are resolving and still depend on, and I have also added extensive inline documentation. I have also removed the redundant line from the developer installation instructions. The CI should pass overnight and then I will request new reviews.

JacksonBurns · 2023-05-16T13:16:04Z

Tagging a lot of people for review since this PR now contains (1) a different "required" check, which requires admin access and (2) changes to the environment that will change how RMG is run on MacOS.

@hwpang and @rwest please provide general approval for this updated CI approach

@calvinp0 please check the documentation changes and ensure they match the comments from the review

@mjohnson541 please review to make sure that I did not describe any dependencies in the documented environment.yml

.github/workflows/CI.yml

hwpang · 2023-05-16T13:18:13Z

.github/workflows/CI.yml

+  # 
+  # taken from https://github.com/orgs/community/discussions/4324#discussioncomment-3477871
+  ci-report-status:
+    runs-on: ubuntu-20.04


Why do we specify the version ubuntu-20.04 here but ubuntu-latest on line 257?

Because the place I copy-pasted from did it that way - should be latest instead so we don't have to update it as GitHub releases new and deprecates old runners. Thanks for catching this!

I will update now.

I will also add a clarification for something that confused myself, this branch is called add-macos11-ci but it actually runs on macos-latest which as of writing (May 16, 2023) is MacOS 12 Monterey

hwpang · 2023-05-16T13:27:02Z

documentation/source/users/rmg/installation/anacondaDeveloper.rst

@@ -84,6 +73,15 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux
    cd RMG-Py
    conda env create -f environment.yml

+.. warning:: Apple Silicon (M1+) Users


Is this the same suggestion for Intel Mac?

No, it only applies to M1 users (cc @calvinp0 who figured this out, thanks!)

rwest

Some suggestions that could be separate pull requests (eg. trying to change channel for pyjulia) but some fixes to the user instructions that should probably be addressed before merging.
And a question about the benefit of adding a changelog to the environment file (which is already in git with - I hope - helpful commit messages).

rwest · 2023-05-16T13:59:05Z

documentation/source/users/rmg/installation/anacondaDeveloper.rst

@@ -128,6 +126,14 @@ Installation by Source Using Anaconda Environment for Unix-based Systems: Linux

     julia -e 'using Pkg; Pkg.add(PackageSpec(name="ReactionMechanismSimulator",rev="main")); using ReactionMechanismSimulator;'

+.. warning:: Apple Silicon (M1+) Users
+   Execute the following commands instead of ``conda env create -f environment.yml``: ::


I don't think you mean this "instead of...".

one difference I notice with the M1 instructions is the "python import julia" comes before the "julia using Pkg" lines instead of after. Is this intentional and important? (if not maybe we can keep consistent)

if the only difference is the M1 needs julia -e 'using Pkg; Pk.add("PyCall");Pkg.build("PyCall") maybe we could reduce the instructions to just say run this. (And is it really needed? and is there harm in everyone running it?)

if the instruction is to do something special before the default instructions, it would be good for the special instructions to come first. (so people don't do the default and then read "but first you should have..."). Like writing a cooking recipe.

you are right, was copied from above

this is intentional, see @calvinp0's comment here

but this is a good point, we can have everyone manually install it. will edit the instructions appropraitely

this admonition was removed, but I will locate the first one to follow this idea

rwest · 2023-05-16T14:00:55Z

environment.yml

+# + some other categories (see below)
+#
+# Changelog:
+# - May 15, 2023 Added this changelog, added inline documentation,


I'm torn about introducing a changelog, since one could always do git log -- environment.yml. I could open to being persuaded though; when is a changelog in a file helpful?

The way I see changelogs is that they provide a "high-level" perspective on how the file has changed over time. The git log is useful for fixing technical issues and precise problems with specific lines, but I think the changelog is better for answering questions like "why do we have some packages from this channel but other from this one" or the like.

environment.yml in particular has been going through substantial changes lately, and as we (hopefully) move toward using fewer and fewer in-house packages I think it will be useful to have this high level history of that activity. This is also a file that community developers will interact with a lot - such info would be particularly useful for them.

rwest · 2023-05-16T14:03:38Z

environment.yml

+  - conda-forge::openbabel >= 3
+
+# general-purpose external software tools
+  - conda-forge::julia=1.8.5


Dare we try >= and let the CI alert us if it breaks?

Can we put - conda-forge::pyjulia here instead of the - rmg::pyjulia down below? My impression from #2381 was that the answer is "yes" but maybe we can test it here?

on second thoughts, that could be a separate pull request if you prefer (and don't mind the churn)

Dare we try >= and let the CI alert us if it breaks?

Move fast and break things, I say. Will try now.

Can we put - conda-forge::pyjulia here instead of the - rmg::pyjulia down below? My impression from #2381 was that the answer is "yes" but maybe we can test it here?
on second thoughts, that could be a separate pull request if you prefer (and don't mind the churn)

Every package we don't have to maintain is welcome - will try this as well.

rwest · 2023-05-16T14:41:26Z

if you're about to force push for new tests, maybe rebase onto main? (and squash some of the commits that undo each other?)

JacksonBurns · 2023-05-16T15:15:36Z

add-macos11-ci is already up to date, so I can't rebase.
If it's all the same to you, I would like to leave the complete git history. By leaving the thing I tried with xcode (df8bc71), for example, I can go back in time and see the results without and avoid redoing work.

IIRC this is done somewhere in the rmg channel Julia custom build

our build of Julia was setting these two environment variables automatically by putting them into the conda activate environment script. we should try to replicate that here, but for now I am hardcoding the result for the runner to see if it will fix the precompilation issues from the previous actions run

we cannot use branch protection rules on jobs that have matrix in them, so we add a summary step that will wait for both the Mac and Linux builds+tests to execute and then return pass or fail. we will need to update the branch protection rule for main to look for ci-report-status instead of -build-and-test-linux

rwest · 2023-05-18T02:02:38Z

Sorry. The "Switch to conda-forge::pyjulia instead of the rmg channel" commit was meant to be on another branch (that's not working yet). Will drop it on rebasing when I get a chance.

- break up the environment creation step s.t. the admonition for Mac M1 users are more clear - change the Julia and Python installation instructions so that they work for all platforms in the same order

This explanation was added in commit 5d20826 but I think it refers to the line ln -sfn $(which python-jl) $(which python) that was added in b6910a7 (2 minutes later) to the continuous integrations workflow, but not actually added to the user instructions. i.e. the linking step (that this explanation refers to) is not in the instructions. So I suggest we remove it.

Mostly whitespace and linebreaks. Added a new step number.

This changed many years ago now. Although some clusters (eg. at Northeastern) still have an old old anaconda, I no longer feel it's our job to explain how to use it, and just makes the instructions longer than needed.

i have added comments to the environment file, the issue is that our build of pyjulia does some `sed` magic that we do not replicate otherwise. would be good to get rid of this in the future, but this PR is already getting out of hand

- who needs pyjulia - why use conda-forge ncurses instead of default - gfortran is not listed in the env file, explain why

Otherwise the PyCall step installs it via conda! It looks like this: 279 dependencies successfully precompiled in 550 seconds. 14 already precompiled. [ Info: Installing rmgpy.molecule via the Conda rmg package... [ Info: Running `conda config --add channels rmg --file /opt/miniconda3/envs/rmg_env3/condarc-julia.yml --force` in root environment [ Info: Running `conda install -y rmg` in root environment (then you get the old binary from the rmg conda channel instead of your developer version) Or at least I *think* this is why that happened. When I did the whole thing again with this step included I didn't have the same problem.

The solver setting is environment-specific. If you set it before you make the rmg_env then it is not set IN the rmg_env and when you later do stuff to that environment, it's super slow.

rwest

Looking good to me. Additional improvements can come in subsequent PRs.

JacksonBurns · 2023-05-18T13:19:05Z

Looking good to me. Additional improvements can come in subsequent PRs.

👍 @rwest could you update the branch protection rules to look for ci-report-status instead of build-and-test-linux, then merge?

rwest · 2023-05-18T13:35:11Z

I set it to this instead:

which if it does what I expect, will mean we can remove the "ci-report-status" step.
Do you think that'll work, or have I misunderstood something?

JacksonBurns · 2023-05-18T13:37:24Z

@rwest that works as well - the ci-report-status is not needed since it is set that way, can be removed. I will roll that into a larger dead code removal PR.

hwpang reviewed May 11, 2023

View reviewed changes

.github/workflows/CI.yml Show resolved Hide resolved

JacksonBurns requested a review from calvinp0 May 11, 2023 22:59

This was linked to issues May 12, 2023

Nosetests on Mac fail #2392

Closed

RMG test run error on MacOS #2355

Closed

RMS error after commit 437739 #2290

Closed

calvinp0 reviewed May 12, 2023

View reviewed changes

JacksonBurns mentioned this pull request May 12, 2023

Nosetests on Mac fail #2392

Closed

JacksonBurns changed the title ~~CI: Run on MacOS in addition to Ubuntu~~ CI.yml and environment.yml updates to run CI on MacOS and Ubuntu May 16, 2023

JacksonBurns mentioned this pull request May 16, 2023

Added Julia into the conda rmg_env #2280

Closed

JacksonBurns requested review from rwest, mjohnson541, hwpang and calvinp0 May 16, 2023 13:11

JacksonBurns added the Status: Ready for Review PR is complete and ready to be reviewed label May 16, 2023

JacksonBurns self-assigned this May 16, 2023

hwpang reviewed May 16, 2023

View reviewed changes

.github/workflows/CI.yml Show resolved Hide resolved

hwpang reviewed May 16, 2023

View reviewed changes

rwest reviewed May 16, 2023

View reviewed changes

rwest mentioned this pull request May 16, 2023

Creating and utilising a custom system image of Julia with RMS #2381

Closed

JacksonBurns and others added 10 commits May 17, 2023 08:50

manually add PyCall to the MacOS global Julia install

5a56fd0

IIRC this is done somewhere in the rmg channel Julia custom build

can we use official Julia? is Killed 9 due to code signing?

39fea32

add missing sparse matrix libs needed by conda julia

27da1c4

update developer install docs for conda Julia and special M1 instruction

3ef7225

add EXTENSIVE inline docs, make the dependence on pyjulia explicit

adf003f

remove redundant line from developer instructions

814419c

changes result aggregation job to use ubuntu-latest

dc03b09

set julia to >=1.8.5 and use conda-forge distribution of pyjulia

a6f23cd

JacksonBurns force-pushed the add-macos11-ci branch from e166e55 to ff4b799 Compare May 17, 2023 13:00

JacksonBurns requested review from rwest and hwpang May 17, 2023 17:16

rwest force-pushed the add-macos11-ci branch from 73c18e4 to a047389 Compare May 17, 2023 22:15

JacksonBurns and others added 8 commits May 17, 2023 23:28

developer installation instructions update, see extended description

8749d17

- break up the environment creation step s.t. the admonition for Mac M1 users are more clear - change the Julia and Python installation instructions so that they work for all platforms in the same order

Rewording install instructions a bit.

e0e2601

Mostly whitespace and linebreaks. Added a new step number.

Remove the note about activating old anaconda environments.

f5e7971

This changed many years ago now. Although some clusters (eg. at Northeastern) still have an old old anaconda, I no longer feel it's our job to explain how to use it, and just makes the instructions longer than needed.

add further clarifying comments for pyjulia, ncurses, and gfortran

45a6de6

- who needs pyjulia - why use conda-forge ncurses instead of default - gfortran is not listed in the env file, explain why

Dev install instructions: Switch the rmg_env to libmamba solver.

25cbbc7

The solver setting is environment-specific. If you set it before you make the rmg_env then it is not set IN the rmg_env and when you later do stuff to that environment, it's super slow.

rwest force-pushed the add-macos11-ci branch from a047389 to 25cbbc7 Compare May 18, 2023 03:29

rwest approved these changes May 18, 2023

View reviewed changes

rwest merged commit 99201fe into main May 18, 2023

rwest deleted the add-macos11-ci branch May 18, 2023 13:31

CI.yml and environment.yml updates to run CI on MacOS and Ubuntu #2415

CI.yml and environment.yml updates to run CI on MacOS and Ubuntu #2415

Conversation

JacksonBurns commented Apr 16, 2023

JacksonBurns commented Apr 17, 2023

JacksonBurns commented May 8, 2023

JacksonBurns commented May 8, 2023

codecov bot commented May 11, 2023 • edited Loading

Codecov Report

JacksonBurns commented May 11, 2023

JacksonBurns commented May 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JacksonBurns May 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjohnson541 May 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

calvinp0 May 12, 2023 • edited Loading

Choose a reason for hiding this comment

JacksonBurns commented May 16, 2023

JacksonBurns commented May 16, 2023

hwpang May 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwest left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwest commented May 16, 2023

JacksonBurns commented May 16, 2023

rwest commented May 18, 2023

rwest left a comment

Choose a reason for hiding this comment

JacksonBurns commented May 18, 2023

rwest commented May 18, 2023

JacksonBurns commented May 18, 2023

`CI.yml` and `environment.yml` updates to run CI on MacOS and Ubuntu #2415

`CI.yml` and `environment.yml` updates to run CI on MacOS and Ubuntu #2415

codecov bot commented May 11, 2023 •

edited

Loading

JacksonBurns May 12, 2023 •

edited

Loading

mjohnson541 May 17, 2023 •

edited

Loading

calvinp0 May 12, 2023 •

edited

Loading

hwpang May 16, 2023 •

edited

Loading