WIP: updated installing.rst and various files it points to.

[rjleveque]

Tried to simplify instructions for Fortran users.

This draft version now appears at
http://www.clawpack.org/installing.html

For comparison, the old is at
http://www.clawpack.org/installing_old.html

Suggestions welcome.

Are the PyClaw instructions still ok?

[mandli]

I read through and have some thoughts. I think the only one that I am particularly concerned about is instaling using the PYTHONPATH variable.

1. Move the prerequisites to the top
2. I highly suggest that we do not suggest using the PYTHONPATH variable as a means of installation for a couple of reasons, this is a fragile way of providing a path to the symlinks and therefore modules, and this provides a different functionality than running pip install clawpack. Since we are having to run a python setup.py command anyway why don't we tailor one that would do what pip install -e . does or alternatively pip install . depending on our intentions for the Fortran only code.
3. I think it would be good to present the "which clawpack" actually first or after the prerequisites.
4. Some of the instructions I think we do not have to break apart so much perhaps, maybe just provide a line separated by ;. We may even want to do something on the front page that would do the "install all clawpack packages".
5. This is a bit beyond what this page is trying to do but I was thinking we might add some best practices for each platform regarding how to install the prerequisites.
6. I was also thinking it might be nice to point to the SWC tutorials that might be useful JIC.

[rjleveque]

Some responses to @mandli's list:

1. I'm ok with that, but maybe just list prereq's with a pointer to another page for more details and hints so it doesn't fill up too much of the page before getting to Clawpack installation.
2. Doesn't "pip install" try to compile all the f2py stuff, which sometimes doesn't work and at any rate isn't needed by those using only the Fortran versions. If nothing else, it makes the installation go much more slowly. I also find use of PYTHONPATH very handy as a developer since I sometimes switch between different versions, e.g. I have an installation of v5.3.1 that's much easier to switch to by resetting CLAW and PYTHONPATH than by checking out the right version of every repo, particularly if I'm in the middle of some development work on a branch in the repos while I'm also trying to do some production runs in another window with v5.3.1. And users might have more than one version installed if they have some legacy code they are working with while also upgrading. But maybe there's a better way to do this?
3. Again I don't want to fill up too much of the page before getting to the installation instructions. Maybe have a section "First read:" with pointers to prerequisites and "Which Clawpack?" and separate this from the "See also" list?
4. I kept the section headings for things like "set environment variables" because it's nice to be able to refer to this section elsewhere in Sphinx or send someone a link to it, e.g.
   http://www.clawpack.org/installing.html#set-environment-variables
5. Fine, on a separate page.
6. Also good idea.

[ketch]

Doesn't "pip install" try to compile all the f2py stuff, which sometimes doesn't work and at any rate isn't needed by those using only the Fortran versions. If nothing else, it makes the installation go much more slowly.

If our attitude is "pip install is broken", then the package shouldn't be on pypi. If the default install method "sometimes doesn't work", we should fix it. If the f2py compilation is taking more than a minute or two, then something is wrong.

What I have seen is people running into trouble because the compiled code is in their site-packages and then they set PYTHONPATH to a directory where they cloned things (but didn't compile). In fact, it seems to me that the new instructions will actually leave users in this (BROKEN) state! Has anyone checked?

I find the phrase "PyClaw Python interface" confusing; let's just say "PyClaw solvers". It should also be pointed out that the PyClaw-only installs does install visclaw; can we change the title to "Install PyClaw and VisClaw"? Also, it does let you run the classic algorithms, whereas the new wording seems to imply that it does not.

Since PyClaw also consists of Fortran and Python, it may be clearer to change

"Install Fortran packages and Python tools"

to

"Install Classic, AMRClaw, GeoClaw, and VisClaw"

[rjleveque]

We really need to sort out the PYTHONPATH / pip issues. It's clear PyClaw users and GeoClaw users run into somewhat orthogonal sets of problems. I've been working with several people who need to (a) easily download and install GeoClaw without ever planning to use PyClaw and (b) update to a more recent version while keeping the old one(s) around for validation comparisons, with the ability to run the code (and use Python tools) from either one.

What's the recommended way to do this if you've done a pip install? I don't think pip install -e . solves this problem if you want to be using different versions from different terminals simultaneously.

I'd be very happy to get rid of setting PYTHONPATH if there's a better mechanism for switching environments. Should we suggest that everyone use conda environments, for example? (And use conda install instead of pip install?)

[ketch]

You can pip install any version you wish; you can even pip install a specific commit from Github. Does that solve the problem? If not, can you explain why not?

Ah, I forgot that you want to run different versions simultaneously in different terminals. Is that really a good idea, from the point of view of provenance and reproducibility? And do we want the default install instructions to be altered to accommodate that? It seems like something we might want to discourage except in the case of expert users, since it could lead to a lot of difficulty in troubleshooting problems.

I think virtualenv is considered the proper way to accomplish what you're looking for, but I've found virtualenv to be hard to work with in the past. Another possibility is the use of conda environments, but I haven't tried that.

[rjleveque]

@ketch: That's great, wish I'd known about it years ago! Or more likely someone told me this and it went in one ear and out the other. At any rate, it's not documented at all in our installation instructions, but we can rectify that.

Here's what I've figured out playing around with it a bit. Does this sound right?

pip install -e git+git@github.com:clawpack/clawpack.git@v5.3.1#egg=clawpack

will create a directory $HOME/src/clawpack (cloned from GitHub and checked out to the v5.3.1 tagged commit), and runs setup.py develop. It will also set $HOME/anaconda/lib/python2.7/site-packages/easy-install.pth to point to this version.

Fortran users should set CLAW to point to $HOME/src/clawpack to use this version.

Installing a different version, e.g. via:

pip install -e git+git@github.com:clawpack/clawpack.git@v5.3.0#egg=clawpack 

will simply check out a different commit in that same directory and run setup.py develop again. Or you can replace v5.3.0 by a specific commit of the top level clawpack repo. No need to change where $CLAW points when switching this way, and the Fortran repositories also get checked out to the appropriate versions too.

To switch to my own clone where I do development work, I can do, e.g:

export CLAW=$HOME/git/clawpack
cd $CLAW
pip install -e .

which resets $HOME/anaconda/lib/python2.7/site-packages/easy-install.pth to point to this directory.

This seems pretty clean and avoids the need for setting PYTHONPATH.

What this doesn't allow, as @ketch points out, is running different versions in different shells simultaneously for dual-debugging. But that's perhaps a special case we should deal with in other ways, e.g. by using conda environments.

Here's a little bash function that may be handy to figure out where things point:

function whichclaw () {
  echo "CLAW is set to: "  $CLAW ;
  echo "Python will use:" `python -c "import sys; print [p for p in sys.path if 'clawpack' in p][0]"`;
}

Of course it's easy to write functions to simplify switching versions too.

[mjberger]

I don’t understand much of this, but it’s not just for dual debugging that 2 versions are needed. I do production runs using one version of clawpack, develop in another, at the same time. Sometimes two different version are running and I”m testing a third. Will this permit that?

— Marsha

On Jul 5, 2016, at 4:49 PM, Randall J. LeVeque notifications@github.com wrote:

@ketch: That's great, wish I'd known about it years ago! Or more likely someone told me this and it went in one ear and out the other. At any rate, it's not documented at all in our installation instructions, but we can rectify that.

Here's what I've figured out playing around with it a bit. Does this sound right?

pip install -e git+git@github.com:clawpack/clawpack.git@v5.3.1#egg=clawpack
will create a directory $HOME/src/clawpack (cloned from GitHub and checked out to the v5.3.1 tagged commit), and runs setup.py develop. It will also set $HOME/anaconda/lib/python2.7/site-packages/easy-install.pth to point to this version.

Fortran users should set CLAW to point to $HOME/src/clawpack to use this version.

Installing a different version, e.g. via:

pip install -e git+git@github.com:clawpack/clawpack.git@v5.3.0#egg=clawpack
will simply check out a different commit in that same directory and run setup.py develop again. Or you can replace v5.3.0 by a specific commit of the top level clawpack repo. No need to change where $CLAW points when switching this way, and the Fortran repositories also get checked out to the appropriate versions too.

To switch to my own clone where I do development work, I can do, e.g:

export CLAW=$HOME/git/clawpack
cd $CLAW
pip install -e .
which resets $HOME/anaconda/lib/python2.7/site-packages/easy-install.pth to point to this directory.

This seems pretty clean and avoids the need for setting PYTHONPATH.

What this doesn't allow, as @ketch points out, is running different versions in different shells simultaneously for dual-debugging. But that's perhaps a special case we should deal with in other ways, e.g. by using conda environments.

Here's a little bash function that may be handy to figure out where things point:

function whichclaw () {
echo "CLAW is set to: " $CLAW ;
echo "Python will use:" python -c "import sys; print [p for p in sys.path if 'clawpack' in p][0]";
}
Of course it's easy to write functions to simplify switching versions too.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[rjleveque]

@berger: For such things it might be best to use conda environments or virtualenv, or just stick with setting CLAW and PYTHONPATH as we do now. Changing what we recommend to users wouldn't preclude doing this.

[mandli]

@mjberger FWIW I have for awhile used virtualenv to do testing with different environments at the same time and it should be straightforward to do what you need. In the background virtualenv sets a bunch of paths beyond PYTHONPATH so that you are ensured of having a protected environment. I think the hitchhiker's guide provides a good introduction along with some very handy additions, most notably the virtualenvwrapper. I would much prefer we go this route perhaps mentioning this route in the developers' section.

[rjleveque]

Once I tried this on another computer I noticed a mistake in my notes above -- the git@github... form requires ssh authentication. It would be more general to suggest

pip install -e git+https://github.com/clawpack/clawpack.git@v5.3.1#egg=clawpack

I've also been experimenting with this in notebooks served via binder. It seems to work as a way to run the Fortran code on a cloud server, since installing with -e causes the source code files to be preserved rather than deleted. Some initial experiments are in the repository https://github.com/rjleveque/binder_experiments. Click on the "lauch binder" icon on that page, or clicking this link will also launch it.

Binder is pretty cool in general. I've also used it successfully with some other non-clawpack repositories, e.g.

• https://github.com/rjleveque/ptha_tutorial
• https://github.com/rjleveque/amath586s2016

and it seems like a good way to distribute experiments people can run without having to install anything.

Note: The binder_experiments repository has a file requirements.txt where the packages to pip install are listed. There is also a notebook from the PyClaw tests in that binder directory that does not work with this installation of Clawpack from the GitHub v5.3.1 tag. But it did work when I just listed clawpack alone in requirements.txt. Something to track down --- it throws the following error in the line status = claw_run():

ImportError: /home/main/anaconda2/bin/../lib/libgfortran.so.3: version `GFORTRAN_1.4' not found 
 (required by /home/main/src/clawpack/clawpack/pyclaw/classic/classic1.so)

[rjleveque]

The gfortran problem on binder is due to an incompatibility in numpy, and the notebooks work fine if you install an earlier version via the following command in a notebook cell as a temporary hack:

! conda install -f -y libgfortran==1

[mandli]

Which version of gfortran is causing the problem and which does this switch it to?

[rjleveque]

It installs version 1.x in place of 3.x. See
ContinuumIO/anaconda-issues#686
@ketch and I were working on this together and he added a query to that issue about the status of a fix.

[mandli]

Wow, this is a much more complex issue than I was imagining! Hopefully something concrete will happen.

[ketch]

On Tuesday, @rjleveque and I went through some installation tests and found a procedure that seemed to work well (both the Fortran codes and PyClaw work fine) with no need to set PYTHONPATH. We both felt that it would make sense to give that as the default install instructions for everyone and avoid having different instructions for pyclaw versus fortran users. If I remember correctly, the install command is

pip install -e git+https://github.com/clawpack/clawpack.git@v5.3.1#egg=clawpack

(the version tag will have to be updated whenever there is a release). Then PyClaw users are done and Fortran code users simply need to set their CLAW variable, unless I'm forgetting something.

[rjleveque]

Yes, I think that's correct. The -e flag (editable) causes the source code to be left in place rather than deleted after building PyClaw, so in particular the Fortran codes and examples are still there.

The one thing we didn't figure out is how to specify where this source code should be left. In an earlier comment I said it ends up in $HOME/src/clawpack, but I had done the pip install from my home directory. If you do it from a different directory then it seems to create a new src directory under that. We tried using the --target flag in pip but didn't get it to work as desired, and also it sometimes seems to also build a new numpy and matplotlib as dependencies in this directory. We haven't had time to sort this part out yet, but maybe someone else has experience with this?

[mandli]

Did you try the --src flag?

[rjleveque]

Great, that seems to work!

pip install --src=$HOME/src -e git+git@github.com:clawpack/clawpack.git@v5.3.1#egg=clawpack

It looks like you can also add the flag --no-deps to avoid possibly re-installing dependencies.

I found the documentation at https://pip.pypa.io/en/stable/reference/pip_install/#description

[mandli]

On my brand new machine running 10.11.6 the following steps produced a successful Clawpack installation with all nose tests passing all packages (checking out master as is today). This would be for a developer's build.

1. Install XCode and command line tools
2. Install homebrew
3. Install gfortran via brew install gcc --without-multilib
   • Note that OpenMP may not work if multilib support is kept (the default)
4. Install python via brew, enable homebrew’s own tcl-tk
5. Upgrade pip
6. Install numpy, scipy, matplotlib, ipython, jupyter, nosetests via pip
7. Install petsc via brew install petsc
   • Remember to set the appropriate PETSC_DIR path (brew will tell you what to set it to)
8. Install petsc4py via pip install petsc4py
9. Clone the clawpack repository git clone git://github.com/clawpack/clawpack.git
10. Set the CLAW environment variable to point to the top-level directory you just cloned to.
11. Install clawpack with pip install -e .

[ketch]

@mandli Thanks; but I think the step where you clone clawpack is missing?

[mandli]

@ketch No wonder the tests all worked before :) Fixed it.

[ketch]

Probably a stupid question, but here goes: for the Fortran package tests to pass, don't you also need to set CLAW and compile other things?

In any case, for the generic install instructions we probably want to suggest what you worked out earlier:

pip install --src=$HOME/src -e git+git@github.com:clawpack/clawpack.git@v5.3.1#egg=clawpack

as the clone+install command.

[mandli]

Thanks, missed that one too.

[rjleveque]

Note that my latest attempt at instructions can be viewed at
http://www.clawpack.org/installing_pip.html
Suggestions welcome.

[mandli]

I think it looks pretty good although I have some comments:

• The XCode command line tools make gcc compatible component available including Make but not exclusive to it.
• I would not recommend using the hpc.sourceforce.net versions of gfortran as they tend to not play well with other system components. This is often problematic when building the Python Fortran components.
• It actually occurred to me that most of the steps could be scripted for a particular platform. This would be easiest on Mac OS X and more complex depending on the Linux distro but I think we could do it.

[rjleveque]

@mandli: what do you recommend for gfortran on the Mac? Is just pointing to https://gcc.gnu.org/wiki/GFortran sufficient for these instructions, without recommending anything different for Mac?

[mandli]

The largest drawback to installing this way is that it is very difficult to upgrade if you would ever like to. I would say we suggest this over the old HPC page though.

[rjleveque]

For now I'm going to merge this in since it's already showing up live in our webpages and I want to merge in some other things. We can tweak further on a new PR if needed.