PyCBC install instructions are confusing

[duncan-brown]

When running

[dbrown@seaview ~]$ pip install pycbc --user

I get the error:

Downloading/unpacking mpld3>=0.3git (from pycbc)
Could not find a version that satisfies the requirement mpld3>=0.3git (from pycbc) (from versions: 0.0.1, 0.1, 0.2)
Some insecure and unverifiable files were ignored (use --allow-unverified mpld3 to allow).

I looked at pypa/pip#1423 which discusses this issue, but I couldn't resolve it with any combination of --allow-unverified and --allow-all-external that I could figure out.

pip 1.5.6

[ahnitz]

Duncan,

Looking at the documentation again, can you try the following

--process-dependency-links

On Tue, Jun 30, 2015 at 3:07 PM, Duncan Brown notifications@github.com
wrote:

When running

[dbrown@seaview ~]$ pip install pycbc --user

I get the error:

Downloading/unpacking mpld3>=0.3git (from pycbc)
Could not find a version that satisfies the requirement mpld3>=0.3git
(from pycbc) (from versions: 0.0.1, 0.1, 0.2)
Some insecure and unverifiable files were ignored (use --allow-unverified
mpld3 to allow).

I looked at pypa/pip#1423 pypa/pip#1423 which
discusses this issue, but I couldn't resolve it with any combination of
--allow-unverified and --allow-all-external that I could figure out.

pip 1.5.6

—
Reply to this email directly or view it on GitHub
#124.

[ahnitz]

The pycbc install documentation page lists the following option.

pip install “numpy>=1.6.4” unittest2
pip install -e git+https://github.com/ligo-cbc/pycbc#egg=pycbc
--process-dependency-links

This installs from current master and creates a git checkout of pycbc.
To install from the current release (and upgrade if you have and old
pycbc install)

pip install “numpy>=1.6.4” unittest2
pip install pycbc --process-dependency-links --upgrade --user

On Tue, Jun 30, 2015 at 3:09 PM, Alex Nitz alex.nitz@ligo.org wrote:

Duncan,

Looking at the documentation again, can you try the following

--process-dependency-links

On Tue, Jun 30, 2015 at 3:07 PM, Duncan Brown notifications@github.com
wrote:

When running

[dbrown@seaview ~]$ pip install pycbc --user

I get the error:

Downloading/unpacking mpld3>=0.3git (from pycbc)
Could not find a version that satisfies the requirement mpld3>=0.3git
(from pycbc) (from versions: 0.0.1, 0.1, 0.2)
Some insecure and unverifiable files were ignored (use --allow-unverified
mpld3 to allow).

I looked at pypa/pip#1423 pypa/pip#1423
which discusses this issue, but I couldn't resolve it with any combination
of --allow-unverified and --allow-all-external that I could figure out.

pip 1.5.6

—
Reply to this email directly or view it on GitHub
#124.

[duncan-brown]

Thanks, I was trying to copy and paste from the email that you sent Sarah.

I'm leaving this ticket open, as the install instructions are somewhat confusing. It's not clear to the new user where to begin. Should I manually install all of the dependencies listed at the top of the page, skip to using virtualenv, or…?

I will take a look at this when I get back to Syracuse.

[scaudill]

Hi Duncan, Alex,

After installing lalsuite, I was able to install the released version of pycbc on CIT with

pip install unittest2 pycbc --upgrade --user --process-dependency-links

From the instructions page here: https://ldas-jobs.ligo.caltech.edu/~cbc/docs/pycbc/install.html, I agree that it was not clear what was the best option for installing pycbc for my uses (using some of the tools). I think the section on virtualenv was close to what I wanted but I didn't realize that. I think that the instructions for doing a pycbc 'pip install' should be moved a bit higher.

[josh-willis]

After spending much time trying to follow the instructions for installing from source, I can find nothing that works, except commenting out the use of setuptools in setup.py.

All of the comments above on this issue are for pip install but the current documentation claims that it should also be possible to install from source using the traditional python setup.py install --prefix==/path/to/pycbc. After manually installing all of the stated dependencies in the documentation (as well as some others that those required), I tried installing per the instructions. That failed because setuptools in its wisdom insists that the path I am installing to already be on PYTHONPATH. After fixing that, it processed a whole host of dependencies, before failing with Error in atexit._run_exitfuncs followed by a lengthy, cryptic traceback.

What methods are currently expected to work? I would argue that it is critical that installing from a local git checkout be supported, to a path of the user's choosing, and without virtualenv if desired.

[ahnitz]

Josh,

It would help if you could post the error you are getting and exact command
lines you are trying. Installing from source and simply doing,

python setup.py install --prefix=/path/to/install/location

Works for me,

and so does doing pip install pycbc --user or --target.

On Mon, Jul 13, 2015 at 6:18 AM, josh-willis notifications@github.com
wrote:

After spending much time trying to follow the instructions for installing
from source, I can find nothing that works, except commenting out the use
of setuptools in setup.py.

All of the comments above on this issue are for pip install but the
current documentation claims that it should also be possible to install
from source using the traditional python setup.py install
--prefix==/path/to/pycbc.

After manually installing all of the stated dependencies in the
documentation (as well as some others that those required), I tried
installing per the instructions.

Is there a reason you are manually installing the dependencies? If there
is a dependency missing that the package requires, then please add it i the
list within setup.py so that we can ensure it is installed.

That failed because setuptools in its wisdom insists that the path I am
installing to already be on PYTHONPATH.

It should warn you immediately that you are doing something invalid
(installing into a location you can't use!), and stop to give you an
opportunity to add it to your path. To be clear, you must have the
location in your pythonpath first as that is the only way to ensure that
dependencies that need to be resolved are installed into a location that
can be immediately used.

After fixing that, it processed a whole host of dependencies, before
failing with Error in atexit._run_exitfuncs followed by a lengthy,
cryptic traceback.

What methods are currently expected to work? I would argue that it is
critical that installing from a local git checkout be supported, to a path
of the user's choosing, and without virtualenv if desired.

Yes, they should all work.

—
Reply to this email directly or view it on GitHub
#124 (comment).

[josh-willis]

I am running python setup.py install --prefix="${HOME}/root/pycbc/branches/cpuopt" on CIT. The exact problem did not recur when I removed several packages in .local, but I still cannot install. It cannot install unittest2 (manually installing that as --user also fails) and if I ignore that I will be unable to run unittests and proceed anyway, it fails with the error:

unittest2 is required to run the test suite with python-2.6
/home/jwillis/.local/lib/python2.6/site-packages/setuptools-18.0.1-py2.6.egg/setuptools/dist.py:294: UserWarning: The version specified ('1a3c36') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
running install
creating /home/jwillis/root/pycbc/branches/cpuopt/etc/pycbc-user-env.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 2] No such file or directory: '/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/test-easy-install-31576.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/

This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

Even though of course I do have write permissions to that directory.

Also, if in your previous comment you are saying that we should be adding a directory that the install procedure should create to PYTHONPATH (and which the pycbc-user-env.sh should add to PYTHONPATH after the install) before the install, then I would argue that this is badly broken. It is essential to development that we be able to easily install isolated installs of just pycbc.

[ahnitz]

Josh,

Thank you for the followup.

On Mon, Jul 13, 2015 at 9:09 AM, josh-willis notifications@github.com
wrote:

I am running python setup.py install
--prefix="${HOME}/root/pycbc/branches/cpuopt" on CIT. The exact problem
did not recur when I removed several packages in .local, but I still
cannot install. It cannot install unittest2 (manually installing that as
--user also fails) and if I ignore that I will be unable to run unittests
and proceed anyway, it fails with the error:

unittest2 is required to run the test suite with python-2.6
/home/jwillis/.local/lib/python2.6/site-packages/setuptools-18.0.1-py2.6.egg/setuptools/dist.py:294:
UserWarning: The version specified ('1a3c36') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
running install
creating /home/jwillis/root/pycbc/branches/cpuopt/etc/pycbc-user-env.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 2] No such file or directory: '/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/test-easy-install-31576.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/

This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

Hmmm.. This is the same error that Sarah reported, but I was never able to
reproduce it. Out of curiosity are you sharing this directory with any
other software installs, such as lalsuite, or is it just for PyCBC. Also,
what version of setuptools are you using?

Even though of course I do have write permissions to that directory.

Also, if in your previous comment you are saying that we should be adding
a directory that the install procedure should create to PYTHONPATH (and
which the pycbc-user-env.sh should add to PYTHONPATH after the install)
before the install, then I would argue that this is badly broken. It is
essential to development that we be able to easily install isolated
installs of just pycbc.

I absolutely agree that people should be installing isolated software
stacks (part of the benefit of virtualenv if you look into it), so you
should not add the directory to your bash_profile, but simply to
temporarily add it to your pythonpath.

The issue is that the packages need to be installed and used immediately
from the directory to complete the rest of the installation. This is the
standard procedure for python packages across the board, so we really
shouldn't try to come up with a custom workaround.

—

Reply to this email directly or view it on GitHub
#124 (comment).

[josh-willis]

Alex,

On Jul 13, 2015, at 5:00 PM, Alex Nitz wrote:

On Mon, Jul 13, 2015 at 9:09 AM, josh-willis notifications@github.com
wrote:

I am running python setup.py install
--prefix="${HOME}/root/pycbc/branches/cpuopt" on CIT. The exact problem
did not recur when I removed several packages in .local, but I still
cannot install. It cannot install unittest2 (manually installing that as
--user also fails) and if I ignore that I will be unable to run unittests
and proceed anyway, it fails with the error:

unittest2 is required to run the test suite with python-2.6
/home/jwillis/.local/lib/python2.6/site-packages/setuptools-18.0.1-py2.6.egg/setuptools/dist.py:294:
UserWarning: The version specified ('1a3c36') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
running install
creating /home/jwillis/root/pycbc/branches/cpuopt/etc/pycbc-user-env.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 2] No such file or directory: '/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/test-easy-install-31576.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/

This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

Hmmm.. This is the same error that Sarah reported, but I was never able to
reproduce it. Out of curiosity are you sharing this directory with any
other software installs, such as lalsuite, or is it just for PyCBC. Also,
what version of setuptools are you using?

I am using setuptools 18.0.1.

This directory is only for PyCBC, and that is the only thing I want installed there.

My general pattern is to have separate install directories for each branch of each LSC authored package I'm using (so, for instance, lalsuite, pycbc-pylal, pycbc-glue, and pycbc). They each have their own install directory trees, with subdirectories for different branches. This is deliberate and not an install configuration I want to change: I want to be sure that by sourcing appropriate config files I know exactly which versions of each package are used, and can easily toggle one without reinstalling another.

The only things I want installed using --user are dependencies that are external to the LSC and reasonably stable, so that I will not be updating them frequently.

Even though of course I do have write permissions to that directory.

Also, if in your previous comment you are saying that we should be adding
a directory that the install procedure should create to PYTHONPATH (and
which the pycbc-user-env.sh should add to PYTHONPATH after the install)
before the install, then I would argue that this is badly broken. It is
essential to development that we be able to easily install isolated
installs of just pycbc.

I absolutely agree that people should be installing isolated software
stacks (part of the benefit of virtualenv if you look into it), so you
should not add the directory to your bash_profile, but simply to
temporarily add it to your pythonpath.

But before switching to setuptools it did NOT need to be explicitly added to PYTHONPATH at all---that was done after the install by sourcing the appropriate configure file. We are now in the perverse situation that we have to create directories that the install process itself should be creating.

The issue is that the packages need to be installed and used immediately
from the directory to complete the rest of the installation. This is the
standard procedure for python packages across the board, so we really
shouldn't try to come up with a custom workaround.

It is vital for a sensible development environment that we be able to install pycbc without it installing a bunch of dependencies into the same target that have no business being there.

Sorry but I have now spent several days debugging this when I needed to be testing cpuopt so that it can be reviewed. It is so far as I can tell impossible to install current pycbc master on CIT in a user environment: pycbc now depends on h5py (even if you don't use the hdf5 workflow) and with python 2.6, h5py depends on unittest2 (even if you aren't running any tests---it will build, but 'import h5py' will fail). And you cannot install unittest2 using --user, because of a bug that has been open since March and not fixed in the process of several releases, despite the fix being simple (one line). I manually did that, commented out the use of setuptools in pycbc, and was then able to get pycbc to install on CIT.

So current install on master is IMHO badly broken, and at the very least should support a distutils only install where that is desired. If there is a way to install without pulling in dependencies using setuptools then that needs to be documented.

Josh

Josh Willis

Associate Professor Email: josh.willis@acu.edu
of Engineering & Physics

Abilene Christian University Phone: (325) 674-2527
Box 27963 Fax: (325) 674-2146
Abilene, TX 79699-7963 Dept: (325) 674-2165

Foster Science Building Room 317

[ahnitz]

On Mon, Jul 13, 2015 at 1:07 PM, josh-willis notifications@github.com
wrote:

Alex,

On Jul 13, 2015, at 5:00 PM, Alex Nitz wrote:

On Mon, Jul 13, 2015 at 9:09 AM, josh-willis notifications@github.com
wrote:

I am running python setup.py install
--prefix="${HOME}/root/pycbc/branches/cpuopt" on CIT. The exact problem
did not recur when I removed several packages in .local, but I still
cannot install. It cannot install unittest2 (manually installing that
as
--user also fails) and if I ignore that I will be unable to run
unittests
and proceed anyway, it fails with the error:

unittest2 is required to run the test suite with python-2.6

/home/jwillis/.local/lib/python2.6/site-packages/setuptools-18.0.1-py2.6.egg/setuptools/dist.py:294:
UserWarning: The version specified ('1a3c36') is an invalid version,
this
may not work as expected with newer versions of setuptools, pip, and
PyPI.
Please see PEP 440 for more details.
running install
creating /home/jwillis/root/pycbc/branches/cpuopt/etc/pycbc-user-env.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 2] No such file or directory:
'/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/test-easy-install-31576.write-test'

The installation directory you specified (via --install-dir, --prefix,
or
the distutils default setting) was:

/home/jwillis/root/pycbc/branches/cpuopt/lib/python2.6/site-packages/

This directory does not currently exist. Please create it and try
again, or
choose a different installation directory (using the -d or
--install-dir
option).

Hmmm.. This is the same error that Sarah reported, but I was never able
to
reproduce it. Out of curiosity are you sharing this directory with any
other software installs, such as lalsuite, or is it just for PyCBC. Also,
what version of setuptools are you using?

I am using setuptools 18.0.1.

This directory is only for PyCBC, and that is the only thing I want
installed there.

My general pattern is to have separate install directories for each branch
of each LSC authored package I'm using (so, for instance, lalsuite,
pycbc-pylal, pycbc-glue, and pycbc). They each have their own install
directory trees, with subdirectories for different branches. This is
deliberate and not an install configuration I want to change: I want to be
sure that by sourcing appropriate config files I know exactly which
versions of each package are used, and can easily toggle one without
reinstalling another.

The only things I want installed using --user are dependencies that are
external to the LSC and reasonably stable, so that I will not be updating
them frequently.

Even though of course I do have write permissions to that directory.

Also, if in your previous comment you are saying that we should be
adding
a directory that the install procedure should create to PYTHONPATH (and
which the pycbc-user-env.sh should add to PYTHONPATH after the install)
before the install, then I would argue that this is badly broken. It
is
essential to development that we be able to easily install isolated
installs of just pycbc.

I absolutely agree that people should be installing isolated software
stacks (part of the benefit of virtualenv if you look into it), so you
should not add the directory to your bash_profile, but simply to
temporarily add it to your pythonpath.

But before switching to setuptools it did NOT need to be explicitly added
to PYTHONPATH at all---that was done after the install by sourcing the
appropriate configure file. We are now in the perverse situation that we
have to create directories that the install process itself should be
creating.

The issue is that the packages need to be installed and used immediately
from the directory to complete the rest of the installation. This is the
standard procedure for python packages across the board, so we really
shouldn't try to come up with a custom workaround.

It is vital for a sensible development environment that we be able to
install pycbc without it installing a bunch of dependencies into the same
target that have no business being there.

Sorry but I have now spent several days debugging this when I needed to be
testing cpuopt so that it can be reviewed. It is so far as I can tell
impossible to install current pycbc master on CIT in a user environment:
pycbc now depends on h5py (even if you don't use the hdf5 workflow) and
with python 2.6, h5py depends on unittest2 (even if you aren't running any
tests---it will build, but 'import h5py' will fail). And you cannot install
unittest2 using --user, because of a bug that has been open since March and
not fixed in the process of several releases, despite the fix being simple
(one line). I manually did that, commented out the use of setuptools in
pycbc, and was then able to get pycbc to install on CIT.

So current install on master is IMHO badly broken, and at the very least
should support a distutils only install where that is desired.

Josh, we should sort out the various issues that crop up, but there is
indeed a way to fall back to the old distutils behavior. Setuptools
provides the following option,

python setup.py install --old-and-unmanageable

If there is a way to install without pulling in dependencies using
setuptools then that needs to be documented.

Josh

Josh Willis

Associate Professor Email: josh.willis@acu.edu
of Engineering & Physics

Abilene Christian University Phone: (325) 674-2527
Box 27963 Fax: (325) 674-2146
Abilene, TX 79699-7963 Dept: (325) 674-2165

Foster Science Building Room 317

—
Reply to this email directly or view it on GitHub
#124 (comment).

[duncan-brown]

Cleaned up the install instructions.

[mairi1996]

what is the best way to install the pycbc?plz recommend