Use python3's venv module instead of virtualenv #1396

dongcarl · 2022-11-04T15:15:09Z

Python 3.3 and above provide a standard venv module which provides all the functionality that install.sh needs from virtualenv.

Detect its existence and use it, fallback to virtualenv if not.

PulpCattel · 2022-11-04T17:01:48Z

I've always used python3 -m venv myself when installing manually.

Could we replace virtualenv entirely? Are there any advantages to keep it?

dongcarl · 2022-11-04T17:50:06Z

Would be happy to expand on this patch to remove it entirely, I don't see any major distros using Python < 3.3: https://repology.org/project/python/versions

kristapsk · 2022-11-04T17:58:02Z

I don't see any major distros using Python < 3.3

That wouldn't be important at all, as JM itself requires Python 3.6+.

dongcarl · 2022-11-04T19:17:30Z

That wouldn't be important at all, as JM itself requires Python 3.6+.

Great, updated to just remove virtualenv and replace all mentions of "virtualenv" with "virtual environment" to avoid confusion.

install.sh

kristapsk · 2022-11-05T02:19:49Z

Seems to be working for me. Tested both with default install.sh settings and also switching to non-default Python version (install.sh -p python3.8).

dongcarl · 2022-11-05T03:15:53Z

Let me know if anything else needs to be done

AdamISZ · 2022-11-05T11:31:02Z

@dongcarl thanks for the extra pair of eyes on our installation :) Looks like a good change.

conceptACK based on a quick read up on venv which I was dumb enough not to even know about before. Our usage as can be seen is very vanilla so if there is some obscure functionality of virtualenv which the former doesn't have, it doesn't matter. And the removal of an extra dependency is of course great.

review: ACK based on file search/grep; I believe this covers all references to virtualenv. And the changes to the install.sh script appear correct. I won't comment on the Dockerfiles though they look correct to me, as I never used them.

partial tACK f960ee1 on Ubuntu 20.04 , specifically testing install and script running. The usual pain point here that apart from Debian and Ubuntu I can't really test anything else, but I guess this should be OK.

@kristapsk feel free to merge whenever you think is reasonable.

Btw @dongcarl do you have any thoughts or knowledge about the possibility of building an AppImage for our Qt app here? (obviously not directly related to the PR but thought I'd throw it out there).

kristapsk · 2022-11-05T11:33:50Z

The usual pain point here that apart from Debian and Ubuntu I can't really test anything else, but I guess this should be OK.

I tested on Gentoo.

PulpCattel · 2022-11-05T12:11:28Z

I didn't have time to test this yet, but I remember in the past having to install the python3-venv package on Debian and Ubuntu.
The error message was the same as in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816740
In particular:

$ python3 -m venv venv
The virtual environment was not created successfully because ensurepip
is not
available.  On Debian/Ubuntu systems, you need to install the python3-
venv
package using the following command.

I don't know if it's still a thing, but it might be prudent to add python3-venv to the dependencies?

PulpCattel · 2022-11-05T16:08:10Z

I can reproduce the above on a clean Debian Bullseye. After installing python3-venv, everything else worked for me too.

Python 3.3 and above provide a standard venv module which provides all the functionality that install.sh needs from virtualenv. As of this commit, JM itself requires Python 3.6+ anyway.

dongcarl · 2022-11-07T18:53:44Z

Thanks @PulpCattel! Didn't know that was the case for Debian/Ubuntu. Fixed now.

@AdamISZ

Btw @dongcarl do you have any thoughts or knowledge about the possibility of building an AppImage for our Qt app here? (obviously not directly related to the PR but thought I'd throw it out there).

Unfortunately not, I don't have any experience with AppImage :-/

kristapsk · 2022-11-07T22:20:21Z

Can somebody test this on macOS?

PulpCattel · 2022-11-08T11:57:53Z

11ddec7 fixes the Debian issue mentioned above. Tested successful installation and basic JM scripts.

AdamISZ · 2022-12-17T16:26:43Z

Proposal: merge this and see if a Mac user complains? And if they do they can stay on the 0.9.8 for now (it's not as if we're planning actual functionality changes, for the next release).

Our inability to do testing on Mac has hurt us quite a bit. Have heard user reports recently about not being able to install on Mac, but as usual, we are not getting anyone helping us out in this area. It's probably hurting the project a fair bit.

Personally while I have messed around with Mac VMs in the past, I have no intention of doing so now, so I will not be giving input on it.

In the extreme we could just say that this software is unsupported on Mac, if we still cannot get anyone to test and report on it. But I don't want to do that because I know how many software devs use it as their main environment.

kristapsk · 2022-12-17T19:35:28Z

Our inability to do testing on Mac has hurt us quite a bit. Have heard user reports recently about not being able to install on Mac, but as usual, we are not getting anyone helping us out in this area. It's probably hurting the project a fair bit.

We could try adding macos-latest in addition to ubuntu-latest in CI. I've done this recently for my bitcoin-scripts project. kristapsk/bitcoin-scripts@2eedf0a. But there were discussions about compute cost in #1218.

My brother has a MacBook, will try to get my hands on it.

AdamISZ · 2022-12-18T10:38:59Z

Our inability to do testing on Mac has hurt us quite a bit. Have heard user reports recently about not being able to install on Mac, but as usual, we are not getting anyone helping us out in this area. It's probably hurting the project a fair bit.

We could try adding macos-latest in addition to ubuntu-latest in CI. I've done this recently for my bitcoin-scripts project. kristapsk/bitcoin-scripts@2eedf0a. But there were discussions about compute cost in #1218.

My brother has a MacBook, will try to get my hands on it.

Thanks. Additionally, though, do you agree with my proposal?

kristapsk · 2022-12-19T21:05:33Z

It looks I will not be able to test on mac this year, probably not worth waiting for so long, agree we can merge this.

About being unsupported officially on mac - as I said above, we could add macos-latest to the CI.

kristapsk · 2022-12-24T07:44:45Z

Merging. And will open a new PR that adds macOS testing to CI. Wanted to add macOS testing on top of current master, but had errors because of not having virtualenv installed and getting rid of it simplifies macOS install.

theborakompanioni · 2023-01-03T16:20:00Z

Just fyi, this broke the Jam dev environment where JM is installed inside docker (base image debian:bullseye-20220801-slim) with --docker-install.
See also: joinmarket-webui/jam#581

Step 8/27 : RUN ./install.sh --docker-install --with-local-tor --without-qt
 ---> Running in f665e77112db
./install.sh: line 10: python: command not found
  Installed: (none)
TERM environment variable not set.
Reading package lists...
Building dependency tree...
Reading state information...
Package python3-venv is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python3-venv' has no installation candidate
Dependecies could not be installed. Exiting.
The command '/bin/sh -c ./install.sh --docker-install --with-local-tor --without-qt' returned a non-zero code: 1

Installing python3-venv manually seems to fix the problem. However, it should not be needed, right?

kristapsk · 2023-01-03T16:32:11Z

Yes, this should not break install.sh --docker-install, we should fix install.sh.

kristapsk · 2023-01-03T16:42:39Z

@theborakompanioni Could you check wasn't #1414 instead that broke Jam dev environment setup?

theborakompanioni · 2023-01-03T21:03:59Z

@theborakompanioni Could you check wasn't #1414 instead that broke Jam dev environment setup?

Did not test yet, but I can tell you that it did work, once I installed python3-venv explicitly. See changes in joinmarket-webui/jam#585.

AdamISZ · 2023-01-03T23:34:39Z

@theborakompanioni I'm curious that the error you have is:

./install.sh: line 10: python: command not found

Maybe I just don't understand the environment it's being run in, but why would that happen? And how would manually installing python3-venv fix it? The set of symptoms seems weird, especially considering we have https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/install.sh#L70 also; why would manual be different?

theborakompanioni · 2023-01-03T23:50:53Z

@theborakompanioni I'm curious that the error you have is:

./install.sh: line 10: python: command not found

Maybe I just don't understand the environment it's being run in, but why would that happen? And how would manually installing python3-venv fix it? The set of symptoms seems weird, especially considering we have https://github.com/JoinMarket-Org/joinmarket-clientserver/blob/master/install.sh#L70 also; why would manual be different?

Was surprising for me as well that this really fixed the build. Did not investigate more, as it was working again. Just wanted to let you know.

This is the Dockerfile: https://github.com/joinmarket-webui/jam/blob/aa2b237e5208d867fd5c108da7a0a89d92d15b38/docker/regtest/dockerfile-deps/joinmarket/latest/Dockerfile (Note: it is for a local testing environment only). The file did not change lately and only a bug report of another user made me rebuild the images, thereby reproducing the error.
Running Docker version 20.10.21, build baeda1f.

Edit: Note that ./install.sh: line 10: python: command not found is not fixed with this. But it does not terminate the whole build.

kristapsk · 2023-01-04T06:23:48Z

Edit: Note that ./install.sh: line 10: python: command not found is not fixed with this. But it does not terminate the whole build.

I modified Dockerfile to run bash -x ./install.sh --docker-install --with-local-tor --disable-secp-check --without-qt, that line actually doesn't terminate script, just gives error. It means there is no python available at the run of install.sh, dep_deps_install, which installs Python is called later. Real problem is E: Package 'python3-venv' has no installation candidate.

Step 8/27 : RUN bash -x ./install.sh --docker-install --with-local-tor --disable-secp-check --without-qt
 ---> Running in b7b6450818f0
++ dirname ./install.sh
+ cd .
+ check_exists gmake
+ command -v gmake
+ make=gmake
+ main --docker-install --with-local-tor --disable-secp-check --without-qt
+ python=python3
+ build_local_tor=
+ no_gpg_validation=
+ use_os_deps_check=1
+ use_secp_check=1
+ with_qt=
+ with_jmvenv=1
+ with_sudo=1
+ reinstall=false
+ parse_flags --docker-install --with-local-tor --disable-secp-check --without-qt
+ :
+ case $1 in
+ with_sudo=0
+ with_jmvenv=0
+ shift
+ :
+ case $1 in
+ build_local_tor=1
+ shift
+ :
+ case $1 in
+ use_secp_check=0
+ shift
+ :
+ case $1 in
+ with_qt=0
+ shift
+ :
+ case $1 in
+ break
+ [[ 0 == '' ]]
+ jm_source=/src
+ '[' 0 == 1 ']'
+ jm_root=
+ export PKG_CONFIG_PATH=/lib/pkgconfig:
+ PKG_CONFIG_PATH=/lib/pkgconfig:
+ export LD_LIBRARY_PATH=/lib:
+ LD_LIBRARY_PATH=/lib:
+ export C_INCLUDE_PATH=/include:
+ C_INCLUDE_PATH=/include:
++ num_cores
++ python -c 'import multiprocessing as mp; print(mp.cpu_count())'
./install.sh: line 10: python: command not found
+ MAKEFLAGS='-j '
++ install_get_os
++ os_is_deb
++ echo debian
+ install_os=debian
+ deps_install
+ debian_deps=('curl' 'build-essential' 'automake' 'pkg-config' 'libtool' 'python3-dev' 'python3-pip' 'python3-setuptools' 'python3-venv' 'libltdl-dev')
+ '[' 0 == 1 ']'
+ darwin_deps=('automake' 'libtool')
+ is_python3
+ [[ python3 == python3* ]]
+ return 0
+ [[ 1 != \1 ]]
+ [[ debian == \d\e\b\i\a\n ]]
+ deb_deps_install curl build-essential automake pkg-config libtool python3-dev python3-pip python3-setuptools python3-venv libltdl-dev
+ deb_deps=("${@}")
+ deb_deps_check
+ apt-cache policy curl build-essential automake pkg-config libtool python3-dev python3-pip python3-setuptools python3-venv libltdl-dev
+ grep 'Installed.*none'
  Installed: (none)
+ clear
TERM environment variable not set.
+ sudo_command=
+ '[' 0 == 1 ']'
+ apt-get install -y --no-install-recommends curl build-essential automake pkg-config libtool python3-dev python3-pip python3-setuptools python3-venv libltdl-dev
Reading package lists...
Building dependency tree...
Reading state information...
Package python3-venv is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python3-venv' has no installation candidate
+ return 1
+ return 1
+ echo 'Dependecies could not be installed. Exiting.'
Dependecies could not be installed. Exiting.
+ return 1
The command '/bin/sh -c bash -x ./install.sh --docker-install --with-local-tor --disable-secp-check --without-qt' returned a non-zero code: 1

kristapsk · 2023-01-04T06:35:47Z

What I don't understand is why Python is not available when https://github.com/joinmarket-webui/jam/blob/0a7b709d597469951aa670b9f2008e2435c4e782/docker/regtest/dockerfile-deps/joinmarket/latest/Dockerfile runs apt-get install with python3-dev in list before running JM install.sh. Some Docker or Debian magic I don't get.

AdamISZ added dependencies Pull requests that update a dependency file Installation labels Nov 4, 2022

dongcarl force-pushed the 2022-11-python-venv branch from 7514fa3 to e5f924b Compare November 4, 2022 19:15

dongcarl changed the title ~~install: Use python3's venv module when available~~ Use python3's venv module instead of virtualenv Nov 4, 2022

kristapsk reviewed Nov 5, 2022

View reviewed changes

install.sh Outdated Show resolved Hide resolved

dongcarl force-pushed the 2022-11-python-venv branch from e5f924b to f960ee1 Compare November 5, 2022 03:15

dongcarl added 3 commits November 7, 2022 13:50

install: Use python3's venv module when available

8bbca60

Python 3.3 and above provide a standard venv module which provides all the functionality that install.sh needs from virtualenv. As of this commit, JM itself requires Python 3.6+ anyway.

Dockerfiles: Use Python venv instead of virtualenv

6e3b6ec

docs: Remove now-confusing mentions of virtualenv

11ddec7

dongcarl force-pushed the 2022-11-python-venv branch from f960ee1 to 11ddec7 Compare November 7, 2022 18:52

kristapsk merged commit 71c0a84 into JoinMarket-Org:master Dec 24, 2022

kristapsk mentioned this pull request Dec 24, 2022

Don't install virtualenv globally #649

Closed

This was referenced Jan 3, 2023

fix: install python3-venv in regtest environment joinmarket-webui/jam#585

Merged

Trying to run this locally and getting error when i run "npm run regtest:up" joinmarket-webui/jam#581

Closed

theborakompanioni mentioned this pull request Jan 3, 2023

Can't build it on a Raspberry Pi joinmarket-webui/jam-docker#85

Closed

kristapsk mentioned this pull request Jan 4, 2023

Fix num_cores() when no Python available before install #1419

Closed

theborakompanioni mentioned this pull request Jan 31, 2023

fix(build): install python3-venv joinmarket-webui/jam-docker#94

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use python3's venv module instead of virtualenv #1396

Use python3's venv module instead of virtualenv #1396

dongcarl commented Nov 4, 2022

PulpCattel commented Nov 4, 2022

dongcarl commented Nov 4, 2022

kristapsk commented Nov 4, 2022

dongcarl commented Nov 4, 2022 •

edited

Loading

kristapsk commented Nov 5, 2022

dongcarl commented Nov 5, 2022

AdamISZ commented Nov 5, 2022

kristapsk commented Nov 5, 2022

PulpCattel commented Nov 5, 2022

PulpCattel commented Nov 5, 2022

dongcarl commented Nov 7, 2022

kristapsk commented Nov 7, 2022

PulpCattel commented Nov 8, 2022

AdamISZ commented Dec 17, 2022

kristapsk commented Dec 17, 2022

AdamISZ commented Dec 18, 2022

kristapsk commented Dec 19, 2022

kristapsk commented Dec 24, 2022

theborakompanioni commented Jan 3, 2023

kristapsk commented Jan 3, 2023

kristapsk commented Jan 3, 2023

theborakompanioni commented Jan 3, 2023

AdamISZ commented Jan 3, 2023

theborakompanioni commented Jan 3, 2023 •

edited

Loading

kristapsk commented Jan 4, 2023

kristapsk commented Jan 4, 2023

Use python3's venv module instead of virtualenv #1396

Use python3's venv module instead of virtualenv #1396

Conversation

dongcarl commented Nov 4, 2022

PulpCattel commented Nov 4, 2022

dongcarl commented Nov 4, 2022

kristapsk commented Nov 4, 2022

dongcarl commented Nov 4, 2022 • edited Loading

kristapsk commented Nov 5, 2022

dongcarl commented Nov 5, 2022

AdamISZ commented Nov 5, 2022

kristapsk commented Nov 5, 2022

PulpCattel commented Nov 5, 2022

PulpCattel commented Nov 5, 2022

dongcarl commented Nov 7, 2022

kristapsk commented Nov 7, 2022

PulpCattel commented Nov 8, 2022

AdamISZ commented Dec 17, 2022

kristapsk commented Dec 17, 2022

AdamISZ commented Dec 18, 2022

kristapsk commented Dec 19, 2022

kristapsk commented Dec 24, 2022

theborakompanioni commented Jan 3, 2023

kristapsk commented Jan 3, 2023

kristapsk commented Jan 3, 2023

theborakompanioni commented Jan 3, 2023

AdamISZ commented Jan 3, 2023

theborakompanioni commented Jan 3, 2023 • edited Loading

kristapsk commented Jan 4, 2023

kristapsk commented Jan 4, 2023

dongcarl commented Nov 4, 2022 •

edited

Loading

theborakompanioni commented Jan 3, 2023 •

edited

Loading