Create virtualenv with python3.5 in local installation

[humitos]

I'm trying to reproduce #2582 and I found a problem in my local instance.

Details

• Project URL: http://readthedocs.org/projects/diofant/

Expected Result

Build the project using python3.5 inside the virtualenv.

Actual Result

The command executed is python3.5 but the virtualenv created is for python2.7

Image

Correct command to be executed

This is the command that creates a virtualenv with python3.5 in my pc:

$ python3.5 -mvirtualenv --python python3.5 --no-site-packages --no-download /home/humitos/readthedocs/readthedocs.org/user_builds/diofant/envs/latest
Already using interpreter /usr/bin/python3.5
Using base prefix '/usr'
New python executable in /home/humitos/readthedocs/readthedocs.org/user_builds/diofant/envs/latest/bin/python3.5
Not overwriting existing python script /home/humitos/readthedocs/readthedocs.org/user_builds/diofant/envs/latest/bin/python (you must use /home/humitos/readthedocs/readthedocs.org/user_builds/diofant/envs/latest/bin/python3.5)
Installing setuptools, pkg_resources, pip, wheel...done.

[agjohnson]

Is this using the docker builder, or the local environment builder?

[humitos]

I suppose that the local environment builder since I didn't change the default values and I have DOCKER_ENABLE = False in readthedocs/settings/base.py

[agjohnson]

Okay, yeah, we don't develop this method for building docs anymore, we only use the local environment to trigger the virtualenv creation. We've talked of just supporting docker build environments as that's all we run in production anyways. This might be solved by using the interpreter name on the other python calls, to force python3.5 to be called

[humitos]

I'm trying the docker build now and there is another problem:

[24/Feb/2017 08:11:17] ERROR [readthedocs.doc_builder.environments:306] (Build) [pip:9.0.0] mem_limit has been moved to host_config in API version 1.19
Traceback (most recent call last):
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/doc_builder/environments.py", line 504, in __enter__
    self.create_container()
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/doc_builder/environments.py", line 627, in create_container
    mem_limit=self.container_mem_limit,
  File "/home/humitos/.virtualenvs/rtd-py2/local/lib/python2.7/site-packages/docker/client.py", line 243, in create_container
    volume_driver
  File "/home/humitos/.virtualenvs/rtd-py2/local/lib/python2.7/site-packages/docker/utils/utils.py", line 552, in create_container_config
    'mem_limit has been moved to host_config in API version 1.19'
InvalidVersion: mem_limit has been moved to host_config in API version 1.19

I had to downgrade the docker-py requirements to

pip install docker-py==1.2.3

Besides, I'm running this version of Docker.

$ docker --version
Docker version 1.13.1, build 092cba3

Now, it passed that section but it failed because I wasn't pulled the docker image yet. I'm pulling it right now but it seems to be very big, so it will take a while.

[humitos]

Now, there is another issue:

[24/Feb/2017 11:42:50] INFO [readthedocs.doc_builder.environments:472] Creating container
[24/Feb/2017 11:42:51] "PUT /api/v2/build/31/ HTTP/1.1" 200 317
[24/Feb/2017 11:42:51] INFO [readthedocs.doc_builder.environments:196] Running in container build-31-project-29-retriever: 'python2.7 -mvirtualenv --no-site-packages --no-download /home/humitos/readthedocs/readthedocs.org/user_builds/retriever/envs/stable' [/home/humitos/readthedocs/readthedocs.org/readthedocs/settings]
[24/Feb/2017 11:42:51] "POST /api/v2/command/ HTTP/1.1" 201 395
[24/Feb/2017 11:42:51] ERROR [readthedocs.doc_builder.environments:306] (Build) [retriever:stable] Command python2.7 -mvirtualenv --no-site-packages --no-download /home/humitos/readthedocs/readthedocs.org/user_builds/retriever/envs/stable failed:
/bin/sh: 1: cd: can't cd to /home/humitos/readthedocs/readthedocs.org/readthedocs/settings
Traceback (most recent call last):
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/projects/tasks.py", line 172, in run
    self.setup_environment()
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/projects/tasks.py", line 315, in setup_environment
    self.python_env.setup_base()
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/doc_builder/python_environments.py", line 100, in setup_base
    bin_path=None,  # Don't use virtualenv bin that doesn't exist yet
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/doc_builder/environments.py", line 313, in run
    return self.run_command_class(cls=self.command_class, cmd=cmd, **kwargs)
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/doc_builder/environments.py", line 350, in run_command_class
    raise BuildEnvironmentWarning(msg)
BuildEnvironmentWarning: Command python2.7 -mvirtualenv --no-site-packages --no-download /home/humitos/readthedocs/readthedocs.org/user_builds/retriever/envs/stable failed:
/bin/sh: 1: cd: can't cd to /home/humitos/readthedocs/readthedocs.org/readthedocs/settings

[24/Feb/2017 11:42:52] INFO [readthedocs.doc_builder.environments:523] Removing container build-31-project-29-retriever
[24/Feb/2017 11:42:52] INFO [readthedocs.doc_builder.environments:537] (Build) [retriever:stable] Build finished

I will try to fix it by myself, but I will probably need some guidance.

[humitos]

OK, the previous error was because I was launching runserver from another path different than where the manage.py is.

Now, the problem is related to permissions since Docker container is using a docs user which doesn't have permissions to edit what humitos (my own user) has created. I will fix that now and continue with this process.

[humitos]

OK, this is a mess but I found a hacky and ugly solution for this.

1. create docs user in my system using the commands from the Dockerfile (be careful here with the group and user id because you can break everything! :) )

    groupadd --gid 205 docs
    useradd -m --uid 1005 --gid 205 docs
   

2. Create the project using the web interface

3. Change the group of the just created folders

    sudo chgrp -R docs user_builds
   

4. Re-run the build

Can you share how is your development environment setup since I think this is not the right way of doing it?

[agjohnson]

My environment is going to be different, because I'm on OSX. I'm running docker on a VM, and much of that won't translate to your set up. You shouldn't need to downgrade the docker python package, just set the version of the docker API that matches your server. On my machine, this is:

DOCKER_ENABLE=True
DOCKER_SOCKET='tcp://127.0.0.1:2375'
DOCKER_VERSION='1.17'

Your fix is actually correct, given you're running docker on your host system. My Vagrant VM just mounts the path with o+wrx,g+wrx,u+wrx permissions.

[humitos]

I'm closing this one. The original problem was solved with #2692 and there is another attempt more general at #3152 and also #3003