-
-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add dockerfile for automated builds to dockerhub #165
Conversation
dockerfile
Outdated
@@ -0,0 +1,9 @@ | |||
FROM ubuntu:latest | |||
RUN apt-get update && apt-get upgrade -y | |||
RUN apt-get install python3-pip python3 python-is-python3 -y |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pin to Python 3.9
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fubuloubu I am having a problem with pinning to 3.9 where it wants me to interactively choose timezone etc. I have pinned to 3.8 in the meantime.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pinning 3.8 is great. Kind of wild to have to select a timezone, must be a way around that 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it has to do with tzdata which i think is a requirement in apt-get in python3.9
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Europe/Moscow
RUN apt-get install -y tzdata
if we want to pin to 3.9
https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image
dockerfile
Outdated
@@ -0,0 +1,9 @@ | |||
FROM ubuntu:latest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you ever try this on alpine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no I haven't built this on alpine yet
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you know if I can do this one after another in the dockerfile? do you know how I can make sure that each build is automatically tagged properly as ubuntu or apline in the dockerhub repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to build with alpine and running into a lot of problems so I'm going to leave it as ubuntu for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the build that isn't working for me:
FROM python:alpine3.13
RUN pip3 install --upgrade eth-ape
RUN ape plugins add ens --yes
RUN ape plugins add etherscan --yes
RUN ape plugins add infura --yes
RUN ape plugins add solidity --yes
RUN ape plugins add vyper --yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when trying to use alpine, in addition to errors building wheels for cffi
and maturin
I get stuck in the build process on when it starts to try to download the wheels for urllib
In file included from c/_cffi_backend.c:2:
/usr/local/include/python3.9/Python.h:11:10: fatal error: limits.h: No such file or directory
11 | #include <limits.h>
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for cffi
Running setup.py clean for cffi
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
In file included from c/_cffi_backend.c:2:
/usr/local/include/python3.9/Python.h:11:10: fatal error: limits.h: No such file or directory
11 | #include <limits.h>
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mye4g0pn/cffi_9920a769f1fa4e24918c148f798fa688/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mye4g0pn/cffi_9920a769f1fa4e24918c148f798fa688/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c9usrg37/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-afl4esu6/overlay --compile --install-headers /tmp/pip-build-env-afl4esu6/overlay/include/python3.9/cffi Check the logs for full command output.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/cf/5a/25aeb636baeceab15c8e57e66b8aa930c011ec1c035f284170cacb05025e/PyNaCl-1.4.0.tar.gz#sha256=54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505 (from https://pypi.org/simple/pynacl/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*). Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-zminvb6v/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-afl4esu6/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
Collecting web3[tester]<6.0.0,>=5.18.0
Downloading web3-5.23.1-py3-none-any.whl (486 kB)
Downloading web3-5.23.0-py3-none-any.whl (485 kB)
Collecting ipfshttpclient==0.7.0
Downloading ipfshttpclient-0.7.0-py3-none-any.whl (82 kB)
Collecting web3[tester]<6.0.0,>=5.18.0
Downloading web3-5.22.0-py3-none-any.whl (484 kB)
Downloading web3-5.21.0-py3-none-any.whl (482 kB)
Downloading web3-5.20.1-py3-none-any.whl (479 kB)
Downloading web3-5.20.0-py3-none-any.whl (480 kB)
Collecting websockets<9.0.0,>=8.1.0
RuntimeError: cargo not found in PATH. Please install rust (https://www.rust-lang.org/tools/install) and try again
----------------------------------------
ERROR: Failed building wheel for maturin
Failed to build maturin
ERROR: Could not build wheels for maturin which use PEP 517 and cannot be installed directly
File "setup.py", line 58, in run
raise RuntimeError(
RuntimeError: cargo not found in PATH. Please install rust (https://www.rust-lang.org/tools/install) and try again
----------------------------------------
ERROR: Failed building wheel for maturin
Failed to build maturin
ERROR: Could not build wheels for maturin which use PEP 517 and cannot be installed directly
Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Downloading urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Downloading urllib3-1.26.1-py2.py3-none-any.whl (136 kB)
Downloading urllib3-1.26.0-py2.py3-none-any.whl (136 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Downloading urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
Downloading urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
Downloading urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
Downloading urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
Downloading urllib3-1.24-py2.py3-none-any.whl (117 kB)
Downloading urllib3-1.23-py2.py3-none-any.whl (133 kB)
Downloading urllib3-1.22-py2.py3-none-any.whl (132 kB)
Downloading urllib3-1.21.1-py2.py3-none-any.whl (131 kB)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Skipping Alpine for now
Dockerfile
Outdated
RUN apt-get update && apt-get upgrade -y | ||
RUN apt-get install python3-pip python3 python-is-python3 -y | ||
RUN pip install --upgrade eth-ape | ||
RUN ape plugins add ens -y |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For scripts, it is generally more readable to use the longer option names, such as --yes
in this case instead of -y
. Shorthand options are great for adhoc, long-hand options are great for scripts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll make the requested changes for this. I also I got an email with your note about docker compose, not sure where that comment went, but would love to chat with you about this, maybe at Wednesday meeting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yeah, the docker-compose thing might be more complicated than I though. It might make sense for ape plugin dependencies though, like running the hardhat network.
Another usability for ape
would be to copy other package managers and allow multiple args for the plugins when adding. That way, you can just do:
ape plugins add ens etherscan infura solidity
and it will install all 4.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing we can do is create a build_ape.sh
script that takes parameters such as which plugins to use. Or maybe it figures it out other ways. Either way, this script can be copied in the container and ran with parameters set from environment variables e.g. --build-arg ARG_NAME=ARG_VALUE
Edit: This will require more thought, I am just trying to think of how to make the containerizing ape more customizable. I think docker-compose will not necessarily solve this on its own. However, I sometimes prefer having compose files because they are easier to start up and tweak and sometimes I just want to run everything with a simple command.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I believe the reason we went with dockerfile and not compose is that we didn't want to have it spinning up an app, we wanted it to be a CLI command. I do have a compose file that I have not committed, but we are not using it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I add docker build to testing for this? |
RUN ape plugins add etherscan --yes | ||
RUN ape plugins add infura --yes | ||
RUN ape plugins add solidity --yes | ||
RUN ape plugins add vyper --yes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of CLI Dockerfiles I see use ENTRYPOINT so it is like running ape when you run the container, which means if you don't give it commands, it just prints the help and exits... I think you mentioned you looked into this pattern, but just thought I'd say that it is common from what I have seen. How much have we considered this approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is GETH example: https://github.com/islishude/geth-docker/blob/master/Dockerfile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Notice it builds uses alpine, uses geth
command as the entrypoint, and it builds from source!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't used ENTRYPOINT before however I want to make sure that we can do other CLI stuff other than just running ape console
for example ape accounts
and ape compile
etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be what ever command the user types in!
They would run the container with the same args that they run ape
with. So if they run the container with console
as the argument, it would launch a console, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like this docker run apeworx/ape console
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be what ever command the user types in!
They would run the container with the same args that they runape
with. So if they run the container withconsole
as the argument, it would launch a console, etc.
yes then the problem here then becomes that they can't do other things if they need to with the cli within that image, right?
Dockerfile
Outdated
FROM ubuntu:20.04 | ||
RUN apt-get update && apt-get upgrade --yes | ||
RUN apt-get install python3-pip python3.8 --yes | ||
RUN pip3 install --upgrade eth-ape |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm.... I'd actually expect this to build from source. That way, you download the repo and can build the container for dev purposes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so like this?
RUN git clone https://github.com/ApeWorX/ape
RUN pip3 install -e ./ape
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking something like python setup.py sdist
and then pip install dist/eth-ape*
since this Dockerfile is in the same repo.
From CI/CD perspective, if we want to run a job to deploy the image to Dockerhub on release, there might be a race condition if relying on the build being available on pypi first
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh youd have to copy the source files to the container as well as use a .dockerignore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lol maybe git
is better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was testing this with git and i'm having a bunch of failures. I'll keep trying and post the failures if i can't git it to work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as per conversation with @fubuloubu:
copy the files from the local directory into the docker image and then use pip install
benefit: not relying on deployed version, it will be whatever is on the master branch and be easier to debug and avoid race condition
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would agree with copying files from local directory as that is also easiest to build for the purposes of development and testing new features on a local instance via a built image.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might have approved too early, I think there is some more to figure out.
The main thing, I think, is to not build from pypi but to use the source files. Additionally, I think it is best to have the container function the exact same as the CLI tool itself, so running the container should exit immediately and it should use the simple ape
as its ENTRYPOINT. It might take some figuring out. Finally, we should try to use alpine to cut the cruft of the image. This should be able to build fast!
ape as an entry point then the user can't do other things if they need to with the cli within that image, right? Not sure if this is a problem or not. I wasn't able to successfully get alpine to build unfortunately. There are many dependency errors and issues with pip install. |
This article is why I originally decided to go with Ubuntu as a base image. https://megamorf.gitlab.io/2020/05/06/why-it-s-better-not-to-use-alpine-linux-for-python-projects/ |
This article details why I'm running into errors trying to build with alpine: https://pythonspeed.com/articles/alpine-docker-python/ |
as per this article I tried to reduce the build size by using python:3.8-slim however then I came into a bunch of errors with
|
Ok, that makes sense! Ubuntu works. |
Correct, they wouldn't be able to do anything that is not an Edit: |
todo: skip for now: |
source code copying is working except that installing vyper step is failing.
then when I go to compile docker
|
@sabotagebeats that warning pops up when the plugin isn't currently functioning properly. Try debugging it locally, and then fixing the |
this is a preexisting issue: ApeWorX/ape-vyper#22 |
@sabotagebeats yes, so the solution to that issue will solve this one and allow the plugin to function in the image. However we should take this oppurtunity to think through how the image is built here, and make sure that all plugins are functioning for when we distribute the image. Perhaps this needs to be "tested" in the image by making sure that |
yes maybe we can do build testing for compilation functionality of the docker image? |
It should be pretty straight forward to upgrade the vyper plugin. Let me know if not! |
|
@sabotagebeats build it locally first, and the move the |
So I did try to build this locally but I don't have the |
the let's stick with copying everything over.can always fix it |
When the vyper contracts and solidity contractts are combined in the same folder:
this causes the contracts not to compile |
this is happening in |
filed issue #178 |
See response (and closure) e.g. just change to |
You should be able to bind it during the install command, e.g. |
f267fc7
to
2b20137
Compare
i had removed git from the apt-get install and it broke the build so i put it back |
This makes sense, |
Hmm, I did not mess with the files that are failing the type check, and I tried to merge |
looks like merging main fixed the types issue :) |
Is this good to merge and see how docker hub takes it? |
yes I believe we can merge this and then I can review the integration to make sure it was successful. |
this is building properly on dockerhub @fubuloubu @unparalleled-js |
I think this happened today because we did 0.1.1 by accident, and Docker also caught it |
What I did
added dockerfile and updated readme to include information about dockerhub
fixes: #
How I did it
added dockerfile
updated readme
How to verify it
we will verify it in the integration
Checklist
New test cases have been added and are passing