pre-built Docker image #42

rdavydov · 2022-11-14T22:34:37Z

This is not the first time people asking for a pre-built Docker image, so I will add this to my TODO list.

Originally posted by @rdavydov in Tkd-Alex#651 (comment)

rdavydov · 2022-11-14T22:53:27Z

Reserved an address https://hub.docker.com/r/rdavidoff/twitch-channel-points-miner-v2

rdavydov · 2022-11-15T18:36:03Z

Docker container is only useful when you already have a cookie file. Docker can only run CLI programs, so it will not start Google Chrome in the non-headless mode.

However, there are workarounds. I think I should experiment with a VNC server in a container, this may be a good cross-plaform approach. More info: https://www.howtogeek.com/devops/how-to-run-gui-applications-in-a-docker-container/

rdavydov · 2022-11-15T19:12:42Z

~~Branch for Docker-related work https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/tree/rdavydov-docker~~

rdavydov · 2022-11-15T19:13:39Z

GitHub action for deploying

~~Disabled GH Actions for now, because they need to be configured first, otherwise they are raising errors on commits ❌~~

bttfw · 2022-11-15T21:58:35Z

First off all thanks for continuing with the project

One question why are you coping the run.py into the docker ?

rdavydov · 2022-11-15T22:22:06Z

First off all thanks for continuing with the project

One question why are you coping the run.py into the docker ?

You're welcome.

Because of this #29

bttfw · 2022-11-15T22:28:49Z

So everytime you change something in your run we will have to redeploy the container ?

In linux it gives this in the error logs

exec /usr/local/bin/python: exec format error

rdavydov · 2022-11-16T02:06:55Z

Good point. I've just started using Docker today, so I'm still learning about it. Copying was indeed unnecessary.

Kazushi-0 · 2022-11-16T13:38:37Z

Any update on armv7 support?

rdavydov · 2022-11-16T13:46:56Z

Any update on armv7 support?

You can test it with 1.5.0. Pretty good chance that it should work now.

rdavydov · 2022-11-16T13:47:44Z

Man, GitHub action that builds a docker image takes ages to complete. It's been more than 2 hours already.
https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/actions/runs/3478905407

I guess I should just build and push it manually. Will do so, stay tuned for an update.

rdavydov · 2022-11-16T14:22:38Z

1.5.0 is now on https://hub.docker.com/r/rdavidoff/twitch-channel-points-miner-v2

Please test and report back any issues and suggestions!

Kazushi-0 · 2022-11-16T14:50:08Z

exec /usr/local/bin/python: exec format error

well I think that cuz im using an arm processor

rdavydov · 2022-11-16T14:59:08Z

exec /usr/local/bin/python: exec format error

well I think that cuz im using an arm processor

Right, seems like I need to create a custom builder for Docker to make it build multi-arch images. Stay tuned.

Kazushi-0 · 2022-11-16T15:06:20Z

alright

btw thx for ur all hard work

rdavydov · 2022-11-16T17:00:36Z

Notes for me:

docker buildx create --name multiarch
docker buildx use multiarch
docker buildx inspect
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t rdavidoff/twitch-channel-points-miner-v2:latest --push .

ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1 seems to be helping with Rust 1.41.1 does not match extension requirement >=1.48.0

but it still hangs on Building wheel for pandas

upgrading pip may fix this hanging, checking.

rdavydov · 2022-11-16T17:39:36Z

Well, it seems that it is not hanging, but compiling pandas from sources takes an awful lot of time.

Trying to create Dockerfile.arm32v7 with pip install --index-url=https://www.piwheels.org/simple --no-cache-dir -r requirements.txt

piwheels are pre-compiled binary Python packages specifically for the Raspberry Pi.

rdavydov · 2022-11-16T18:38:41Z

Dockerfile.arm32v7 didn't help, had to add a condition to Dockerfile:

ARG TARGETPLATFORM

  && if [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then \
        pip install --index-url=https://www.piwheels.org/simple --no-cache-dir -r requirements.txt; \
     else \
        pip install -r requirements.txt; \
     fi \

Condition works, but now it can't get pillow from piwheels 😩:

#26 370.9 ERROR: Could not find a version that satisfies the requirement pillow (from versions: none)
#26 370.9 ERROR: No matching distribution found for pillow

Now I'm trying to only get pandas from piwheels:

  && if [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then \
        pip install --index-url=https://www.piwheels.org/simple -U pandas; \
     fi \
  && pip install -r requirements.txt \

Man, this is tiresome. And pandas are only needed for Analytics to work! 😫

rdavydov · 2022-11-16T18:50:00Z

Same error for pandas.

Indeed, armv7 is not listed in https://pypi.org/project/pandas/#files

I'll try one more trick with apt-get -y install python3-pandas

rdavydov · 2022-11-16T19:11:07Z

@Kazushi-0 are you using Analytics on your armv7 miner instance(s)? Maybe I should just drop pandas for armv7.

rdavydov · 2022-11-16T19:44:02Z

Notes for me:

#26 631.9           import numpy
#26 631.9       ModuleNotFoundError: No module named 'numpy'

pandas also need numpy

rdavydov · 2022-11-16T20:06:31Z

Man, this was quite a challenge!

  && if [ "$TARGETPLATFORM" = "linux/arm/v7" ]; then \
        apt-get -y install python3-pandas; \
        sed -i '/pandas/d' requirements.txt; \
     fi \
  && pip install -r requirements.txt \

seems to did the job.

rdavydov · 2022-11-16T20:09:10Z

1.5.0 is now on https://hub.docker.com/r/rdavidoff/twitch-channel-points-miner-v2 for linux/amd64, linux/arm64 and linux/arm/v7.

Please test and report back any issues and suggestions!

@Kazushi-0 👆

bttfw · 2022-11-16T22:50:49Z

Starts on pi3 but gets allot of

16/11 23:49:04 - Error with GQLOperations (ReportMenuItem): HTTPSConnectionPool(host='gql.twitch.tv', port=443): Max retries exceeded with url: /gql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff7d91d220>: Failed to establish a new connection: [Errno 110] Connection timed out'))

and streamer does not exist

Do i need to regenrate cookies or change the run.py ?

rdavydov · 2022-11-16T23:34:30Z

gets allot of

16/11 23:49:04 - Error with GQLOperations (ReportMenuItem): HTTPSConnectionPool(host='gql.twitch.tv', port=443): Max retries exceeded with url: /gql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff7d91d220>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Either a connection issue or too many concurrent connections. If your miner is still able to get WATCH and CLAIM points between those errors, then you can just ignore them.

and streamer does not exist

Does this streamer really exist? Better double-check that. 😉

Do i need to regenrate cookies or change the run.py ?

If you already have a "console" cookie and your miner is claiming bonuses, you don't need to obtain a new cookie.
Check your run.py for this non-existent streamer.

Kazushi-0 · 2022-11-17T07:24:07Z

sorry for late reply...
anyway IT WORK perfectly fine with no error and new features add to miner

btw Thx for making Docker image for easy installation

Kazushi-0 · 2022-11-17T09:07:16Z

apt-get -y install python3-pandas

about the Analytics if you cant do that then i can live without it...

rdavydov · 2022-11-17T10:00:28Z

about the Analytics if you cant do that then i can live without it...

I managed to add pandas even for armv7, so Analytics should work just fine.
But it is still better to disable Analytics on low-memory (RAM) devices. Memory consumption drops significantly when you disable it.

I added it just for the sake of challenge. :)

Kazushi-0 · 2022-11-17T11:25:11Z

what a gigachad you are dude

btw did you already release to docker image?for Analytics?

Kazushi-0 · 2022-11-17T11:32:28Z

what a gigachad you are dude

btw did you already release to docker image?for Analytics?

Is this how to enable Analytics?or im just doing it wrong

rdavydov · 2022-11-17T11:47:06Z

You're doing it right, but it can't see the pandas module. 😩

Did you pull the latest rdavidoff/twitch-channel-points-miner-v2 image?

What armv7 device are you using BTW? How much RAM do you have in there?

Kazushi-0 · 2022-11-17T12:40:50Z

You're doing it right, but it can't see the pandas module. 😩

Did you pull the latest rdavidoff/twitch-channel-points-miner-v2 image?

What armv7 device are you using BTW? How much RAM do you have in there?

Yes, im pulling the latest docker image Os/Arch linux/arm/v7
Raspberry pi 4 model B
4GB Ram

rdavydov · 2022-11-17T12:57:35Z

Notes for me:

Running an armv7 image on a different platform:

docker run --platform linux/arm/v7
-v C:\Users\User\Downloads\Twitch-Channel-Points-Miner-v2\run.py:/usr/src/app/run.py:ro --rm docker.io/rdavidoff/twitch-channel-points-miner-v2:latest@sha256:ebcc5b555a9ea555daa541e3cbb74cf9e040d2f9156bbf82a142f5e2dc6306a7

where sha256:ebcc5b555a9ea555daa541e3cbb74cf9e040d2f9156bbf82a142f5e2dc6306a7 is a hash for the armv7 image from Docker Hub.

I'm going to replace python3-pandas with (or add) apt-get -y install python-pandas in the Dockerfile. This might help.

Kazushi-0 · 2022-11-17T13:08:25Z

Notes for me:

Running an armv7 image on a different platform:
docker run --platform linux/arm/v7
-v C:\Users\User\Downloads\Twitch-Channel-Points-Miner-v2\run.py:/usr/src/app/run.py:ro --rm docker.io/rdavidoff/twitch-channel-points-miner-v2:latest@sha256:ebcc5b555a9ea555daa541e3cbb74cf9e040d2f9156bbf82a142f5e2dc6306a7
where sha256:ebcc5b555a9ea555daa541e3cbb74cf9e040d2f9156bbf82a142f5e2dc6306a7 is a hash for the armv7 image from Docker Hub.

I'm going to replace python3-pandas with (or add) apt-get -y install python-pandas in the Dockerfile. This might help.

do you have to put -p 5000:5000 on docker run?

rdavydov · 2022-11-17T13:16:58Z

do you have to put -p 5000:5000 on docker run?

Yes, for Analytics. This publishes or exposes the port (-p, --expose). It binds port 5000 of the container to TCP port 5000 on the host machine.

Kazushi-0 · 2022-11-17T13:19:29Z

do you have to put -p 5000:5000 on docker run?

Yes, for Analytics. This publishes or exposes the port (-p, --expose). It binds port 5000 of the container to TCP port 5000 on the host machine.

alright btw how ur testing doing?

rdavydov · 2022-11-17T13:20:55Z

alright btw how ur testing doing?

It's currently building images https://github.com/rdavydov/Twitch-Channel-Points-Miner-v2/actions/runs/3488493817/jobs/5837384999

I'll post the result, stay tuned.

rdavydov · 2022-11-17T14:04:35Z

Didn't work. python-pandas is for Python 2 and python3-pandas is for Python 3.

What really strange is this:

root@89662d16d8ab:/usr/src/app# apt-get -y install python3-pandas
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-pandas is already the newest version (0.23.3+dfsg-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@89662d16d8ab:/usr/src/app# python
Python 3.8.15 (default, Nov 16 2022, 07:31:08)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

python3-pandas is already the newest version (0.23.3+dfsg-3).
@
ModuleNotFoundError: No module named 'pandas'

🤷‍♂️🙄

Kazushi-0 · 2022-11-17T14:11:49Z

Didn't work. python-pandas is for Python 2 and python3-pandas is for Python 3.

What really strange is this:

root@89662d16d8ab:/usr/src/app# apt-get -y install python3-pandas
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-pandas is already the newest version (0.23.3+dfsg-3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@89662d16d8ab:/usr/src/app# python
Python 3.8.15 (default, Nov 16 2022, 07:31:08)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

python3-pandas is already the newest version (0.23.3+dfsg-3). @ ModuleNotFoundError: No module named 'pandas'

🤷‍♂️🙄

same i got the same error while it already the newest version
btw is this cause by the armv7 processor?

rdavydov · 2022-11-17T14:18:14Z

Yes. You can disable analytics for now and run the miner without it while I'm figuring out the solution to make it see pandas.

Kazushi-0 · 2022-11-17T14:31:05Z

Yes. You can disable analytics for now and run the miner without it while I'm figuring out the solution to make it see pandas.

alright... goodluck

rdavydov · 2022-11-17T14:48:07Z

root@89662d16d8ab:~# find / -name pandas*
/usr/lib/python3/dist-packages/pandas
/usr/lib/python3/dist-packages/pandas-0.23.3+dfsg.egg-info

root@89662d16d8ab:~# python -c 'import sys; print(sys.path);'
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages']

root@89662d16d8ab:~# echo "/usr/lib/python3/dist-packages/" >> /usr/local/lib/python3.8/site-packages/site-packages.pth

root@89662d16d8ab:~# python -c 'import sys; print(sys.path);'
['', '/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages', '/usr/lib/python3/dist-packages']

Now it sees modules installed by apt-get. But:

root@89662d16d8ab:~# python
Python 3.8.15 (default, Nov 16 2022, 07:31:08)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pandas/__init__.py", line 18, in <module>
    raise ImportError(
ImportError: Missing required dependencies ['numpy']
>>> import numpy
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

Man, this is too much. 😞

rdavydov · 2022-11-17T15:54:49Z

I found the culprit.

root@89662d16d8ab:/usr/lib/python3/dist-packages/numpy# find / -name numpy*
/usr/include/python3.7m/numpy
/usr/lib/python3/dist-packages/numpy-1.16.2.egg-info
/usr/lib/python3/dist-packages/numpy
/usr/lib/python3/dist-packages/numpy/distutils/numpy_distribution.py
/usr/lib/python3/dist-packages/numpy/distutils/__pycache__/numpy_distribution.cpython-37.pyc

apt's version of numpy is for python 3.7.

And it doesn't work with python 3.8.

So I guess we're gonna have to downgrade to python 3.7 on armv7.

This needs some testing.

rdavydov · 2022-11-17T21:11:44Z

update-alternatives method and alias python=python3.7 didn't work.

Created a different Dockerfile for armv7 called Dockerfile.arm32v7, added a new step to the GH action "deploy-docker", but GH doesn't understand dockerfile option:

Warning: Unexpected input(s) 'dockerfile', valid inputs are ['add-hosts', 'allow', 'build-args', 'build-contexts', 'builder', 'cache-from', 'cache-to', 'cgroup-parent', 'context', 'file', 'labels', 'load', 'network', 'no-cache', 'no-cache-filters', 'outputs', 'platforms', 'pull', 'push', 'secrets', 'secret-files', 'shm-size', 'ssh', 'tags', 'target', 'ulimit', 'github-token']

Adding ln -sf /usr/bin/python3.7 /usr/local/bin/python as the last command in the RUN sequence did work in the end, but then I discovered that other modules such as colorama and irc don't work.

Now there are two solutions:

Drop Analytics support from armv7 Docker image
Wait several hours for armv7 Docker image to build after each commit

🤔

I guess I will stay with a second one. I can just manually stop the GH action when it is not an important commit and when I don't want it to rebuild images.

P.S. Also changed from FROM python:3.8-slim-buster to FROM python:3.11-slim-buster accepting a pull request (without 3.11.0).

So much work just to make Analytics work on armv7. 🙄 Now I know what it feels like to be a DevOps engineer. 😂

rdavydov · 2022-11-18T01:31:21Z

Not so bad, it took "only" 2h 40m 7s to build the armv7 image. 😏

@Kazushi-0 please pull the latest armv7 image and check again. Analytics should finally work!

Kazushi-0 · 2022-11-18T03:34:49Z

Not so bad, it took "only" 2h 40m 7s to build the armv7 image. 😏

@Kazushi-0 please pull the latest armv7 image and check again. Analytics should finally work!

Now im having a problem pulling the image saying docker: invalid reference format

Docker run command :
docker run --name twitch-miner -v /my/path/run.py:/usr/src/app/run.py:ro -p 5000:5000 rdavidoff:sha@3e79383d3b7e

Edited : after doing that command add volume to my cookie and other stuff

rdavydov · 2022-11-18T08:50:16Z

docker run --name twitch-miner -v /my/path/run.py:/usr/src/app/run.py:ro -p 5000:5000 docker.io/rdavidoff/twitch-channel-points-miner-v2:latest@sha256:3e79383d3b7eb28a8776dc4ba8ec6fbfb929dc971b89dae16cf8b37505c63a8b

Kazushi-0 · 2022-11-18T13:58:25Z

docker run --name twitch-miner -v /my/path/run.py:/usr/src/app/run.py:ro -p 5000:5000 docker.io/rdavidoff/twitch-channel-points-miner-v2:latest@sha256:3e79383d3b7eb28a8776dc4ba8ec6fbfb929dc971b89dae16cf8b37505c63a8b

again sorry for late reply
and IT FRICKING WORK!!

HOLY SHIT, THX MAN

rdavydov added 🧱 enhancement New feature or request ➕ todo labels Nov 14, 2022

rdavydov self-assigned this Nov 14, 2022

rdavydov removed the ➕ todo label Nov 16, 2022

rdavydov closed this as completed Nov 21, 2022

rdavydov added the 📘 documentation Improvements or additions to documentation label Nov 28, 2022

rdavydov mentioned this issue Apr 1, 2023

Inefficient docker-ing? #232

Closed

pre-built Docker image #42

pre-built Docker image #42

Comments

rdavydov commented Nov 14, 2022

rdavydov commented Nov 14, 2022

rdavydov commented Nov 15, 2022

rdavydov commented Nov 15, 2022 • edited Loading

rdavydov commented Nov 15, 2022 • edited Loading

bttfw commented Nov 15, 2022

rdavydov commented Nov 15, 2022

bttfw commented Nov 15, 2022

rdavydov commented Nov 16, 2022

Kazushi-0 commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

Kazushi-0 commented Nov 16, 2022

rdavydov commented Nov 16, 2022

Kazushi-0 commented Nov 16, 2022 • edited Loading

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022 • edited Loading

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

rdavydov commented Nov 16, 2022

bttfw commented Nov 16, 2022

rdavydov commented Nov 16, 2022 • edited Loading

Kazushi-0 commented Nov 17, 2022 • edited Loading

Kazushi-0 commented Nov 17, 2022

rdavydov commented Nov 17, 2022 • edited Loading

Kazushi-0 commented Nov 17, 2022 • edited Loading

Kazushi-0 commented Nov 17, 2022

rdavydov commented Nov 17, 2022 • edited Loading

Kazushi-0 commented Nov 17, 2022 • edited Loading

rdavydov commented Nov 17, 2022

Kazushi-0 commented Nov 17, 2022

rdavydov commented Nov 17, 2022

Kazushi-0 commented Nov 17, 2022

rdavydov commented Nov 17, 2022

rdavydov commented Nov 17, 2022

Kazushi-0 commented Nov 17, 2022 • edited Loading

rdavydov commented Nov 17, 2022

Kazushi-0 commented Nov 17, 2022 • edited Loading

rdavydov commented Nov 17, 2022

rdavydov commented Nov 17, 2022 • edited Loading

rdavydov commented Nov 17, 2022 • edited Loading

rdavydov commented Nov 18, 2022

Kazushi-0 commented Nov 18, 2022 • edited Loading

rdavydov commented Nov 18, 2022

Kazushi-0 commented Nov 18, 2022 • edited Loading

rdavydov commented Nov 15, 2022 •

edited

Loading

rdavydov commented Nov 15, 2022 •

edited

Loading

Kazushi-0 commented Nov 16, 2022 •

edited

Loading

rdavydov commented Nov 16, 2022 •

edited

Loading

rdavydov commented Nov 16, 2022 •

edited

Loading

Kazushi-0 commented Nov 17, 2022 •

edited

Loading

rdavydov commented Nov 17, 2022 •

edited

Loading

Kazushi-0 commented Nov 17, 2022 •

edited

Loading

rdavydov commented Nov 17, 2022 •

edited

Loading

Kazushi-0 commented Nov 17, 2022 •

edited

Loading

Kazushi-0 commented Nov 17, 2022 •

edited

Loading

Kazushi-0 commented Nov 17, 2022 •

edited

Loading

rdavydov commented Nov 17, 2022 •

edited

Loading

rdavydov commented Nov 17, 2022 •

edited

Loading

Kazushi-0 commented Nov 18, 2022 •

edited

Loading

Kazushi-0 commented Nov 18, 2022 •

edited

Loading