Dockerized Mycodo #637
Comments
|
Below are commands to get it up and running on your system (any linux system, not just a Raspberry Pi). If you have Mycodo already running from a standard install, first stop the services with these commands: sudo service mycodo stop
sudo service mycodoflask stop
sudo service nginx stop
sudo service influxdb stopCheckout the mycodo_docker branch and install prerequisites (docker, docker-compose): git clone https://github.com/kizniche/Mycodo
cd Mycodo
git checkout mycodo_docker
sudo /bin/bash ./mycodo.sh install-dependenciesAdd your user to the docker group so you can run without root privileges (replacing USER with your user): Log out and back in for the changes to take affect, then build: Once it's built and running, you can navigate to https://127.0.0.1/ to access the web UI. |
|
I did this. On the stretch distro for a long run node. This is my CO2 and particle detector station. Got this (good to know because I use that on Pi.) 15.8 mb on my system. I'm a couple of beers into this so I'll cut and paste what Kyle wrote above. Hold my beer, please. I had added the user Pi to the docker group. I have some problem with my Docker account. Going away to fix that. -reboot- (because there are times that you gotta) Beauty. Getting another beer. Your move, Kyle. I'll leave this as it is and await better ideas to continue further. Almost is pretty good in my book. I hear that some other embedded applications are going to containers because they are also being built on linux and obscure isn't good enough for security anymore. People around you are already driving those applications. A Tesla isn't just a big battery, with wheels. |
|
Looks like gcc is needed. I just recently took it out of the install process, but hadn't built on the Pi to verify it was still working. I'll push a fix shortly. |
|
Sorry your first build experience was a flop. I pushed a fix for the issue building on Pis. I also updated my second comment with more detailed commands to shutdown any previously-installed Mycodo to free up ports and adding the user to the |
|
No worries Kyle, I still had a good time while trying. I'll follow your suggestions and try some more. Its worthwhile to do, I think. Thank you for giving containers a go with Mycodo! |
|
Full Moon coming. Guess it is time. Successfully built bbcfefa76198 And guess what I got. Magic. Gonna go explore now, however I have a few comments just for fun Collecting itsdangerous>=0.24 (from Flask==1.0.2->-r /home/mycodo/requirements.txt (line 4)) That guy stole my best bad idea for a library. The only misadventure was not adding 'sudo' to the 'make build' instruction. I'm starting to make a habit of writing instructions to include the 'sudo' and I bold the rest of the command-line. This makes it easy to cut and paste for lazy, root-less folks such as myself. I don't wanna be root. Being root warps the perceptions of usage, usually in unfortunate ways. I have a GUI. Nice work Kyle! I logged in and looked around a little so far. Will try some of the basic stuff and see what works. I have this big smile right now. That saying about how 'nothing good ever happens after midnight' came from someone who was doing it wrong. At least that is my perspective. |
|
Well the first thing I tried was a USB connected sensor - the MH-z19. I cannot change the port from hardware serial due to some access issue: Error: Modify Input: Invalid device or improper permissions to read device I tried changing perms on the device, that isn't the issue. Its a container thing. And don't I like a challenge. Containers are attractive to me because as you become more ambitious with Mycodo, the platform becomes the limiting factor, even with four cores. For straight forward PID tasks with a few sensors, the Pi is more than cost effective. As the sensors become more sophisticated or you are simply ambitious ( voice control, integration with other code ) containers could allow the kind of isolation that prevents total system failure when a feature doesn't work as expected. Like the code I write. I will have to come back to this later. So much to do. But it is a beginning and that is pretty exciting. The OS can take this to a number of higher performance platforms if ported with that in mind. There may be a way (I think there is) to push most of the work and storage to a cloud instance and have a lower end device like the Pi be the real-world proxy. Once you program a PID, the work is local and the rest can be more or less done elsewhere. This provides scale, if needed and still maintain a single pane of glass for complex configurations. Anything built in the cloud can also be done on a local server with virtualization, if you are allergic to clouds. |
|
Glad to see it's running! Here are a few notes before I begin my own investigation into getting external sensors working:
After you issued the command
😃
I haven't attempted to do this yet, so I'll see how I can fix this. I have In other news, today I'm implementing my recently-built outdoor LoRaWAN gateway (photo attached) on my university farm and sending sensor data from the surrounding greenhouses to The Things Network before being acquired by Mycodo (or multiple Mycodos). I got the BME280 working on my TTGO T-Beam dev. platform last night and just recently added support for downloading data from The Things Network into Mycodo via a TTN Input module. I'm going to be working on a way to make this Input module able to have a variable number if measurements selected (opposed to the current hard-coded ones for just the BME280), then each measurement can have its own unit/measurement configured, so there won't be any limitation to the number or type of measurements.
|
|
The gateway was successfully deployed on the farm and one node is currently measuring temperature, humidity, pressure, dew point, and vapor pressure deficit in one of the hydroponic greenhouses. Now to see if the system remains stable over a paltry cellular internet connection. I've already set up the Mycodo on the farm to download the data from TTN, just as I have at home, and it's working perfectly so far. Regarding docker, I found the missing configuration settings I needed to detect devices after the container has started. I just pushed an update and successfully tested the MH-Z16/UART over USB, connected after the container started! |
|
I added Telegraf and Grafana to the docker build to experiment a bit with them. They were easily integrated, and add a new set of system monitoring and dashboard capability for Mycodo data. I'm not sure if they'll stay, but for the time being, they're included in the Docker build. I also fixed several issues with the dependency install script, moved everything to using Python 3, and successfully tested on a fresh install of Raspbian for everything to work. |
|
After much toiling, I have the latest version of Mycodo (8.1.1) running in Docker containers, without sacrificing its ability to be normally installed on a Raspberry Pi. I'll push my edits to master soon, after I tweak some of the volumes and directory structures. Here's it running on Ubuntu 18.04 (64-bit):
|
|
The mycodo_docker branch is now defunct (and will be removed soon), as I've been successful at integrating the docker code to master without sacrificing any normal functionality. The Docker configuration/build files can be found under the There is a lot of functionality that doesn't work, but I'll be working on this gradually. This is a good step to bringing Mycodo available to other platforms. Containers running on a full buildGrafana/Telegraf (only part of full stats available)
|
|
Thanks Kyle! I'm going to spend some time looking at how this might work on a Mac with dockerdesktop. I'm just looking for serial port inputs right now and maybe a "sensor pod" on a FTDI-4232 with python code input to Mycodo. I can do the initial work on the standard release on a pi and then try it on a laptop with containers. USB-C changes what you can attach to a laptop (for instrumentation) and power has always been my biggest challenge. Hang enough sensors on a Pi and the power gets flaky with most wall-wart supplies. And some sensors I want to use ( AS7265x spectral sensor ) just crush the CPU on a pi with the matlab code to generate the spectral chart. |
|
I'm just beginning my journey towards Dockerized mycodo. I'm very keen to port to balenaOS for my RPi's but I'm shooting towards I'm specifically having trouble building Why does mycodo manage it's own binaries? Vs venv or the like? Anyways I was able to get mycodo_flask to build by commenting some of the following, but then get runtime failure. |
|
@nargetdev You should use the code from the master branch, not the latest release. You're currently using old docker code. Try the master branch and it should work. |
|
Ah.. forgot to |
|
Yes it worked this time. Nice work Kyle. I'll stay posted regarding balena. Should be relatively straight forward. |
|
For what it's worth I managed to get the following fork to successfully upload the docker images to the RPi via https://github.com/nargetdev/balena-mycodo More to come as I dig in further.. |
|
I'm getting errors with the host networking in the compose context. (because compose renames network hostnames) the below example is illustrative: It should be found at: I'm wondering as you'd said previously @kizniche you don't want to break compatibility with the "normal" build. Without making a ton of defines/exceptions for every single mention of localhost/127.0.0.1 towards microservices names in docker-compose context what do you think is the path of least resistance here? |
|
Fortunately just a few weeks ago I added the ability to configure influxdb settings in the Mycodo database. This includes the ability to set the hostname to connect to influxdb. I haven't tested it, but I suspect if you change it from "localhost" to "mycodo_influxdb", it should work. You'll need to install from the master branch, because this hasn't been released yet. |
|
The docker influxdb install script run.sh was only designed for influxdb 1.x, and since 1.x and 2.x have incompatible commands, the commands used to create the db/user/password likely don't work with 2.x. |
|
See here for the difference between the two: Mycodo/mycodo/scripts/upgrade_commands.sh Lines 428 to 465 in eb1b7c6 |
|
Yes I just found the install commands that are invoked on the upstream (non-docker) I tested this out (outside of docker), but i still somehow end up with an influxdb1.8 install Anyways, no worries. In due time I may take a look at porting influxdb2.x to the docker side, but ver 1.x is fine for now. -- UPDATE, dirty system.. clean install from .. for anyone onlooking if you need to get around the SSL issues before setting up certs you can do the following |
I'm not familiar with this. Can you give give a breakdown of how this works or how to do it?
Why is this? Is the only solution to put the docker root directory (or just docker-compose.yaml) at the root of the Mycodo repository? I guess this wouldn't be too much of an issue if all the other directories were placed inside a docker/ directory. The directory structure of nargetdev/balena-mycodo currently has all these docker directories in the root of Mycodo. Could these be moved to docker/ and still work? If so, I'd be more open to merging the pull request since it keeps the docker and non-docker portions of Mycodo more segregated. Docker is still an experimental feature that needs some improvement, so is still a secondary feature at this point. |
|
Hey there. So for the time being I've actually graduated to a lightweight Kubernetes (k3s) based deployment to manage my RPis and I'm finding it to be delicious, albeit with a learning curve (k8s manifests are much more verbose than docker-compose.yml) k8s is not opinionated about a specific file structure in the way that BalenaOS mandates - so it's definitely possible now without mutilating the structure / mucking up bare metal compatibility. My workflow for my own services has generally involved a build command like the following: ... in order to build cross platform image for RPi from my Macbook (very zippy build) My motivation for switching to k3s from balena initially was their 10 device limit, but since then I've realized quite a few advantages that come packaged with k8s ecosystem in terms of observability, and extensive tooling of the mainstream de-facto standard for container management. P.S. regarding BalenaOS / BalenaCloud - best way to grok I think is to just make an account at https://www.balena.io/ and flash some RPis with their OS and push some containers to the runtime (they have tons of tutorials). It's a very nice docker distro with bells and whistles (like a filesystem watcher that automatically rebuilds and pushes images to your devices as you edit source), but they do have a 10 device limit for their cloud PaaS - at which point you could just run your own open source instance, but I haven't tried, as I'm all in on k3s for now. |
|
Ah.. one gotcha with k3s - you can choose to do this either the k3s way or the baremetal way at boot. For k3s on the pi the |
|
@kizniche do you have a map for which volume mounts are populated on initial creation by the file system of the container? I didn't realize how Docker handles this until just now (I.e. initialized volume will clobber Docker file system, vs empty volume will assume Docker file system). I haven't looked at this closely yet, but I noticed it when containers were crashing because /home/mycodo/env/python was getting clobbered. The python executable for example should be in a RO location. What is needing to be persisted across containers and for what reasons? I'd like to limit or eliminate the possibility of clobber / runtime inconsistency due to initialized / uninitialized volumes. Cheers. To containers. |
|
Are you referring to these? Mycodo/docker/docker-compose.yml Lines 38 to 48 in 9b42db8 |
|
@nargetdev could you please provide some more information about building/running on kubernetes? I'd like to explore this option more, but my experience is virtually all with Docker, so I could use some help getting this going and want to avoid fumbling around and hitting common issues that can be avoided with some guidance. Thanks. |
|
Hi, I am having a few issues running Mycodo on my Raspberry Pi Zero via docker. I have run the following commands successfully: However, when I then ran Which I fixed by repacing I now have another error that I am not able to resolve: I checked and the file in instead. I have of coursed tried correcting this, running I will try the standard installation method instead now, which should be sufficient for my purposes. I just thought I would let you know in case I am doing anything wrong here. Looking forward to working with Mycodo! Cheers, Jonny |
|
I've actually been working on Docker all day today. |
|
Brilliant, thanks for the speedy response! I retried the steps outlined above on the docker-01 branch, and the orginial issue is solved. However I now have a new error message: I had a quick look, but tbh I don't understand where the issue lies. |
You can detect which chip architecture a linux machine has with the On the pi 4 you should get I believe. |
|
I just committed (and merged into master) my final changes for Docker. I haven't tested on a Pi Zero.
Is this all there was in the log? No other errors? |
|
OK, I will try again then with the master branch. I ran the docker-compose up command with the --verbose option and here are the last few lines of code before the error appears: |
|
This command starts with printf, so for that to not be in the logs indicates an unusual issue. Mycodo/mycodo/scripts/upgrade_commands.sh Lines 711 to 715 in 2c20036 |
|
I just tested with the main branch again, but still no dice:
Yes that's what I thought. I'll take a closer look when I have more time, will probably be this weekend though. I'll update you if I find anything. |
|
I just did a full test on a freshly installed Ubuntu 20.04 virtual machine. I changed the Linux image used and updated the build commands. I was able to build and run without issue. It also appears "docker compose" changed how output is displayed in the terminal from "docker-compose", which is much less verbose. I'm looking to see if it's possible to get the old behavior back. --verbose doesn't seem to have any effect, which I believe is a bug. |
|
Hi @kizniche, sorry for the late reply. I still couldn't get this to work, and I have decided to write my own code for my needs. However I think the crux of the issue was that my raspberry pi zero w (1st generation) supported 32-bit OS only, and Docker does not support 32 bit OS (I had to install Docker using unofficial 3rd party software: https://gitlab.com/docker-32bit). This caused all kinds of issues trying to fix it, so I think you should maybe add in the README that 32-bit raspberry pis are not supported for the docker installation of Mycodo. |
|
architecture heterogeneity can be addressed with |
@kizniche right now my workflow is using Here's my kompose output directory (nothing special .. just vanilla output from On the one hand Insofar as this concerns us we need to some method to bootstrap the volumes on first launch. P.S. I'm going to make a separate issue but i'm just starting to look into the upgrade/restore function in container context. |
|
Actually i'll consolidate here regarding upgrade backup/restore since it's in the checklist above.. The first thing I'm facing down right off the batt is that the Docker containers are all set up with user Many of the scripts are hard coded with To illustrate: on baremetal in container Is this a mistake or is there a reason for the inconsistency? === Secondly, perhaps more fundamentally, baremetal instance plays with
From a 20,000' view the "docker" way of handling an upgrade/restore would be to rip down the container, rehydrate the stateful filesystem via external process (the volumes), and then boot the containers. This could be orchestrated nicely with k8s control plane. Let me know your thoughts on the two avenues / sanity check my logic @kizniche |
|
Great success! Mycodo on k8s.. |
I found it easier to work with, but there's no reason other than that. The symlink mycodo-root is what I've been using to identify the root wherever it may be, since different users could be installing it in their home directory. Really, it should be installed in a more defined directory like /opt or /usr, for example, but this is difficult to change after having it at a different location for so many users. I haven't been able to figure out a reliable way to perform an upgrade in docker. I agree this needs to be done external of the containers, with some sort of script. Discourse is one such project I use that utilizes docker and an external upgrade system with scripts, so perhaps we can look to that for inspiration. |
|
Off the bat there are no systemd services to stop. These are rather the containers themselves, which should be ephemeral (able to be destroyed without notice) An update could Docker "commit" the changes (I.e. dependencies installed) and use the newly committed image, but this seems messy.. Finally a "fat" container could be built with all possible dependencies preinstalled. What else is stateful outside of the running version of the sources? mycodo.db .. what else? If you could help me just enumerate a bullet point list of stateful components so I can start my study? Thank you kindly |
|
I've run into an issue where libcamera-apps dependency won't install But following that:
|
|
This is because Raspberry Pi OS is not the operating system installed in the Mycodo container. For that package, you need to use a Raspberry Pi OS variant to have access to their apt package source. You could try adding the package source to the current OS, but YMMV. Are you running Mycodo in Docker on actual Pi hardware? If so, why use Docker instead of installing on the actual hardware? If you look in the Docker file, you'll see "debian:stable-slim" is used. You will need to duplicate the Function and create a new one that uses a more generic package for libcamera than the Raspberry Pi OS-specific libcamera-apps, if you want to use a libcamera Function on an OS other than Raspberry Pi OS. |
|
Thanks, will give that a go. Yes running in Docker on a Pi, just to test it out. Interested in running it under https://github.com/balena-os The reason being that I could then swap the underlying hardware for any of their supported devices https://www.balena.io/os#download-os including this one https://www.compulab.com/products/iot-gateways/iot-gate-imx8plus-industrial-arm-iot-gateway/ which I like the look of. |
|
Just adding some notes in case this is useful to someone else later (including future-me) Using the https://www.portainer.io/ webui I accessed the terminal for the mycodo_flask container. apt-install nano nano /etc/apt/sources.list Did a wget http://archive.raspberrypi.org/debian/raspberrypi.gpg.key I haven't yet successfully captured an image with it. apt install sudo apt-get install v4l-utils root@Mycodo:/home/mycodo/mycodo# v4l2-ctl --list-devices bcm2835-isp (platform:bcm2835-isp): unicam (platform:fe801000.csi): rpivid (platform:rpivid): HiCamera: UVC Camera (usb-0000:01:00.0-1.3): oot@Mycodo:/home/mycodo/mycodo# ls /dev/video* I can apt install raspi-config But there is no camera option root@Mycodo:/home/mycodo/mycodo# v4l2-ctl --all This looks potentially useful https://www.losant.com/blog/how-to-access-the-raspberry-pi-camera-in-docker |
This is my experiment with using docker and docker-compose to run Mycodo. Thus far I successfully have Mycodo running fully in docker containers. This includes the Flask/Gunicorn frontend, Nginx, Influxdb, and the Mycodo daemon (additionally with Grafana and Telegraf for further data acquisition/presentation). This issue can serve as a discussion thread and a way I can keep users updated on the progress and where this is going. For current information about setup/building, see the docker/README.md.
Current state (check means working):
More will be added as they are found
The text was updated successfully, but these errors were encountered: