-
Notifications
You must be signed in to change notification settings - Fork 912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docker support #2272
Add docker support #2272
Conversation
added Dockerfile for the ruby on rails app inside a separate docker directory. this will build a image from the official ruby (specifically ruby:2.5-slim).
…rror postgresql-client throws a dpkg error when it can't create a softlink in for man page and aborts installation. found a solve at stack overflow and modified Dockerfile to add a comment to mention the problem with slove source.
#install packages | ||
#fixes dpkg man page softlink error while installing postgresql-client [source: https://github.com/debuerreotype/debuerreotype/issues/10#issuecomment-438342078] | ||
RUN mkdir -p /usr/share/man/man1 && mkdir -p /usr/share/man/man7 | ||
RUN apt-get update && apt-get install curl -y |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each time apt-get is run you should update and then remove the APT cache files at the end of the command.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So that the cache files don't make it into the docker image
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this just building a container for local use though? Not anything that's going to be published?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, but what I've read indicates it as a best practice in general
docker/Dockerfile
Outdated
RUN mkdir -p /usr/share/man/man1 && mkdir -p /usr/share/man/man7 | ||
RUN apt-get update && apt-get install curl -y | ||
RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && bash nodesource_setup.sh && rm nodesource_setup.sh | ||
RUN apt-get install -y --no-install-recommends ruby-dev libarchive-dev libmagickwand-dev libxml2-dev libxslt1-dev build-essential libpq-dev libsasl2-dev imagemagick libffi-dev locales postgresql-client-9.6 nodejs && apt-get clean && rm -rf /var/lib/apt/lists/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The rm -rf /var/lib/apt/lists/*
doesn't reclaim the space since the space is still used by earlier layers of the dockerfile.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all looks horribly wrong anyway - why is is creating directories by hand?
And downloading and installing node from upstream using a dodgy "run this shell script as root" install method isn't going to fly. Why not just use the ubuntu packaged one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all looks horribly wrong anyway - why is is creating directories by hand?
as I have mentioned on the comment above the command, while installing postgresql-client
, because update-alternatives
can't create symbolic link, dpkg
throws an error and exits with code 100. A quick search pointed to this stack answer which solves the problem.
And downloading and installing node from upstream using a dodgy "run this shell script as root" install method isn't going to fly. Why not just use the ubuntu packaged one?
the image ruby:2.5-slim
usages Debian 9 (stretch) as base image. Though I could install nodejs
using the office Debian repo , I couldn't install npm
using the apt-get
command as it gives E: Unable to locate package npm
error. I followed the official download instruction which leads to NodeSource's GitHub instruction and installed nodejs
and npm
.
If there is any better way to solve this, I would be happy to hear. 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there is any better way to solve this, I would be happy to hear. blush
I'm also not a Docker expert, but it might be worth considering starting from a different base image, like Ubuntu 18.04, rather than starting from a ruby base image. This would more closely match the installations instructions, and also the Vagrant build.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm also not a Docker expert, but it might be worth considering starting from a different base image, like Ubuntu 18.04, rather than starting from a ruby base image.
Neither am I. As far as I understand the purpose of using a ruby image it to remove the legwork required to install ruby on an Ubuntu base image. Maybe someone with better docker understanding can explain it better? ping @pnorman
I am building on top of the previous PR (#1290) and at first I wanted to see if I can build the web
and db
images using docker with the new code because the code base and the installation procedure has changed since that PR. If we all agree on the pros, we can change the base image to Ubuntu 18.04.
docker/Dockerfile
Outdated
RUN apt-get update && apt-get install curl -y | ||
RUN curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh && bash nodesource_setup.sh && rm nodesource_setup.sh | ||
RUN apt-get install -y --no-install-recommends ruby-dev libarchive-dev libmagickwand-dev libxml2-dev libxslt1-dev build-essential libpq-dev libsasl2-dev imagemagick libffi-dev locales postgresql-client-9.6 nodejs && apt-get clean && rm -rf /var/lib/apt/lists/* | ||
RUN npm install yarn -g |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docs state bundle exec rake yarn:install
. If there's a reason to differ, can you add a comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Different thing - you need to install yarn first (unfortunately it's not packaged in Ubuntu like it is in Fedora) and then bundle exec rake yarn:install
runs yarn to install the modules listed in package.json
.
It may be worthwhile taking a look at: https://github.com/developmentseed/osm-seed |
While I'm happy to add Docker support, I feel like we're ending up with a lot of duplication in installation instructions. We have the human-readable INSTALL.md instructions, the travis config, the Vagrant provisioning scripts and now a Dockerfile - all of these cover similar ground. Is there any way that we can simplify things, or at least reduce some of the duplication? It's becoming increasingly hard to make sure these are all in line when we make changes. |
…ed jobs from bundle install modified Dockerfile to separate apt-get cache cleaning command, added a comment explaining why and how npm is added and removed --jobs flag from bundle install command
added docker_postgres.sh for PostgreSQL extension and functions that installs PostgreSQL Btree-gist Extension, compiles libpgosm.so and installs maptile_for_point, tile_for_point and xid_to_int4 functions
added Dockerfile.postgres for the postgres database app. this will build a image from the official postgres (specifically postgres:9.4) image.
Please give me some time to review this over the weekend. I have a very good understanding of docker. |
So it would seem I have hit a wall. 😞 I can build the application and run database migrations and nodejs package installation with
However, when the docker files are moved to docker directory the I have changed the |
This is probably because the default settings for connecting to the database are to use a Unix domain socket, whereas I suspect the Docker config is set up to use networking between the containers. Unfortunately I can't provide any guidance here as to what is best from a Docker point of view! |
There's a few commandline things which are pretty important to having a useful development environment, such as:
I'd like to make sure all of these things are possible in the Dockerized development setup, since at the moment I only see mention of running the webserver. With Vagrant, for example, you can just If they are possible already (great!) then it would be worthwhile documenting how that works, since it's not obvious to me at least! |
ok I have some good news. I was able solve the deployment problem mentioned on my previous comment. I forgot to change the volume mount point when I moved the
@gravitystorm they all are possible but for some reason I have setback regarding the
I just went through the old PR (#1290) and found that @paulsturgess made a nice doc for docker usage. Will try to re-use/re-purpose that and push it asap. Cheers! |
The developmentseed repo I mentioned earlier in this issue uses TCP connections instead of unix sockets to connect to he postgresql database. Also, check out https://github.com/hadolint/hadolint to get some recommendations on your Docker file, based on best practices. |
come to think of it, wouldn't that UNIX socket vs Docker tpc connection issue be applicable for |
You should also configure an unprivileged user for the Docker images. |
hi @robbyoconnor. I don't know a lot about unprivileged user in docker but saw note to self: This medium articles seems to have a good explanation. Might come handy. |
@fazlerabbi37 here's the thing: Docker does have isolation HOWEVER as root -- you can escape it...in theory is it likely..probably not but I'm not willing to chance it. Always treat root in a docker container as equal to root on the host...you are root, you can escape namespaces. |
…ntomjs via npm modified Dockerfile to changes postgresql-client package from postgresql-client-10 to postgresql-client, and added phantomjs installation via npm
@@ -1,4 +1,4 @@ | |||
FROM postgres:9.4 | |||
FROM postgres:11 | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Production still uses 9.5 as of now.
Only because of the xid index issues though, which relate to how we do the diff feed. I use 11 for my development setup. |
web: | ||
image: openstreetmap-website:v1 | ||
build: | ||
context: .. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not put the Dockerfiles in this directory?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which directory? Right now all docker related files are in one directory named docker
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh wait, nevermind.
db: | ||
image: openstreetmap-db:v1 | ||
build: | ||
context: .. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here as above?
modified Dockerfile.postgres file to changes docker_postgres.sh location from db to docker
modified Dockerfile.postgres file to add app location setup and gem install via bundle
modified Dockerfile.postgres file to move database function directory from /db to /app/db
modified docker_postgres.sh to fix the location of libpgosm as it moved from /db/functions to /app/db/functions
modified Dockerfile to add osmosis package needed for populating database with osm extract
modified DOCKER.md to add osmosis instructions needed to use osmosis and populate database
modified Dockerfile.postgres to add ruby compilation from source as we need ruby=>2.4 and also removed bundle installation via gem install command because bundle is already compiled installed while ruby source is being compiled and installed.
I am really sorry for not being able to push any update since July but I have good news to share. Docker support is now ready for review. 🎉 /cc: @mmd-osm @robbyoconnor have a look. can't re-request review for you guys. |
Hi @fazlerabbi37 , I'm a newbie in osm, I have tried you docker-compose file and I got the error ActiveRecord::PendingMigrationError ( Migrations are pending. To resolve this issue, run:
): Then, I have ran docker exec -it openstreetmap-website /bin/bash -c "bin/rails db:migrate RAILS_ENV=development" But, it's still error. |
@goonux did you followed this instructions to build it? If not please give it a try. In the meantime I will try to do a fresh build with latest |
I'm sorry, I didn't read it yet, I will try now. Thanks. |
TODO: create dedicated user & group to run the server, don't just use root. |
@mmd-osm I am making very slow progress on this due to SotM Asia 2019. meanwhile, it would be great if you can point me towards a few resource on this topic. |
I took a run at improving on this PR by adding my own knowledge of Docker best practices along with some of the recommendations in this PR. I created PR #2406 which allows you to very quickly have the Rails app up and running using docker-compose. Please have a look - thanks. |
Closing in favour of #2409 , feel free to reopen if necessary. |
Docker is one of the most popular container platform that makes it easier to build and deploy application. This pull request will add docker support for development setup of openstreetmap-website.
This PR is build on top of an old abandoned PR (#1290) and follows the Installation guide provided with this project.
Please keep in mind that I am just starting out with Docker and feel free to suggest and discuss anything that comes to mind. I will try my best to answer them. 😄
Todo:
rake test:db
error