Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable the app to be built using Docker #1290

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*
!Gemfile
!Gemfile.lock
!db/functions
!db/docker_postgres.sh
!lib/quad_tile
67 changes: 67 additions & 0 deletions DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Using Docker to run OpenStreetMap

Using [Docker](https://www.docker.com/) will allow you to install the OpenStreetMap application and all its' dependencies in a container, almost with a single command.

These instructions gloss over the precise details of the dependencies and their configuration but these can be found in full detail at [INSTALL.md](INSTALL.md).

The first step is to fork/clone the repo to your local machine. Then run these commands:

### App configuration

```
cp config/example.application.yml config/application.yml
```

### Database

```
cp config/example.database.yml config/database.yml
```

Set `username` to postgres and `host` to db leave the password blank
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be automated, since we always know what the DB info is when running in docker?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guides I've read keep this kind of configuration separate. I would suggest leaving it out for now and it could be reviewed later.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the configuration variables can be set using the environment: OSM_...

Defaults could be set in the Dockerfile and overwritten with -e or --env-file or using equivalent docker-compose.yml configuration.


### Installation

In the root directory run:

```
docker-compose up
```

### Migrations

```
docker-compose exec web bundle exec rake db:migrate
```

Once these are complete you should be able to visit the app at http://localhost:3000

If localhost does not work, you can use the IP address of the docker-machine.

### Tests

```
docker-compose exec web bundle exec rake test:db
```

### Bash

If you want to get onto the web container and run specific commands you can fire up bash via:

```
docker-compose exec web /bin/bash
```

Similarly, if you want to get onto the db container use:

```
docker-compose exec db /bin/bash
```

### General Information

The [docker-compose.yml](docker-compose.yml) specifies the configuration for the two web and db containers. For example port that the Postgres database is exposed on that you can point your local db admin tool at.

Note that the [Dockerfile.postgres](Dockerfile.postgres) for the db container includes various build tools to run [db/docker_postgres.sh](db/docker_postgres.sh). This script installs extensions and functions required by the database and it is run automatically because it is specifically added to the location `docker-entrypoint-initdb.d` on the container.

There is a [.dockerignore](.dockerignore) that ignores all files except those required to be added to the containers.
35 changes: 35 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
FROM ruby:2.3-slim
MAINTAINER OpenStreetMap
ENV REFRESHED_AT 2016-09-15

# Install packages
RUN apt-get update
RUN apt-get install -y --no-install-recommends build-essential
RUN apt-get install -y --no-install-recommends ruby-dev
RUN apt-get install -y --no-install-recommends libxml2-dev
RUN apt-get install -y --no-install-recommends libxslt1-dev
RUN apt-get install -y --no-install-recommends libpq-dev
RUN apt-get install -y --no-install-recommends libsasl2-dev
RUN apt-get install -y --no-install-recommends imagemagick
RUN apt-get install -y --no-install-recommends libmagickwand-dev
RUN apt-get install -y --no-install-recommends nodejs
RUN apt-get install -y --no-install-recommends file
RUN apt-get install -y --no-install-recommends postgresql-client
RUN apt-get install -y --no-install-recommends locales
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standard practice is to have all the apt-get stuff on one line


# Setup app location
RUN mkdir -p /app
WORKDIR /app
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also consider creating/running as an unprivileged user.


# Install gems
ADD Gemfile /app/Gemfile
ADD Gemfile.lock /app/Gemfile.lock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're adding Gemfile and Gemfile.lock, but shouldn't they come over when you bring all of the files from the local directory into docker?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are required so that bundler can install all the gem dependencies required for the app. They have to be manually added so that they are available during the image build process.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added benefit is that gems will only be reinstalled when Gemfile* changes, rather than when anything else in the source tree does.

RUN bundle install
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other images I've seen a CMD set so you get a meaningful console when you access it, e.g. CMD ["python2"]. Is it possible to do the same here to get a console? bundle exec rails console is the command used in CONFIGURE.md

Copy link
Author

@paulsturgess paulsturgess Sep 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CMD is actually declared in the docker-compose.yml as:
./script/rails s -p 3000 -b '0.0.0.0'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That script/rails is kind of ancient history - it should really have been deleted long since I think. A newly created rails app wouldn't have it.

I think something like bundle exec rails s -p 3000 -b '0.0.0.0' is more like what you should be doing these days.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including puma or similar would be nice if this image were also going to be used for "production" purposes (e.g. POSM, where we currently use puma).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Commenting here because it's the end of the file.)

COPY . /app should be included if this image is intended to be used independently of a local git clone, otherwise none of the source files will be available in the image.

(This can co-exist happily with mapping $(pwd) to /app as a volume; files present in the image will be shadowed by local paths.)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also consider adding -j $(nproc) as an argument to speed up gem installation (it will run a number of processes matching the cores available on the system doing the building).


RUN sed -i -e 's/# en_GB.UTF-8 UTF-8/en_GB.UTF-8 UTF-8/' /etc/locale.gen && \
echo 'LANG="en_GB.UTF-8"'>/etc/default/locale && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_GB.UTF-8

ENV LANG en_GB.UTF-8
Copy link

@mojodna mojodna Oct 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the sed command be bypassed by setting this before packages are installed? Reconfiguring locales isn't fast, so doing that earlier would make it quicker to iterate when using this for development purposes.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(For those new to Docker, each Docker "command" creates a filesystem layer. Filesystem layers are (typically) cached, allowing changes to later parts of the process to skip installation steps that were already taken care of (that were also defined earlier in the Dockerfile).)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Changes to files COPY'd or ADDed into an image invalidate those layers (and subsequent ones), similar to make.)

11 changes: 11 additions & 0 deletions Dockerfile.postgres
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
FROM postgres:9.4

ADD db/docker_postgres.sh docker-entrypoint-initdb.d/docker_postgres.sh
ADD db/functions/ db/functions/
ADD lib/quad_tile/ lib/quad_tile/

RUN apt-get update
RUN apt-get install -y make \
postgresql-server-dev-all \
build-essential \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
4 changes: 3 additions & 1 deletion INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
These instructions are designed for setting up The Rails Port for development and testing.
If you want to deploy the software for your own project, then see the notes at the end.

You can install the software directly on your machine, which is the traditional and probably best-supported approach. However, there is an alternative which may be easier: Vagrant. This installs the software into a virtual machine, which makes it easier to get a consistent development environment and may avoid installation difficulties. For Vagrant instructions, see [VAGRANT.md](VAGRANT.md).
You can install the software directly on your machine following the instructions below, which is the traditional and probably best-supported approach.

Alternatively there are guides to use either [Vagrant](VAGRANT.md) or [Docker](DOCKER.md).

These instructions are based on Ubuntu 12.04 LTS, which is the platform used by the OSMF servers.
The instructions also work, with only minor amendments, for all other current Ubuntu releases, Fedora and MacOSX
Expand Down
7 changes: 7 additions & 0 deletions db/docker_postgres.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE EXTENSION btree_gist" openstreetmap
make -C db/functions libpgosm.so
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 AS '/db/functions/libpgosm', 'maptile_for_point' LANGUAGE C STRICT" openstreetmap
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 AS '/db/functions/libpgosm', 'tile_for_point' LANGUAGE C STRICT" openstreetmap
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE FUNCTION xid_to_int4(xid) RETURNS int4 AS '/db/functions/libpgosm', 'xid_to_int4' LANGUAGE C STRICT" openstreetmap
23 changes: 23 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: '2'
services:
web:
image: openstreetmap-website:v1
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/app
ports:
- "3000:3000"
command: bundle exec rails s -p 3000 -b '0.0.0.0'
depends_on:
- db
db:
image: openstreetmap-db:v1
build:
context: .
dockerfile: Dockerfile.postgres
ports:
- "5432:5432"
environment:
POSTGRES_DB: openstreetmap