-
Notifications
You must be signed in to change notification settings - Fork 912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Docker Compose Support for Development Environment #2409
Changes from all commits
c514fd6
2eb8e76
60a18ce
544daa0
b5bf4e3
db3cbf9
4b4217b
d5fe598
6624510
5a431a5
255ce87
395300d
6bcd214
f07314c
b635cf6
c17cb84
b71a6fd
188fc8f
1ab5e3b
c936063
bc98021
810a941
1e5cc59
8baebab
60f8fb9
c66041c
2da3ce6
033c3b2
8c2db90
e19e58f
4fac47a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
docker-db-data | ||
log | ||
tmp | ||
docker-cache | ||
.travis.yml | ||
.git |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,36 @@ | ||||||
name: Docker | ||||||
on: | ||||||
- push | ||||||
- pull_request | ||||||
jobs: | ||||||
test: | ||||||
name: Docker | ||||||
runs-on: ubuntu-20.04 | ||||||
steps: | ||||||
- name: Checkout source | ||||||
uses: actions/checkout@v1 | ||||||
- name: Poke config | ||||||
run: | | ||||||
cp config/example.storage.yml config/storage.yml | ||||||
cp config/docker.database.yml config/database.yml | ||||||
touch config/settings.local.yml | ||||||
- name: Build Docker Image | ||||||
run: | | ||||||
docker-compose build | ||||||
- name: Start Docker-Compose | ||||||
run: | | ||||||
docker-compose up -d | ||||||
sleep 15 # let the DB warm up a little | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's a failure waiting to happen... Is there really not a way to actually detect when it's ready instead of guessing? |
||||||
- name: Prepare Database | ||||||
run: | | ||||||
docker-compose run --rm web rake db:migrate | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Since we are on a fairly new version of Rails, this is the same and looks a bit nicer and is more up to date. (I will comment only this one random instance, but there are a few of them) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tested locally and this seems to work well! I’m hesitant to apply this change since we seem to have the “rake” and “Rakefile” naming convention throughout the repo. Maybe we can change this globally in another PR? |
||||||
docker-compose run web bundle exec rake i18n:js:export | ||||||
docker-compose run --rm web osmosis --rx docker/null-island.osm.xml --wd host=db database=openstreetmap user=openstreetmap password=openstreetmap validateSchemaVersion=no | ||||||
- name: Test Basic Website | ||||||
run: | | ||||||
curl -siL http://127.0.0.1:3000 | egrep '^HTTP/1.1 200 OK' | ||||||
curl -siL http://127.0.0.1:3000 | grep 'OpenStreetMap is the free wiki world map' | ||||||
curl -siL http://127.0.0.1:3000/api/0.6/node/1 | grep 'Null Island' | ||||||
- name: Test Complete Suite | ||||||
run: | | ||||||
docker-compose run --rm web bundle exec rails test:db | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I always use |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,3 +18,6 @@ public/attachments | |
public/export | ||
storage | ||
tmp | ||
|
||
# docker-compose database directory | ||
docker-db-data |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
# Using Docker and Docker Compose for Development and Testing | ||
|
||
These instructions are designed for setting up The Rails Port for development and testing using [Docker](https://www.docker.com/). This will allow you to install the OpenStreetMap application and all its dependencies in Docker images and then run them in containers, almost with a single command. You will need to install Docker and Docker Compose on your development machine: | ||
|
||
- [Install Docker](https://docs.docker.com/install/) | ||
- [Install Docker Compose](https://docs.docker.com/compose/install/) | ||
|
||
The first step is to fork/clone the repo to your local machine: | ||
|
||
git clone https://github.com/openstreetmap/openstreetmap-website.git | ||
|
||
Now change working directory to the `openstreetmap-website`: | ||
|
||
cd openstreetmap-website | ||
|
||
## Initial Setup | ||
|
||
### Storage | ||
|
||
cp config/example.storage.yml config/storage.yml | ||
|
||
### Database | ||
|
||
cp config/docker.database.yml config/database.yml | ||
|
||
## Prepare local settings file | ||
|
||
This is a workaround. [See issues/2185 for details](https://github.com/openstreetmap/openstreetmap-website/issues/2185#issuecomment-508676026). | ||
|
||
touch config/settings.local.yml | ||
|
||
## Installation | ||
|
||
To build local Docker images run from the root directory of the repository: | ||
|
||
docker-compose build | ||
|
||
If this is your first time running or you have removed cache this will take some time to complete. Once the Docker images have finished building you can launch the images as containers. | ||
|
||
To launch the app run: | ||
|
||
docker-compose up -d | ||
|
||
This will launch one Docker container for each 'service' specified in `docker-compose.yml` and run them in the background. There are two options for inspecting the logs of these running containers: | ||
|
||
- You can tail logs of a running container with a command like this: `docker-compose logs -f web` or `docker-compose logs -f db`. | ||
- Instead of running the containers in the background with the `-d` flag, you can launch the containers in the foreground with `docker-compose up`. The downside of this is that the logs of all the 'services' defined in `docker-compose.yml` will be intermingled. If you don't want this you can mix and match - for example, you can run the database in background with `docker-compose up -d db` and then run the Rails app in the foreground via `docker-compose up web`. | ||
|
||
### Migrations | ||
|
||
Run the Rails database migrations: | ||
|
||
docker-compose run --rm web bundle exec rake db:migrate | ||
|
||
### Tests | ||
|
||
Run the test suite by running: | ||
|
||
docker-compose run --rm web bundle exec rake test:db | ||
|
||
### Loading an OSM extract | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since loading an extract via osmosis stopped working after PR #2432 I'm inclined to remove this section until I can offer a working solution. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There’s a work around available: #2543 (comment) Just add it to the description. You would need to call psql in any case to correct the various Postgres sequences, so the additional pain to add and drop a column should be close to zero. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It’s a super-ugly workaround, almost to the point that I think it’d be harmful just to recommend that people monkey with the schema pre- and post-import. Let’s remove the section for now? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It’s as ugly as having to tweak sequences. My point was, it was already ugly before, we’re not adding much to that by this workaround. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Osmosis 0.47.1 now correctly writes to the DB so this section should work fine. |
||
|
||
This installation comes with no geographic data loaded. You can either create new data using one of the editors (Potlatch 2, iD, JOSM etc) or by loading an OSM extract. Here an example for loading an OSM extract into your Docker-based OSM instance. | ||
|
||
For example, let's download the District of Columbia from Geofabrik or [any other region](https://download.geofabrik.de): | ||
|
||
wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf | ||
|
||
You can now use Docker to load this extract into your local Docker-based OSM instance: | ||
|
||
docker-compose run --rm web osmosis \ | ||
-verbose \ | ||
--read-pbf district-of-columbia-latest.osm.pbf \ | ||
--write-apidb \ | ||
host="db" \ | ||
database="openstreetmap" \ | ||
user="openstreetmap" \ | ||
validateSchemaVersion="no" | ||
|
||
Once you have data loaded for Washington, DC you should be able to navigate to [`http://localhost:3000/#map=12/38.8938/-77.0146`](http://localhost:3000/#map=12/38.8938/-77.0146) to begin working with your local instance. | ||
|
||
### Additional Configuration | ||
|
||
See [`CONFIGURE.md`](CONFIGURE.md) for information on how to manage users and enable OAuth for iD, JOSM etc. | ||
|
||
### Bash | ||
|
||
If you want to get into a web container and run specific commands you can fire up a throwaway container to run bash in via: | ||
|
||
docker-compose run --rm web bash | ||
|
||
Alternatively, if you want to use the already-running `web` container then you can `exec` into it via: | ||
|
||
docker-compose exec web bash | ||
|
||
Similarly, if you want to `exec` in the db container use: | ||
|
||
docker-compose exec db bash |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
FROM ubuntu:20.04 | ||
|
||
ENV DEBIAN_FRONTEND=noninteractive | ||
|
||
# Install system packages | ||
RUN apt-get update \ | ||
&& apt-get install --no-install-recommends -y \ | ||
build-essential \ | ||
curl \ | ||
default-jre-headless \ | ||
file \ | ||
firefox-geckodriver \ | ||
imagemagick \ | ||
libarchive-dev \ | ||
libffi-dev \ | ||
libgd-dev \ | ||
libmagickwand-dev \ | ||
libpq-dev \ | ||
libsasl2-dev \ | ||
libxml2-dev \ | ||
libxslt1-dev \ | ||
locales \ | ||
nodejs \ | ||
postgresql-client \ | ||
ruby2.7 \ | ||
ruby2.7-dev \ | ||
tzdata \ | ||
unzip \ | ||
yarnpkg \ | ||
&& apt-get clean \ | ||
&& rm -rf /var/lib/apt/lists/* | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is this removing part of apt's internal state? |
||
|
||
# Install current Osmosis | ||
RUN curl -OL https://github.com/openstreetmap/osmosis/releases/download/0.47.2/osmosis-0.47.2.tgz \ | ||
&& tar -C /usr/local -xzf osmosis-0.47.2.tgz | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hard coding an omsosmis version number is a bad idea as it will need updating when osmosis changes. Why are we wasting time downloading osmosis anyway? How many people will use it? It's not like it even works very well for loading an API database which I assume is what you have it here for... |
||
|
||
ENV DEBIAN_FRONTEND=dialog | ||
|
||
# Setup app location | ||
RUN mkdir -p /app | ||
WORKDIR /app | ||
|
||
# Install Ruby packages | ||
ADD Gemfile Gemfile.lock /app/ | ||
RUN gem install bundler \ | ||
&& bundle install | ||
|
||
# Install NodeJS packages | ||
ADD package.json yarn.lock /app/ | ||
RUN yarnpkg install | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should be done as |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__) | ||
|
||
require "bundler/setup" # Set up gems listed in the Gemfile. | ||
require "bootsnap/setup" # Speed up boot time by caching expensive operations. | ||
require "bootsnap/setup" if ENV.fetch("ENABLE_BOOTSNAP", "true") == "true" # Speed up boot time by caching expensive operations. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm sorry but there has to be a better solution to the bootsnap problem that polluting core rails setup files with this nonsense. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# This configuration is tailored for use with docker-compose. See DOCKER.md for more information. | ||
|
||
development: | ||
adapter: postgresql | ||
database: openstreetmap | ||
username: openstreetmap | ||
password: openstreetmap | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you're using trust auth (which you appear to be) then you shouldn't need a password. |
||
host: db | ||
encoding: utf8 | ||
|
||
# Warning: The database defined as 'test' will be erased and | ||
# re-generated from your development database when you run 'rake'. | ||
# Do not set this db to the same as development or production. | ||
test: | ||
adapter: postgresql | ||
database: osm_test | ||
username: openstreetmap | ||
password: openstreetmap | ||
host: db | ||
encoding: utf8 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
version: "3" | ||
|
||
services: | ||
web: | ||
build: | ||
context: . | ||
volumes: | ||
- .:/app | ||
# Prevent these directories from mounting so they're not shared between host OS and Docker | ||
- /app/node_modules/ | ||
- /app/tmp/ | ||
# Mount these upload directories so they persist between runs | ||
- web-traces:/home/osm/traces | ||
- web-images:/home/osm/images | ||
ports: | ||
- "3000:3000" | ||
environment: | ||
# https://github.com/Shopify/bootsnap/issues/262 | ||
ENABLE_BOOTSNAP: 'false' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see where ENABLE_BOOTSNAP would be used right now. Did you maybe forget to check in your changes to config/boot.rb here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is me chasing down some Ruby 2.5/2.7 vs. Ubuntu 18/20 weirdness when migrating away from the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some details on why Bootsnap. Without these references in
With these lines included, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Clarification: the error output above happens on my local machine (MacOS, Docker 2.3). Same process seems happy on GH Actions: https://github.com/migurski/openstreetmap-website/runs/1618711125 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, on Github, it's pretty much a no-op, as this environment variable isn't being used anywhere b/c config/boot.rb isn't part of this pull request. I also haven't seen this before on Ubuntu 20.04 host. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any advice? I don’t know what Bootsnap is or does, I’m just blindly searching error messages here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the Github issue you've linked to, they also replaced one line in a file named On my local box, there's one line to load the bootsnap/setup module, which is used to speed up Rails application loading:
They replace that line by making the loading process depend on your new environment variable ENABLE_BOOTSNAP:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That makes sense — I’ve added this check in |
||
command: bundle exec rails s -p 3000 -b '0.0.0.0' | ||
depends_on: | ||
- db | ||
|
||
db: | ||
build: | ||
context: . | ||
dockerfile: docker/postgres/Dockerfile | ||
ports: | ||
- "54321:5432" | ||
environment: | ||
POSTGRES_HOST_AUTH_METHOD: trust | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you're doing trust auth why did you put a password in the database configuration? |
||
POSTGRES_DB: openstreetmap | ||
volumes: | ||
# Mount the Postgres data directory so it persists between runs | ||
- db-data:/var/lib/postgresql/data | ||
|
||
volumes: | ||
web-traces: | ||
web-images: | ||
db-data: |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
<?xml version='1.0' encoding='UTF-8'?> | ||
<osm version="0.6" generator="Osmosis 0.47.2"> | ||
<bounds minlon="-180.00000" minlat="-90.00000" maxlon="180.00000" maxlat="90.00000" origin="Osmosis 0.47.2"/> | ||
<node id="1" version="1" timestamp="1970-01-01T00:00:00Z" uid="1" user="nobody" lat="0" lon="0"> | ||
<tag k="name" v="Null Island"/> | ||
</node> | ||
</osm> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
FROM postgres:11 | ||
|
||
# Add db init script to install OSM-specific Postgres functions/extensions. | ||
ADD docker/postgres/openstreetmap-postgres-init.sh /docker-entrypoint-initdb.d/ | ||
|
||
# Custom database functions are in a SQL file. | ||
ADD db/functions/functions.sql /usr/local/sbin/osm-db-functions.sql | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A couple of comments:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you have an alternate suggestion for a location? Tests fail when these functions are excluded. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you recall which test did in fact fail? My guess here would be the /changes endpoint stuff. Regarding the location, can't you use /app/db/functions/functions.sql as is? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here’s the failing test suite that was fixed by adding functions.sql: https://github.com/migurski/openstreetmap-website/runs/1619876640?check_suite_focus=true We don’t use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, yes, right, the script needs to go to another container. Assumption confirmed, it was the /changes endpoint. I really want to get rid of this code, as it's outdated and not used by anyone except for some single unknown user. (-> re-opened #2486). Once this is done, we don't need the sql script anymore. For the time being we can leave it as-is, it's not worth fiddling much with it.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's a bit odd that you need those migrations on an empty database inside a container. I mean, we're starting fresh with a modern state of the database sql scripts, and skip any ancient stuff. Would we ever even run those ancient migrations inside the Docker container? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To answer your second question, yes, you run those old migrations - the instructions have But the wider topic is whether we should keep those old migrations, whether we see the database defined by either a) the sum of all the migrations or b) the structure.sql file (opinions vary), and finally whether if we do drop old migrations should we have a 'fake' starter migration, or should we just keep a certain number of migrations and change all the documentation to use rake db:structure:load (or similar), and finally (finally) if we don't keep all the migrations, how do we ensure that the structure file matches the sum of the migrations that have been run production. So that's not a simple set of decisions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Somehow I assumed a new database would be set up using Maybe the mechanism works in a completely different way, but then I don't really know what's the purpose of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In any case, running the migration passes on a fresh install even without the db scripts, so I'm even more confused now why they would be needed. We don't really have any data to be backfilled on an empty database, don't we? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's discuss this on a new issue - otherwise this conversation will never be found! |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
#!/bin/bash | ||
set -ex | ||
|
||
# Create 'openstreetmap' user | ||
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" <<-EOSQL | ||
CREATE USER openstreetmap SUPERUSER PASSWORD 'openstreetmap'; | ||
GRANT ALL PRIVILEGES ON DATABASE openstreetmap TO openstreetmap; | ||
EOSQL | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is where the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As you appear to be using trust auth there shouldn't be any need to set a password and it would also be best not to set superuser unless have to - the only thing that is likely to be needed for installing the extension which os done as the postgres user anyway. |
||
|
||
# Create btree_gist extensions | ||
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE EXTENSION btree_gist" openstreetmap | ||
|
||
# Define custom functions | ||
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -f "/usr/local/sbin/osm-db-functions.sql" openstreetmap |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is out of date - there is a v2 now.