Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker: implement multi-stage-builds for ubuntu #3170

Merged
merged 15 commits into from
Oct 13, 2023
Merged

docker: implement multi-stage-builds for ubuntu #3170

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
bacaaf2
test grass.script.setup
nvesbl Sep 21, 2023
91baf3f
test script for docker images
nvesbl Sep 21, 2023
265d757
black
nvesbl Sep 21, 2023
4464096
add test instructions
nvesbl Sep 21, 2023
3fb56ef
multi-stage-build similar to alpine
nvesbl Sep 21, 2023
991ec3c
Apply suggestions from code review
ninsbl Sep 21, 2023
68a8a4d
add test for GDAL-plugin
nvesbl Sep 22, 2023
cb8363b
copy GDAL-plugin and set path
nvesbl Sep 22, 2023
8d9630f
Update docker/ubuntu/Dockerfile
ninsbl Sep 22, 2023
4203822
ignore dist.* in docker builds
nvesbl Sep 22, 2023
4d4da07
address review comments
nvesbl Sep 26, 2023
1822d00
separate datum grid (proj >= 7) stage
nvesbl Sep 27, 2023
587012c
use network grids
nvesbl Sep 27, 2023
60e930c
GUI build argument
nvesbl Sep 28, 2023
58e4042
GUI build argument
nvesbl Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 39 additions & 48 deletions docker/ubuntu/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# syntax=docker/dockerfile:1.4
FROM ubuntu:22.04 as common

LABEL authors="Carmen Tawalika,Markus Neteler,Anika Weinmann,Stefan Blumentrath"
Expand Down Expand Up @@ -28,20 +29,22 @@ ENV GRASS_RUN_PACKAGES="\
libcairo2 \
libcurl4-gnutls-dev \
libfftw3-bin \
libfftw3-dev \
libfreetype6 \
libgdal-dev \
libgsl27 \
libjpeg-turbo8 \
libjsoncpp-dev \
libncurses5 \
#libopenblas-dev \
libopenblas-dev \
libopenblas-base \
libopenjp2-7 \
libomp5 \
libomp-dev \
libgeos-dev \
#libpdal-dev \
#libproj-dev \
#libpq-dev \
libpdal-dev \
libproj-dev \
libpq-dev \
libgsl-dev \
libpdal-base13 \
libpdal-plugin-hdf \
Expand Down Expand Up @@ -90,6 +93,22 @@ RUN apt-get update \
RUN echo LANG="en_US.UTF-8" > /etc/default/locale
RUN echo en_US.UTF-8 UTF-8 >> /etc/locale.gen && locale-gen

## fetch vertical datums for PDAL and store into PROJ dir
WORKDIR /src
RUN mkdir vdatum && \
cd vdatum && \
wget -q http://download.osgeo.org/proj/vdatum/usa_geoid2012.zip && unzip -j -u usa_geoid2012.zip -d /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/usa_geoid2009.zip && unzip -j -u usa_geoid2009.zip -d /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/usa_geoid2003.zip && unzip -j -u usa_geoid2003.zip -d /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/usa_geoid1999.zip && unzip -j -u usa_geoid1999.zip -d /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/vertcon/vertconc.gtx && mv vertconc.gtx /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/vertcon/vertcone.gtx && mv vertcone.gtx /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/vertcon/vertconw.gtx && mv vertconw.gtx /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/egm96_15/egm96_15.gtx && mv egm96_15.gtx /usr/share/proj; \
wget -q http://download.osgeo.org/proj/vdatum/egm08_25/egm08_25.gtx && mv egm08_25.gtx /usr/share/proj; \
cd .. && \
rm -rf vdatum
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these vdatum downloads aren't needed for the compilation of grass, these can be a really good candidate to be in a separate stage, and copied over with a COPY --link. Using a fast base image, that is, either a small image or an image already downloaded, will allow to download in parallel to the build. Copying them over in a COPY --link layer will make that layer not change when the rest of the build changes, and could be cached locally (and could even be the same if used in other Docker images that would result in the same layer hash).

I couldn't add the comment to these lines to the file, so I added them to the specific commit that reverted a change.


# Start build stage
FROM common as build

Expand All @@ -98,13 +117,7 @@ ENV GRASS_BUILD_PACKAGES="\
cmake \
libbz2-dev \
libcairo2-dev \
libopenblas-dev \
libpdal-dev \
libproj-dev \
libpq-dev \
libfftw3-dev \
libfreetype6-dev \
libomp-dev \
zlib1g-dev \
libnetcdf-dev \
libopenjp2-7-dev \
Expand Down Expand Up @@ -144,28 +157,17 @@ RUN (echo "Install Python" \
numpy \
Pillow \
ply \
setuptools \
matplotlib \
psycopg2 \
&& rm -r /root/.cache \
&& rm -rf /tmp/pip-* \
)

# Users are encouraged to fetch vertical datums and store into PROJ dir, e.g.:
# WORKDIR /src
# RUN mkdir vdatum && \
# cd vdatum && \
# wget -q http://download.osgeo.org/proj/vdatum/usa_geoid2012.zip && unzip -j -u usa_geoid2012.zip -d /usr/share/proj; \
# cd .. && \
# rm -rf vdatum

# copy grass gis source
COPY . /src/grass_build/

WORKDIR /src/grass_build

# Cleanup potentially leftover GISRC file with wrong path to "demolocation"
RUN rm -f /src/grass_build/dist.*/demolocation/.grassrc*

# Set environmental variables for GRASS GIS compilation, without debug symbols
# Set gcc/g++ environmental variables for GRASS GIS compilation, without debug symbols
ENV MYCFLAGS "-O2 -std=gnu99 -m64"
Expand Down Expand Up @@ -203,7 +205,14 @@ RUN ./configure \
--with-openmp \
--without-opengl \
&& make -j $NUMTHREADS \
&& make install && ldconfig
&& make install && ldconfig \
&& cp /usr/local/grass84/gui/wxpython/xml/module_items.xml module_items.xml; \
rm -rf /usr/local/grass84/demolocation; \
rm -rf /usr/local/grass84/fonts; \
rm -rf /usr/local/grass84/gui; \
rm -rf /usr/local/grass84/share; \
mkdir -p /usr/local/grass84/gui/wxpython/xml/; \
mv module_items.xml /usr/local/grass84/gui/wxpython/xml/module_items.xml;

# Build the GDAL-GRASS plugin
RUN git clone https://github.com/OSGeo/gdal-grass && \
Expand All @@ -215,25 +224,6 @@ RUN git clone https://github.com/OSGeo/gdal-grass && \
cd /src && \
rm -rf "gdal-grass"

# Reduce the image size - Remove unnecessary grass files
RUN cp /usr/local/grass84/gui/wxpython/xml/module_items.xml module_items.xml; \
rm -rf /usr/local/grass84/demolocation; \
rm -rf /usr/local/grass84/fonts; \
rm -rf /usr/local/grass84/gui; \
rm -rf /usr/local/grass84/share; \
mkdir -p /usr/local/grass84/gui/wxpython/xml/; \
mv module_items.xml /usr/local/grass84/gui/wxpython/xml/module_items.xml;

# Unset environmental variables to avoid later compilation issues
ENV INTEL ""
ENV MYCFLAGS ""
ENV MYLDFLAGS ""
ENV MYCXXFLAGS ""
ENV LD_LIBRARY_PATH ""
ENV LDFLAGS ""
ENV CFLAGS ""
ENV CXXFLAGS ""

# Leave build stage
FROM common as grass

Expand All @@ -250,19 +240,20 @@ ENV GRASSBIN="/usr/local/bin/grass" \
GISBASE="/usr/local/grass/" \
GRASSBIN="/usr/local/bin/grass" \
PYTHONPATH="${PYTHONPATH}:/usr/local/grass/etc/python/" \
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/grass/lib"
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/grass/lib" \
GDAL_DRIVER_PATH="/usr/lib/gdalplugins"

# Copy GRASS GIS from build image
COPY --from=build /usr/local/bin/grass* /usr/local/bin/
COPY --from=build /usr/local/grass84 /usr/local/grass84/
COPY --from=build /src/site-packages /usr/lib/python3.10/
COPY --from=build /usr/lib/gdalplugins /usr/lib/gdalplugins
COPY --link --from=build /usr/local/bin/* /usr/local/bin/
COPY --link --from=build /usr/local/grass84 /usr/local/grass84/
COPY --link --from=build /src/site-packages /usr/lib/python3.10/
COPY --link --from=build /usr/lib/gdalplugins /usr/lib/gdalplugins

# Create generic GRASS GIS lib name regardless of version number
RUN ln -sf /usr/local/grass84 /usr/local/grass && \
ldconfig /etc/ld.so.conf.d

ENV GDAL_DRIVER_PATH="/usr/lib/gdalplugins"
RUN ls /usr/local/bin/

# Data workdir
WORKDIR /grassdb
Expand Down
Loading