Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dockerfile and facilitate nrel/openfast Docker Hub registry #2124

Merged
merged 62 commits into from
Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
51a2902
FIX: Update dockerfile base to get required `cmake` version
cortadocodes Mar 22, 2024
0184e60
CHO: Add `.idea` to `.gitignore`
cortadocodes Mar 22, 2024
2900399
ENH: Make dockerfile base configurable
cortadocodes Mar 22, 2024
8519687
ENH: Make dockerfile timezone configurable
cortadocodes Mar 22, 2024
79f5910
FIX: Fix timezone build arg in dockerfile
cortadocodes Mar 22, 2024
5dc82f4
ENH: Make openfast version configurable in dockerfile
cortadocodes Mar 22, 2024
f6bc0cc
ENH: Make cores available for building configurable in dockerfile
cortadocodes Mar 22, 2024
57a1fbd
ENH: Clone openfast with shallow submodules in dockerfile
cortadocodes Mar 22, 2024
561eb22
REF: Combine apt installation commands into one
cortadocodes Mar 22, 2024
0302ea8
ENH: Use `apt-get` instead of `apt` and remove cache
cortadocodes Mar 22, 2024
5e63e58
ENH: Make `openfast` command available in docker image
cortadocodes Mar 22, 2024
965bfa1
ENH: Make building testing tree optional
cortadocodes Mar 22, 2024
025f117
ENH: Avoid cloning submodules if not building test tree
cortadocodes Mar 22, 2024
bfde527
REF: Sort `apt-get install` lines
cortadocodes Mar 22, 2024
db0a7e7
REF: Factor out repository URL
cortadocodes Mar 25, 2024
5eb65ab
ENH: Remove commented out numpy installation
cortadocodes Mar 25, 2024
38af413
ENH: Set timezone to UTC by default
cortadocodes Mar 25, 2024
cc8a9c9
ENH: Set `BUILD_TESTING` to "OFF" by default
cortadocodes Mar 25, 2024
ce8fa16
REF: Rename `VERSION` build arg to `OPENFAST_VERSION`
cortadocodes Mar 25, 2024
9822a4c
ENH: Remove need to prefix version numbers with `v`
cortadocodes Mar 25, 2024
2b97678
ENH: Build from local context instead of cloning repository
cortadocodes Mar 25, 2024
c8814e0
ENH: Use a multi-stage build to reduce final image size
cortadocodes Mar 25, 2024
c146096
FIX: Ensure system dependencies are available in final image
cortadocodes Mar 25, 2024
dc326b3
ENH: Reduce docker image size by minimising included packages
cortadocodes Mar 25, 2024
19e4051
ENH: Remove extra packages from first build stage
cortadocodes Mar 25, 2024
fb67cc3
ENH: Allow injection of all `cmake` args
cortadocodes Mar 25, 2024
367aca0
REF: Move environment variables and build args to more optimal places
cortadocodes Mar 25, 2024
782041b
ENH: Set `CMAKE_BUILD_TYPE` to `RELEASE` by default
cortadocodes Mar 25, 2024
d994413
FIX: Add essential dependencies back to build stage
cortadocodes Mar 25, 2024
d1c8982
ENH: Add `nano` back to production image
cortadocodes Mar 25, 2024
60a689b
ENH: Add comments explaining build stages
cortadocodes Mar 25, 2024
5ae67cc
CHO: Add dockerfiles to `.dockerignore`
cortadocodes Mar 25, 2024
e07542e
FIX: Add `git` back to build stage
cortadocodes Mar 25, 2024
96a33da
DOC: Add readme explaining dockerfile
cortadocodes Mar 26, 2024
b35146f
OPS: Add `build-docker-image` workflow
cortadocodes Mar 26, 2024
392492b
CHO: Update copyright year in dockerfile
cortadocodes Mar 26, 2024
27a5c66
ENH: Get dockerfile if running workflow on an older version
cortadocodes Mar 26, 2024
ffae93a
MRG: Merge pull request #1 from octue/update-dockerfiles
cortadocodes Mar 26, 2024
e43e760
OPS: Use correct docker action versions
cortadocodes Mar 26, 2024
e63477b
MRG: Merge branch 'update-dockerfiles'
cortadocodes Mar 26, 2024
a51922a
OPS: Use environment context instead of environment variables
cortadocodes Mar 26, 2024
6a47596
MRG: Merge branch 'update-dockerfiles'
cortadocodes Mar 26, 2024
24bd1ff
ENH: Use `--no-install-recommends` in dockerfile
cortadocodes Mar 26, 2024
8cbac60
OPS: Factor out and fix docker repository in docker workflow
cortadocodes Mar 26, 2024
ff51c68
MRG: Merge branch 'update-dockerfiles'
cortadocodes Mar 26, 2024
8de46aa
OPS: Add comment explaining Dockerfile download step
cortadocodes Mar 27, 2024
c6aee2b
OPS: Rename environment variable for consistency
cortadocodes Mar 27, 2024
9a12135
CHO: Move `.idea` into correct section of `.gitignore`
cortadocodes Mar 27, 2024
520931c
ENH: Add 2016 to start of copyright notice in Dockerfile
cortadocodes Mar 27, 2024
aab4a6d
OPS: Build docker image for `linux/aarch64` too
cortadocodes Mar 27, 2024
d59fccb
MRG: Merge branch 'update-dockerfiles'
cortadocodes Mar 27, 2024
14e77db
OPS: Add description to docker workflow
cortadocodes Mar 27, 2024
3e06eab
OPS: Add automatic docker build on merge into `main` from RC branches
cortadocodes Mar 27, 2024
0e8a61a
OPS: Add `test-build-docker-image` workflow
cortadocodes Mar 27, 2024
4a04caa
ENH: Use long-term-support ubuntu base by default
cortadocodes Mar 28, 2024
a6d4197
DOC: Update documentation
cortadocodes Mar 28, 2024
6b2e890
DOC: Update dockerfile readme
cortadocodes Mar 28, 2024
26c48da
DOC: Fix link in docs
cortadocodes Mar 28, 2024
7bd86cc
MRG: Merge branch 'update-dockerfiles'
cortadocodes Mar 28, 2024
11c8cbf
OPS: Fix version extraction
cortadocodes Mar 28, 2024
1c3115f
OPS: Use extracted tag in test workflow
cortadocodes Mar 28, 2024
5e80427
OPS: Fix tag extraction in automated docker workflow
cortadocodes Mar 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# System, cache and editor files
**/__pycache__
**/.DS_Store

.editorconfig
.idea
.git
.github
.gitignore
.pytest_cache
.pre-commit-config.yaml
.readthedocs.yml
.coverage
Procfile
README.md

# Never bake in environment variables
.env
.env*
google_credentials.json
# See: https://github.com/google-github-actions/auth/issues/123
gha-creds-*.json
gcp-creds-*.json

.devcontainer/.zsh_history
share/docker
48 changes: 48 additions & 0 deletions .github/workflows/build-docker-image-automatic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# This workflow builds an OpenFAST docker image for the linux/amd64 and linux/aarch64 architectures on merge into
# `main` from a release candidate branch. The image is tagged both with "latest" and the version extracted from the
# release candidate branch's name (e.g. "rc-3.5.3") before being pushed to the `nrel/openfast` repository. The build
# cache is stored in GitHub actions.
name: build-docker-image-automatic

on:
pull_request:
types:
- closed
branches:
- main

jobs:
build-and-push:
if: ${{ (github.event.pull_request.merged == true) && startsWith(github.head_ref, 'rc-') }}
runs-on: ubuntu-latest
timeout-minutes: 300
env:
DOCKERFILE_PATH: share/docker/openfast_ubuntu/Dockerfile
DOCKERHUB_REPOSITORY: octue/openfast
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Extract tag from release candidate branch name
id: extract-tag
run: echo "openfast-tag=${"${{ github.head_ref }}":3}" >> $GITHUB_OUTPUT

- name: Build and push to registry
uses: docker/build-push-action@v5
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
platforms: linux/amd64,linux/aarch64
tags: ${{ env.DOCKERHUB_REPOSITORY }}:${{ steps.extract-tag.outputs.openfast-tag }},${{ env.DOCKERHUB_REPOSITORY }}:latest
push: true
cache-from: type=gha
cache-to: type=gha,mode=max
58 changes: 58 additions & 0 deletions .github/workflows/build-docker-image-manual.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# This manually-triggered workflow builds OpenFAST docker images for the linux/amd64 and linux/aarch64 architectures for
# the specified git ref (this can be a branch, tag, or commit hash). The image is tagged with the given tag and pushed
# to the `nrel/openfast` repository. The build cache is stored in GitHub actions.
name: build-docker-image-manual

on:
workflow_dispatch:
inputs:
tag:
description: 'Tag for Docker image (excluding the "v" prefix e.g. 3.5.3)'
required: true
ref:
description: 'Branch, tag, or commit SHA to build from'
required: true
default: main

jobs:
build-and-push:
runs-on: ubuntu-latest
timeout-minutes: 300
env:
DOCKERFILE_PATH: share/docker/openfast_ubuntu/Dockerfile
DOCKERFILE_PERMALINK: https://raw.githubusercontent.com/octue/openfast/update-dockerfiles/share/docker/openfast_ubuntu/Dockerfile
DOCKERHUB_REPOSITORY: octue/openfast
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.ref }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

# The updated Dockerfile is only available in the repository at the tag v3.5.3 and above. To build versions of
# OpenFAST that are below this version, the updated Dockerfile from that tag of the repository has to be acquired
# before building. For versions >= v3.5.3, the Dockerfile is already there so this step does nothing.
- name: Get Dockerfile for versions < 3.5.3
run: |
if [ ! -f ${{ env.DOCKERFILE_PATH }} ]; \
then wget -O ${{ env.DOCKERFILE_PATH }} ${{ env.DOCKERFILE_PERMALINK }}; \
fi

- name: Build and push to registry
uses: docker/build-push-action@v5
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
platforms: linux/amd64,linux/aarch64
tags: ${{ env.DOCKERHUB_REPOSITORY }}:${{ github.event.inputs.tag }}
push: true
cache-from: type=gha
cache-to: type=gha,mode=max
36 changes: 36 additions & 0 deletions .github/workflows/test-build-docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This workflow tests building an OpenFAST docker image for the linux/amd64 architecture on push to a release candidate
# branch. The build cache is stored in GitHub actions.
name: test-build-docker-image

on:
pull_request:
branches:
- main

jobs:
test-build:
if: startsWith(github.head_ref, 'rc-')
runs-on: ubuntu-latest
timeout-minutes: 300
env:
DOCKERFILE_PATH: share/docker/openfast_ubuntu/Dockerfile
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Extract tag from release candidate branch name
id: extract-tag
run: echo "openfast-tag=${"${{ github.head_ref }}":3}" >> $GITHUB_OUTPUT

- name: Test building docker image
uses: docker/build-push-action@v5
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
platforms: linux/amd64
push: false
cache-from: type=gha
cache-to: type=gha,mode=max
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ vs-build/
.atom
.fortls
.devcontainer
.idea

# backup files
*.asv
~$*.xlsx
Expand All @@ -54,4 +56,3 @@ vs-build/
#Simulink cache files
varcache
*.slxc

70 changes: 45 additions & 25 deletions share/docker/openfast_ubuntu/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#
# Copyright 2016 National Renewable Energy Laboratory
# Copyright 2016-2024 National Renewable Energy Laboratory
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -13,42 +12,63 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
ARG BASE=ubuntu:mantic
andrew-platt marked this conversation as resolved.
Show resolved Hide resolved

FROM ubuntu:bionic
# Build stage 1: builds openfast.
FROM ${BASE} AS build

# Install dependencies

# For gfortran-8
# RUN add-apt-repository ppa:ubuntu-toolchain-r/test -y
# apt-get install gfortran-8

ENV DEBIAN_FRONTEND=noninteractive TZ=America/Denver

RUN apt update -qq && \
apt install -y software-properties-common build-essential && \
add-apt-repository ppa:git-core/ppa -y && \
apt install -y python3-pip && \
apt install -y cmake cmake-curses-gui && \
apt install -y gcc gfortran make && \
apt install -y libblas-dev liblapack-dev && \
apt install -y git && \
apt install -y nano
ENV DEBIAN_FRONTEND=noninteractive

RUN pip3 install numpy
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y \
software-properties-common \
build-essential \
cmake \
cmake-curses-gui \
gcc \
gfortran \
git \
libblas-dev \
liblapack-dev \
cortadocodes marked this conversation as resolved.
Show resolved Hide resolved
make \
&& rm -rf /var/lib/apt/lists/*

# Configure the environment
ENV FC=/usr/bin/gfortran

# Clone the project
RUN git clone --recursive https://github.com/openfast/openfast.git openfast
# Copy in the checked-out code version.
WORKDIR /openfast
COPY . .

# Build the project
# Build the project.
RUN mkdir build
WORKDIR /openfast/build

# NOTE: building with optimizations on (RELEASE or RELWITHDEBINFO), the virtual machine
# will require about 6GB of memoery. Otherwise, the gfortran compiler will exit with an
# will require about 6GB of memory. Otherwise, the gfortran compiler will exit with an
# "internal error"
RUN cmake .. -DBUILD_TESTING=ON -DDOUBLE_PRECISION=ON -DCMAKE_BUILD_TYPE=RELWITHDEBINFO
RUN make -j4 install
ENV FC=/usr/bin/gfortran
ARG CMAKE_OPTIONS="-DBUILD_TESTING=OFF -DDOUBLE_PRECISION=ON -DCMAKE_BUILD_TYPE=RELEASE"
cortadocodes marked this conversation as resolved.
Show resolved Hide resolved
RUN cmake .. ${CMAKE_OPTIONS}

ARG BUILD_CORES=4
RUN make -j${BUILD_CORES} install

# Build stage 2: provides built openfast in a small image.
FROM ${BASE} as production
COPY --from=build /openfast/install /openfast/install

ARG TIMEZONE=UTC
ENV DEBIAN_FRONTEND=noninteractive TZ=${TIMEZONE}

RUN apt-get update -qq && \
apt-get install --no-install-recommends -y \
libblas-dev \
liblapack-dev \
nano \
&& rm -rf /var/lib/apt/lists/*

# Make `openfast` command work.
ENV PATH=/openfast/install/bin:$PATH
42 changes: 42 additions & 0 deletions share/docker/openfast_ubuntu/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# OpenFAST docker images
cortadocodes marked this conversation as resolved.
Show resolved Hide resolved

## Summary
The `Dockerfile` in this directory can be used to reliably build OpenFAST as a docker image that can be run locally and
in the cloud without much setup. By default, it's based on Ubuntu Mantic and is optimised in size and performance for
production use. A multi-stage build is used, producing an Ubuntu image with just `libblas-dev`, `liblapack-dev`, `nano`
and `openfast` added. The image built by this `Dockerfile` can be customised at build time using build arguments.

## Image registry
Production images of OpenFAST for the `linux/amd64` platform are available on the
[NREL docker hub](https://hub.docker.com/r/nrel/openfast).

## Build arguments
Provide any of the following build arguments to customise the image at build time.

| Name | Type | Allowed values | Default | Description |
| --------------- | ------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------- |-----------------------------------------------------------|
| `BASE` | String | Any valid docker image URI that has the `apt` package manager installed. | `ubuntu:mantic` | The docker image to base the OpenFAST image on. |
| `CMAKE_OPTIONS` | String | Any number of valid space-separated `cmake` options in the same format they're normally passed to `cmake` directly. See the options relevant to OpenFAST [here.](https://openfast.readthedocs.io/en/main/source/install/index.html#openfast-cmake-options) | `-DBUILD_TESTING=OFF -DDOUBLE_PRECISION=ON -DCMAKE_BUILD_TYPE=RELEASE` | Options to control how CMake is used to build OpenFAST. |
| `BUILD_CORES` | Integer | Any integer greater than 0. | `4` | The number of cores to use to build OpenFAST with `make`. |
| `TIMEZONE` | String | Any [valid timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | `UTC` | The timezone to use when running OpenFAST. |

For example, to build OpenFAST v3.5.3 for the `linux/amd64` platform and set `CMAKE_OPTIONS` so the testing tree is built:

```shell
# Run from the root of this repository.
git checkout v3.5.3
docker build -f share/docker/openfast_ubuntu/Dockerfile -t openfast:3.5.3 --platform=linux/amd64 --build-arg=CMAKE_OPTIONS='-DBUILD_TESTING=ON' .
```

**NOTE:** This version of the `Dockerfile` is only available in v3.5.3 and up of this repository. To build earlier
versions of OpenFAST, check out the code at that version and recreate the `Dockerfile` from v3.5.3 (or above) in the
checked-out repository first.

## Building development images
Development images can be built from the production image as a base. Simply start a new `Dockerfile` with:

```dockerfile
FROM nrel/openfast:3.5.3
```

Images can be built for different platforms using the `--platform` option when building the image.