Docker in the Cosmos SDK #11688
Comments
|
Another example of using arch to save time and effort while testing against the latest versions of relevant software |
|
Dockerfile reduced to a single line: |
|
ACK to:
|
|
ack your ack on 2/3 items, let's compare an alpine Dockerfile to an Arch Dockerfile to a Debian Dockerfile and keep in mind that no one uses musl, and that musl does cause problems. alpine # Simple usage with a mounted data directory:
# > docker build -t simapp .
#
# Server:
# > docker run -it -p 26657:26657 -p 26656:26656 -v ~/.simapp:/root/.simapp simapp simd init test-chain
# TODO: need to set validator in genesis so start runs
# > docker run -it -p 26657:26657 -p 26656:26656 -v ~/.simapp:/root/.simapp simapp simd start
#
# Client: (Note the simapp binary always looks at ~/.simapp we can bind to different local storage)
# > docker run -it -p 26657:26657 -p 26656:26656 -v ~/.simappcli:/root/.simapp simapp simd keys add foo
# > docker run -it -p 26657:26657 -p 26656:26656 -v ~/.simappcli:/root/.simapp simapp simd keys list
# TODO: demo connecting rest-server (or is this in server now?)
FROM golang:alpine AS build-env
# Install minimum necessary dependencies
ENV PACKAGES curl make git libc-dev bash gcc linux-headers eudev-dev python3
RUN apk add --no-cache $PACKAGES
# Set working directory for the build
WORKDIR /go/src/github.com/cosmos/cosmos-sdk
# Add source files
COPY . .
# install simapp, remove packages
RUN make build-linux
# Final image
FROM alpine:edge
# Install ca-certificates
RUN apk add --update ca-certificates
WORKDIR /root
# Copy over binaries from the build-env
COPY --from=build-env /go/src/github.com/cosmos/cosmos-sdk/build/simd /usr/bin/simd
EXPOSE 26656 26657 1317 9090
# Run simd by default, omit entrypoint to ease using container with simcli
CMD ["simd"]debian # This file defines the container image used to build and test tm-db in CI.
# The CI workflows use the latest tag of tendermintdev/docker-tm-db-testing
# built from these settings.
#
# The jobs defined in the Build & Push workflow will build and update the image
# when changes to this file are merged. If you have other changes that require
# updates here, merge the changes here first and let the image get updated (or
# push a new version manually) before PRs that depend on them.
FROM golang:1.17-bullseye AS build
ENV LD_LIBRARY_PATH=/usr/local/lib
RUN apt-get update && apt-get install -y --no-install-recommends \
libbz2-dev libgflags-dev libsnappy-dev libzstd-dev zlib1g-dev \
make tar wget
FROM build AS install
ARG LEVELDB=1.20
ARG ROCKSDB=7.0.3
# Install cleveldb
RUN \
wget -q https://github.com/google/leveldb/archive/v${LEVELDB}.tar.gz \
&& tar xvf v${LEVELDB}.tar.gz \
&& cd leveldb-${LEVELDB} \
&& make \
&& cp -a out-static/lib* out-shared/lib* /usr/local/lib \
&& cd include \
&& cp -a leveldb /usr/local/include \
&& ldconfig \
&& cd ../.. \
&& rm -rf v${LEVELDB}.tar.gz leveldb-${LEVELDB}
# Install Rocksdb
RUN \
wget -q https://github.com/facebook/rocksdb/archive/v${ROCKSDB}.tar.gz \
&& tar -zxf v${ROCKSDB}.tar.gz \
&& cd rocksdb-${ROCKSDB} \
&& DEBUG_LEVEL=0 make -j4 shared_lib \
&& make install-shared \
&& ldconfig \
&& cd .. \
&& rm -rf v${ROCKSDB}.tar.gz rocksdb-${ROCKSDB}Arch # This file defines the container image used to build and test tm-db in CI.
# The CI workflows use the latest tag of tendermintdev/docker-tm-db-testing
# built from these settings.
#
# The jobs defined in the Build & Push workflow will build and update the image
# when changes to this file are merged. If you have other changes that require
# updates here, merge the changes here first and let the image get updated (or
# push a new version manually) before PRs that depend on them.
FROM faddat/archlinux
RUN pacman -Syyu --noconfirm rocksdb leveldb go git base-develthese do the same thing, except that the arch image will stay up to date and has all of the tooling needed to use any database we support and if we wanted to add Rust for some reason it would look like: # This file defines the container image used to build and test tm-db in CI.
# The CI workflows use the latest tag of tendermintdev/docker-tm-db-testing
# built from these settings.
#
# The jobs defined in the Build & Push workflow will build and update the image
# when changes to this file are merged. If you have other changes that require
# updates here, merge the changes here first and let the image get updated (or
# push a new version manually) before PRs that depend on them.
FROM faddat/archlinux
RUN pacman -Syyu --noconfirm rocksdb leveldb go git base-devel rustI did remove cruft from both Arch examples, like: because it is not necessary. And when Go upgrades, the container does, too. |
|
I'm fine with using Arch, tbh this isn't my area of expertise. But is there a reason why we'd depend on a fork or custom image (yours)? Is there not a canonical or official arch distro image? |
|
Awesome-- ok so, yeah this part is less than ideal: Arch linux is two: https://archlinuxarm.org - ARM and this repo: https://github.com/faddat/archlinux-docker grabs both the AMD64 and ARM filesystem snapshots, and then combines them into a multiarch docker manifest so that when we're using it, we do not need to do gymnastics like: https://github.com/CosmWasm/wasmd/blob/57ead1ade3d4ce7c6c510977ea6dfdbab0e6c970/Dockerfile#L4 Instead you just tell which system architectures you're targeting: The Docker image and build env can stay exactly the same from cloud to metal to embedded. https://hub.docker.com/r/faddat/archlinux It's proven fairly popular, and the image builds every hour in Github: https://github.com/faddat/archlinux-docker/actions If there are single point of failure (me) matienance concerns, we could fork it into either the Notional or Cosmos org, too. If it weren't for the musl issue, I'd likely prefer Alpine because of the larger user base, but I've seen musl cause unpleasantness time and again so I'd rather just avoid it. |
|
SGTM! |
Stage twodecide weather or not to keep all of base-devel:
|
|
Id prefer not to merge into one docker image. this will cause lots of confusion, in the near future we will be deprecating some images. Lets use the default image provided, if archlinux has one lets use that. |
|
Marko, yeah.... So, I've scrapped arch and reduced everything to a single docker image. One thing to note though, my environment is still a rolling release. Arch does have a default image, but that doesn't natively support arm. Here's what the new dockerfile looks like: # This dockerfile will be used to bootstrap the docker changes for the cosmos SDK, and will be built hourly.
FROM manjarolinux/base
ENV GOPATH=/go
ENV PATH=$PATH:/go/bin
# everything needed for cosmos development
RUN pacman -Syyu --noconfirm go base-devel git leveldb rocksdb make python snappy git-lfs jq wget curl protobuf yarn rustup npm unzip docker
# Useful tools
# concept: we want every tool frequently used when building in cosmos.
RUN go install mvdan.cc/gofumpt@latest && \
go install github.com/cweill/gotests/gotests@latest && \
go install github.com/fatih/gomodifytags@latest && \
go install github.com/josharian/impl@latest && \
go install github.com/haya14busa/goplay/cmd/goplay@latest && \
go install github.com/go-delve/delve/cmd/dlv@latest && \
go install honnef.co/go/tools/cmd/staticcheck@latest && \
go install golang.org/x/tools/gopls@latest && \
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest && \
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest && \
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest && \
go install github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway@v1.16.0 && \
go install github.com/cosmos/cosmos-proto/cmd/protoc-gen-go-pulsar@latest && \
go install honnef.co/go/tools/cmd/staticcheck@latest && \
go install github.com/bufbuild/buf/cmd/buf@latest
RUN rustup toolchain install stable
# Install slightly more complicated but still useful tools
RUN git clone https://github.com/cosmos/cosmos-proto && \
cd cosmos-proto && \
go install ./... && \
cp /go/bin/* /usr/bin && \
rm -rf /go && \
mkdir /go && \
chmod -R 777 /goThis natively supports arm so we can build for arm and x86 without all the crazy stuff in my former PR. |
|
@faddat are you still open to working on this? |
|
@tac0turtle is this still needed ? I can take a look |
|
yes this is still open, but i think there are two docker files that may be able to be combined, otherwise the other ones should stay separate. A single docker image doesnt make sense here |
|
@tac0turtle - I am open to working on it, and open to working on it with @mridhul (does anyone have any sane way to manage gh notifications? this kills me) @mridhul - if you want to, feel free to ping me on twitter (twitter.com/gadikian) |
Hi, I've been working on the SDK directly lately, and it's been fun. But the Docker situation has not been fun at all.
I shall opine now.
Few years back, it became fashionable-- not effective-- to minimize Docker containers in every possible case, culminating in the use of so-called distroless containers.
Maybe the distroless containers are fine as execution environments, but I'm not even too sure about that.
I'd like to get feedback on several options, and suggest the one that I prefer:
Remove all Dockerfiles: this is mean to users and we should not do it
Have a single Dockerfile: This is my preference. I've surveyed the containers and we could easily have one Dockerfile. This dockerfile could work for external repositories, like IAVL and tm-db, too. It could have cleveldb and rocksdb preinstalled, and we could generate it frequently.
Have many Dockerfiles and dependencies on Dockerfiles that aren't in this repo: this is mean to users, and we should not do it
Let's do versioning
We should always use the latest versions of Go and the container OS (we mainly use Alpine) so for example if we wanted to start a docker container we would:
FROM golang:alpine apk add rocksdb leveldb git python-3...and so on
This way, the OS in our containers would always be up to date, as would the Go programming language. If a build breaks because of versioning, like RocksDB would have when v7 was released, that should be a signal to us that we need to fix something, instead of specifying potentially obsolete software.
The container image
I figure that we should default to either Alpine (this isn't my preference because no one uses musl in real life) or https://hub.docker.com/r/faddat/archlinux -- which is my preference because Arch is a rolling release distribution that has the latest security and stability patches applied almost immediately, almost all the time, and it uses glibc, which is what peoiple actually use.
The disadvantage of faddat/archlinux is that I made it, which is somewhat sus, compared to alpine or ubuntu.
we should not use ubuntu for containers
The reality that Ubuntu thinks that it's alright to ship years-old releases of golang in their package manager says it all.
If colleagues could please opine-- I'd love to reduce the SDK to a single Dockerfile located in the repository root, with all relevant dependencies in it, and it is fine by me if that's alpine based or based on faddat/archlinux.
but sir, why not just use arch
Sadly, they do not ship ARM builds in any official manner, and faddat/archlinux combines arch linux and arch linux arm to provide a multiarch build environment
Let's drop docker hub
using ghcr would allow us to have the images right here, and I think that is better.
The text was updated successfully, but these errors were encountered: