feat: docker build and tag from ci

[olizilla]

• build docker image in ci so we can see failures.
• tag and push to dockerhub from ci so we have more control and visibility on the process.
• use a script to decide what docket tag to apply given the current branch name or tag name.
• automatically create docker tags with a build number and short git commit hash for each build from the feat/stabilize-dht branch on go-ipfs. e.g.
  • bifrost-1-a44c610c
  • bifrost-2-78347be9
  • bifrost-<build number>-<short git sha>
  • the build number lets us operators talk about "rolling back" and "rolling forward" through the tags
  • the commit sha means we all know exactly which version of go-ipfs we are talking about, and lets us find out how far behind the latest HEAD of the branch we are.

I've recreated the rest of the dockerhub autobuild rules we have currently, but it is worth taking a moment to review them as we may not want all of them, see:

https://hub.docker.com/repository/docker/ipfs/go-ipfs/builds

Of note the alternative here is to keep using dockerhub for autobuilds, and just have a job to build the docker image in CI to verify it builds, and a job to do the custom bifrost tag... but the dockerhub config is not version controlled, and is a faff to change, so I went the whole hog and ported the lot. Very much up for discussion though.

Also of note, the current set up means the workflow we have called test is executed for all branches but not tags (which is how its always been). I've made it so that the docker-build step runs in parallel, but the docker-push where we tag and publish to dockerhub requires all the tests to pass, which is much stricter than the current dockerhub autobuild, which will run regardless of our test failures.

I've created a seperate workflow to trigger the docker steps when the repo is tagged, but it doesn't currently require any test jobs. It probably should, but I wanted to get a sanity check on what's here so far first.

This PR currently has the safety swtiches engaged in that it's currently configured to publish to an image on dockerhub called olizilla/test-go-ipfs rather than ipfs/go-ipfs for now, and the script to do the tag and push to docker is in dry-run mode, so it would just log out the tag and push step rather than do them.

TODO:

• create dockerhub token
• create circleci context called dockerhub scoped to the go-ipfs release github team
• remove dryrun flag from call to push-docker-tags.sh
• set IMAGE_NAME to be ipfs/go-ipfs
• decide on test requirements for branch and tag builds before pushing to dockerhub
• remove autobuilds from dockerhub

License: MIT
Signed-off-by: Oli Evans oli@tableflip.io

[olizilla]

This PR was based on ipfs-inactive/ci-websites#10 which was inspired by the handy article on exactly this topic https://circleci.com/blog/using-circleci-workflows-to-replicate-docker-hub-automated-builds/

[Stebalien]

When are these credentials provided?

[Stebalien]

Every branch, it looks like? That's not a good idea.

[olizilla]

I keep running into this same wall. 🤦‍♂

[Stebalien]

Resolution: only users in a specific group will get this token. That will also be the group with release permissions.

[olizilla]

@hsanjuan is the cluster-{\1} autobuild still needed?

[olizilla]

I've discovered docker hub supports in repo hooks that allow us to add extra tags for a given auto build, and drafting a PR to try it out. https://docs.docker.com/docker-hub/builds/advanced/#override-build-test-or-push-commands

[hsanjuan]

@hsanjuan is the cluster-{\1} autobuild still needed?

No

[olizilla]

Ok. I'm going to drag this version over the line. I'm going with date stamps rather than using circleci build numbers as a datestamp does the same job of telling you that a given image was created before or after another, and isn't tied to circleci...if we move ci in the future it would be dull to have the build number reset.

I'm going to test it out with my personal docker account, and the credentails will be added to a circle context with access limted to a specific admin github team https://github.com/orgs/ipfs/teams/admin/members

[olizilla]

@Stebalien ...sanity check on our current docker builds in dockerhub, whatever gets pushed to the release branch gets tagged as release and latest on docker, and git tags that look like semver get tagged on docker with a copy of that git tag name.

How do you handle releases, in terms of git branches and git tags? do you merge to the release branch and tag from there? How so you manage patch releases? do you keep a version branch going so you can backport things from master?

...the release docker tag feels redundant, and it seems a little racey that it's set from a release branch rather than a pointer to a git tag.

I could fix it in this PR so that we have docker tags just for:

1. semver docker tags for semver git tags
2. latest as a pointer to the latest semver tag
3. master-<date>-<commit> - for testground testing
4. master-latest pointing to latest master - for testground canary testing
5. bifrost-<date>-<commit> - for infra testing, until we can merge the streams.
6. bifrost-latest pointing to latest stabilize-dht. - for infra canary testing

[olizilla]

pushing from version tag is work! (tested with a tag on my fork of go-ipfs, and on my personal dockerhub profile)

[Stebalien]

I'm going to test it out with my personal docker account, and the credentails will be added to a circle context with access limted to a specific admin github team

Can we restrict this to some docker-release team? Or even a go-ipfs-release team?

[Stebalien]

How do you handle releases, in terms of git branches and git tags? do you merge to the release branch and tag from there? How so you manage patch releases? do you keep a version branch going so you can backport things from master?

To release, we currently:

1. Tag on master.
2. Merge the tag into release.

Same for patch releases.

For the next major release, we're probably going to have to perform some git magic to clobber the patch release changes.

...the release docker tag feels redundant, and it seems a little racey that it's set from a release branch rather than a pointer to a git tag.

I'm not actually sure why we do this. Possibly because "latest" isn't reliable? Latest also includes RCs, IIRC.

@whyrusleeping says "no reason really"

[Stebalien]

I could fix it in this PR so that we have docker tags just for:

I'm fine with that as long as latest actually points to the latest release. But is there any way to make release an alias for latest?

[olizilla]

Possibly because "latest" isn't reliable? Latest also includes RCs, IIRC.

With the current dockerhub autobuilds, the docker tags latest and release both point to whatever is the head of the release branch.

The proposal is to have the docker tag latest point to the same docker image as the most recent semver tag, and drop the release tag... aka, have none of our docker tags pointing to the head of the release branch.

[olizilla]

@Stebalien in it's current form, the circleci config requires that all tests pass on master or the stabilize-dht branch for use to push a new master-<date>-<commit> docker tag, while semver tags only require that the docker-build step succeeds.

This may seem a little backwards... we may well want bleeding edge docker tags fo branches that dont always pass all tests, but we dont want to publish a semver release until all the tests are green... in its current form, it's left up to the releaser to decide when a commit is good enough to tag, and as long as it builds, there will be a corresponding docker tag, regardless of tests, which may be fine.

Do you have preferences here? I'm thinking we could reduce the requirement for branch builds. Where should we set the bar for those? I'm inclined to leave the tag builds as they are, at the discression of the releaser, on the assumption the are tagging from a branch that has passing tests already. If you decide to tag a release in spite of a failing test, it would be annoying to not automatically get a docker tag for that because of ci.

[olizilla]

Branch tagging also works!
https://app.circleci.com/pipelines/github/olizilla/go-ipfs/20/workflows/cdd10c75-998f-48df-8009-80b0c4732371/jobs/152

Tagging ********/test-go-ipfs:bifrost-2020-03-13-a527a7a and pushing to dockerhub
The push refers to repository [docker.io/********/test-go-ipfs]

b3a71461: Preparing 
120503b3: Preparing 
5cbec343: Preparing 
c9d8d2e3: Preparing 
34e54786: Preparing 
83601870: Preparing 
b220ea20: Preparing 
04fb0a52: Preparing 
63bdbaf4: Preparing 
9e45ad9e: Preparing 
eaf85be0: Preparing 
335f47a8: Preparing 
c115622f: Preparing 
bifrost-2020-03-13-a527a7a: digest: sha256:722b1f290829c45cd5a437ceece44fc38fb281984c34b09c7298fe62cf14eb34 size: 3241
Tagging ********/test-go-ipfs:bifrost-latest and pushing to dockerhub
The push refers to repository [docker.io/********/test-go-ipfs]

b3a71461: Preparing 
120503b3: Preparing 
5cbec343: Preparing 
c9d8d2e3: Preparing 
34e54786: Preparing 
83601870: Preparing 
b220ea20: Preparing 
04fb0a52: Preparing 
63bdbaf4: Preparing 
9e45ad9e: Preparing 
eaf85be0: Preparing 
335f47a8: Preparing 
c115622f: Preparing 
bifrost-latest: digest: sha256:722b1f290829c45cd5a437ceece44fc38fb281984c34b09c7298fe62cf14eb34 size: 3241

CircleCI received exit code 0

a wild bifrost-latest tag and a bifrost-2020-03-13-a527a7a tag appear! (on the test repo)

[olizilla]

The dockerhub context is set up on circleci and limited to the go-ipfs release github team. the image name is updated to use the real deal ipfs/go-ipfs.

To land this we just need to verify that we're happy with branch builds requiring all the tests to pass, while tags require none, but assume the tagger will make the call... then remove all the autobuilds from dockerhub and merge it.

[Stebalien]

Do you have preferences here?

I'd say everything must always pass. If this becomes an issue, we can reduce the requirements (or just fix the issue).

[lidel]

master-${BUILD_NUM}-${GIT_SHA1_SHORT} is extremely useful to have, thank you! 👍