automatically update vendored dependencies

[pohly]

In our various Kubernetes-CSI repos we should regularly update dependencies to ensure that we have included all upstream bug fixes.

This includes:

• running dep ensure -update
• running git subtree pull for release-tools (see https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating)
• creating a new PR for changes (if none is pending currently) or updating an existing PR by force-pushing to the branch

Then once the PR is pending, CI testing will cover the initial round of testing. Even if that fails, having a PR pending is a reminder to the maintainer that the component is not up-to-date.

The steps above could be automated with a script, like update-dependencies.sh in this repo. The hub tool might be useful for managing the PRs. Bonus points for making it possible to update release branches.

Running this script for one or more component is probably best done by the same person at a regular cadence. If multiple people do it, we end up with multiple competing PRs. We could also try to move it into a Prow job, but that would be a second step.

@msau42 does that sound useful?

[msau42]

Yes I think that sounds valuable. It would be great if there was also a wrapper script that can update and create prs across all the branches

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[pohly]

Quick-and-dirty script that I just used to update csi-relase-tools:

#!/bin/sh -x

die () {
    echo >&2 "\nERROR: $@\n"
    exit 1
}

while read repo branches; do
    if [ "$repo" != "#" ]; then
    (
        cd $repo || die "$repo: does not exit"
        git fetch origin || die "$repo: git fetch"
        for i in $branches; do
            git checkout -B prow-update-$i origin/$i || die "$repo:$i checkout"
            git subtree pull --prefix=release-tools -m "release-tools: update" https://github.com/kubernetes-csi/csi-release-tools.git master || die "$repo:$i update release tools"
            changes=$(git log --no-merges --oneline origin/$i..prow-update-$i)
            if [ "$changes" ]; then
                release-tools/update-vendor.sh || die "update vendor"
                git add vendor
                git commit --amend -F - <<EOF
release-tools: update

Commit summary:
$changes
EOF
                make test || die "$repo:$i make test"
                git push pohly prow-update-$i || die "pohly:prow-update-$i push failed - probably there is already an unmerged branch and pending PR"
                hub pull-request -F - -b kubernetes-csi/$repo:$i -h pohly/$repo:prow-update-$i <<EOF
$i: update release-tools

Commit summary:
$changes

\`\`\`release-note
NONE
\`\`\`
EOF
                if [ "$?" -ne 0 ]; then
                    die "$repo:$i pull-request"
                fi
            fi
        done
    ) || die "failed"
    fi
done <<EOF
csi-driver-host-path master
csi-driver-iscsi master
csi-driver-nfs master
csi-lib-utils master
csi-proxy master
csi-test master
external-attacher master
external-health-monitor master
external-provisioner master
external-resizer master
external-snapshotter master
livenessprobe master
node-driver-registrar master
EOF

[msau42]

/lifecycle frozen

[pohly]

In kubernetes-csi/csi-driver-smb#28 we were discussing how to use squashing instead of including the commits as done so far. We came up with a revised script, too. However, when I just tried to do the Go 1.15 update (see PR #97) I found that importing fails if (and only if) --squash was used. It could be because I am pulling from a different repo than usual, but I doubt it. I'll investigate once that PR is merged and in the meantime, continue with importing all commits.

/cc @animeshk08

csi-driver-host-path$ git subtree pull --prefix=release-tools https://github.com/pohly/csi-release-tools.git go-1.15 --squash
From https://github.com/pohly/csi-release-tools
 * branch              go-1.15    -> FETCH_HEAD
Auto-merging release-tools/travis.yml
CONFLICT (content): Merge conflict in release-tools/travis.yml
Auto-merging release-tools/SIDECAR_RELEASE_PROCESS.md
CONFLICT (add/add): Merge conflict in release-tools/SIDECAR_RELEASE_PROCESS.md
Recorded preimage for 'release-tools/SIDECAR_RELEASE_PROCESS.md'
Recorded preimage for 'release-tools/travis.yml'
Automatic merge failed; fix conflicts and then commit the result.

csi-driver-host-path$ git reset --hard origin/master
HEAD is now at 55f7ce61 Merge pull request #193 from msau42/changelog

csi-driver-host-path$ git subtree pull --prefix=release-tools https://github.com/pohly/csi-release-tools.git go-1.15 
From https://github.com/pohly/csi-release-tools
 * branch              go-1.15    -> FETCH_HEAD
hint: Waiting for your editor to close the file... Waiting for Emacs...
Merge made by the 'recursive' strategy.
 release-tools/SIDECAR_RELEASE_PROCESS.md | 6 ++++++
 release-tools/travis.yml                 | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

[Jiawei0227]

Thanks for putting up this script!

I am able to run the script #7 (comment) with some small changes.
The prerequisite for running the scripts is installing hub command. And have all the repos you want to update in place.

One thing people need to notice is that it depends on how their local git remote setup is. My remote setup is origin -> my fork and upstream -> the upstream repo. Which is not working in this script so I have to manually change some of the stuff there. If your setup is origin->upstream and <your github name>->your fork. Then the script above can be easily adopted.

One thing to investigate is the squash thing mentioned #7 (comment).

[Jiawei0227]

Also to copy my revised script here for anyone who is also using origin as fork and upstream as the upstream remote. The only thing need to change is line 31, where you can change my name to your github name.

#!/bin/sh -x

die () {
    echo >&2 "\nERROR: $@\n"
    exit 1
}

while read repo branches; do
    if [ "$repo" != "#" ]; then
    (
        cd $repo || die "$repo: does not exit"
        git fetch upstream || die "$repo: fetch upstream"
        git fetch origin || die "$repo: git fetch"
        for i in $branches; do
            git checkout $i
            git merge upstream/$i
            git checkout -B prow-update-$i origin/$i || die "$repo:$i checkout"
            git subtree pull --prefix=release-tools -m "release-tools: update" https://github.com/kubernetes-csi/csi-release-tools.git master || die "$repo:$i update release tools"
            changes=$(git log --no-merges --oneline origin/$i..prow-update-$i)
            if [ "$changes" ]; then
                release-tools/update-vendor.sh || die "update vendor"
                git add vendor
                git commit --amend -F - <<EOF
release-tools: update

Commit summary:
$changes
EOF
                make test || die "$repo:$i make test"
                git push origin prow-update-$i || die "origin:prow-update-$i push failed - probably there is already an unmerged branch and pending PR"
                hub pull-request -F - -b kubernetes-csi/$repo:$i -h Jiawei0227/$repo:prow-update-$i <<EOF
$i: update release-tools

Commit summary:
$changes

\`\`\`release-note
NONE
\`\`\`
EOF
                if [ "$?" -ne 0 ]; then
                    die "$repo:$i pull-request"
                fi
            fi
        done
    ) || die "failed"
    fi
done <<EOF
external-resizer release-1.0
external-provisioner release-2.0
external-attacher release-3.0
external-snapshotter release-3.0
EOF

[msau42]

It may be nice to actually check in the script to the repo. We can probably sub out some of the hardcoded values with variables?

[msau42]

It looks like having the "Merge pull request" messages from importing all the individual commits from csi-release-tools is messing up the release-notes generation tool.

See https://github.com/kubernetes-csi/external-resizer/pull/112/files#r501164729, where wrong PRs were picked up.

I believe what's happening is the release-notes tool is seeing "Merge pull request #98" from the release-tools update commit, and then going to PR 98 in the sidecar repo and sees that there's a release note for it.

So this is another reason why we should move towards the commit squashing model.

[pohly]

Here's a version of the update script which uses --squash with scripted resolution of the merge conflicts that this options sometimes seems to run into.

WARNING: "origin" in "git fetch origin" below must reference the upstream repository under github.com/kubernetes-csi.

#!/bin/sh -x

die () {
    echo >&2 "\nERROR: $@\n"
    exit 1
}

while read repo branches; do
    if [ "$repo" != "#" ]; then
    (
        cd $repo || die "$repo: does not exit"
        git fetch origin || die "$repo: git fetch"
        for i in $branches; do
            git checkout -B prow-update-$i origin/$i || die "$repo:$i checkout"
            rm -rf .git/MERGE*
            if ! git subtree pull --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master; then
                # Sometimes "--squash" leads to merge conflicts. Because we know that "release-tools"
                # is an unmodified copy of csi-release-tools, we can automatically resolve that
                # by replacing it completely.
                if [ -e .git/MERGE_MSG ] && [ -e .git/FETCH_HEAD ] && grep -q "^# Conflict" .git/MERGE_MSG; then
                    rm -rf release-tools
                    mkdir release-tools
                    git archive FETCH_HEAD  | tar -C release-tools -xf - || die "failed to re-create release-tools from FETCH_HEAD"
                    git add release-tools || die "add release-tools"
                    git commit --file=.git/MERGE_MSG || die "commit squashed release-tools"
                else
                    die "git subtree pull --squash failed, cannot reover."
                fi
            fi
            changes=$(git log --no-merges --format=%B -n 1 HEAD)
            if [ "$changes" ]; then
                # Avoid "Merge pull request #<number>" in the new commit messages because
                # https://github.com/kubernetes/release/blob/9d099f0757b46c7f3d209751e8e2a6056c8bca5b/pkg/notes/notes.go#L912
                # will misinterpret them as pull requests from the current repo.
                git filter-branch -f --msg-filter "sed -e 's;^\\([0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]*\\) ;https://github.com/kubernetes-csi/csi-release-tools/commit/\\1 ;' -e 's;Merge pull request #\\([0-9]*\\);Merge https://github.com/kubernetes-csi/csi-release-tools/pull/\\1;'" origin/$i..HEAD

                # Also make our PR message below nicer by turning commits and pull request numbers into URLs.
                changes="$(echo "$changes" | sed -e 's;^\([0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]*\) ;[\1](https://github.com/kubernetes-csi/csi-release-tools/commit/\1) ;' -e 's;Merge pull request #\([0-9]*\);Merge [pull request #\1](https://github.com/kubernetes-csi/csi-release-tools/pull/\1);')"

                release-tools/update-vendor.sh || die "update vendor"
                git add vendor || die "add vendor"
                git commit --amend --no-edit || die "commit vendor"
                make test || die "$repo:$i make test"
                git push pohly prow-update-$i || die "pohly:prow-update-$i push failed - probably there is already an unmerged branch and pending PR"
                hub pull-request -F - -b kubernetes-csi/$repo:$i -h pohly/$repo:prow-update-$i <<EOF
$i: update release-tools

$changes

\`\`\`release-note
NONE
\`\`\`
EOF
                if [ "$?" -ne 0 ]; then
                    die "$repo:$i pull-request"
                fi
            fi
        done
    ) || die "failed"
    fi
done <<EOF
csi-driver-host-path master
csi-driver-iscsi master
csi-driver-nfs master
csi-lib-utils master
csi-proxy master
csi-test master
external-attacher master
external-health-monitor master
external-provisioner master
external-resizer master
external-snapshotter master
livenessprobe master
node-driver-registrar master
EOF

[pohly]

I created PRs using this script, for example:
kubernetes-csi/node-driver-registrar#115

One downside (at least initially?) is that the list of commits includes everything. This might be because "git subtree" was unable to identify what was already there, which might also explain the merge conflict. Perhaps it'll work better after we switched. The squash commit contains some annotations which may help with that.

[sunnylovestiramisu]

Here is the wrapper script I used to generate PRs for 1.26 sidecar release:

#!/bin/bash

die () {
    echo >&2 "\nERROR: $@\n"
    exit 1
}

while read repo branches; do
    if [ "$repo" != "#" ]; then
    (
        cd $repo || die "$repo: does not exit"
        git fetch origin || die "$repo: git fetch"
        for i in $branches; do
            git checkout -B module-update-$i origin/$i || die "$repo:$i checkout"
            rm -rf .git/MERGE*
            if ! git subtree pull --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master; then
                # Sometimes "--squash" leads to merge conflicts. Because we know that "release-tools"
                # is an unmodified copy of csi-release-tools, we can automatically resolve that
                # by replacing it completely.
                if [ -e .git/MERGE_MSG ] && [ -e .git/FETCH_HEAD ] && grep -q "^# Conflict" .git/MERGE_MSG; then
                    rm -rf release-tools
                    mkdir release-tools
                    git archive FETCH_HEAD  | tar -C release-tools -xf - || die "failed to re-create release-tools from FETCH_HEAD"
                    git add release-tools || die "add release-tools"
                    git commit --file=.git/MERGE_MSG || die "commit squashed release-tools"
                else
                    die "git subtree pull --squash failed, cannot reover."
                fi
            fi
            while ! /usr/local/google/home/songsunny/Documents/GKE/SidecarRelease/$repo/release-tools/go-get-kubernetes.sh -p 1.26.0-rc.0
                do
                  go mod tidy && go mod vendor && go mod tidy
            done   
            go mod vendor && go mod tidy || die "last go mod vendor && go mod tidy failed"
            git add --all || die "git add -all failed"
            git commit -m "Update dependency go modules for k8s v1.26.0-rc.0" || die "commit update modules"
            git remote set-url origin https://github.com/sunnylovestiramisu/$repo.git || die "git remote set-url failed"
            make test || die "$repo:$i make test"
            git push origin module-update-$i --force || die "origin:module-update-$i push failed - probably there is already an unmerged branch and pending PR"
            gh pr create --title="Update dependency go modules for k8s v1.26.0-rc.0" --body "Ran kubernetes-csi/csi-release-tools go-get-kubernetes.sh -p 1.26.0-rc.0"  --head "sunnylovestiramisu:module-update-master" --base "master" --repo="kubernetes-csi/$repo" <<EOF
$i: update release-tools

$changes

\`\`\`release-note
NONE
\`\`\`
EOF
                if [ "$?" -ne 0 ]; then
                    die "$repo:$i pull-request"
                fi
        done
    ) || die "failed"
    fi
done <<EOF
external-attacher master
external-health-monitor master
external-provisioner master
external-resizer master
external-snapshotter master
livenessprobe master
node-driver-registrar master
EOF