(0.37.0) flux bootstrap git failed with 'unable to checkout repo with an empty set of auth options'

[karunsiri]

Describe the bug

After each Flux release, I would re-run the bootstrap command to upgrade flux resources running within a cluster.
The problem occurs when I try to upgrade from 0.36.0 to 0.37.0. When I run bootstrap, the command fails and outputs
failed to clone repository: unable to checkout repo with an empty set of auth options.

The pass versions can be re-bootstrapped (upgraded) successfully.

Steps to reproduce

1. To Rerun bootstrap onto existing flux-ready cluster. Given cluster currently running flux 0.36.0.
2. Upgrade flux cli to 0.37.0
3. Rerun bootstrap command:

flux bootstrap git \
  --url ssh://git@github.com/ORG/REPO.git \
  --branch master \
  --path path/to/manifests \
  --interval 1m0s \
  --ssh-key-algorithm ed25519

Expected behavior

The command should succeed and upgrade flux resources running within the cluster to the latest version.

Screenshots and recordings

OS / Distro

Fedora 37

Flux version

0.37.0

Flux check

► checking prerequisites
✔ Kubernetes 1.21.14-eks-fb459a0 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.26.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.30.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.28.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.31.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

GitHub

Container Registry provider

No response

Additional context

Additional Info: I can git clone <repo_url> directly just fine.

Code of Conduct

• I agree to follow this project's Code of Conduct

[aryan9600]

hi @karunsiri, could you describe what your ~/.ssh/config looks like?

[karunsiri]

Hello @aryan9600, here it is

Host *
ServerAliveInterval 120
IdentityFile ~/.ssh/private-key

Host github.com
User git
Hostname github.com

Additional Info: I can git clone <repo_url> directly just fine with this config. I'll update the issue to include this info.

[aryan9600]

is your private key file not protected by a passphrase? also could you try passing in --private-key-file=~/.ssh/private-key to command? (if you have a passphrase, then you can specify with it via the --password arg). that should probably fix it.

[karunsiri]

@aryan9600 It does work, yes, but that's going to replace the secret flux-system within the cluster, which is different than my private-key. It's basically uploading my key pair to be used by the cluster, on my behalf, right?

I never had to do this for the past version up to 0.36.0. I can upgrade progressively by running the same command, and flux will pickup the existing secret (flux-system) from the cluster and use that.

[aryan9600]

hey @karunsiri, i've opened PR #3377 which should fix this. would you be able to test it? these steps should work:

• git clone https://github.com/fluxcd/flux2
• gh pr checkout 3377
• make build
• ./bin/flux bootstrap git ....

[jjzazuet]

Can confirm that --private-key-file= can be used as a work-around.

Thanks.

[karunsiri]

@aryan9600 Sorry for delay response, I was on vacation.
I tried the given steps but the build failed while building helm-controller.yaml.

The build log:

❯❯❯ make build
./manifests/scripts/bundle.sh
[INFO]  using {kustomize/v4.5.7  2022-08-02T16:35:54Z  }
[INFO]  building helm-controller.yaml
Error: accumulating resources: accumulation err='accumulating resources from 'https://github.com/fluxcd/helm-controller/releases/download/v0.27.0/helm-controller.deployment.yaml': URL is a git repository': evalsymlink failure on '/tmp/kustomize-588017855/releases/download/v0.27.0/helm-controller.deployment.yaml' : lstat /tmp/kustomize-588017855/releases: no such file or directory
make: *** [Makefile:53: cmd/flux/.manifests.done] Error 1

[aryan9600]

i just ran the command after doing a fresh clone of the repo and checking out to the PR branch and its working. its probably an issue on GitHub's side. could you try again? @karunsiri

[ppmathis]

I have been facing the same issue as @karunsiri and noticed that I suddenly could no longer rely on Flux using the default SSH configuration for bootstrapping, which would then use my SSH authentication agent to communicate with GitHub.

Based on the instructions in #3358 (comment) I tested #3377 by @aryan9600 and can indeed confirm that said PR fixed the issue for me. Unlike the current release, this allowed me once again to bootstrap Flux with an SSH-based Git repository without specifying any credentials.