Cannot fetch metadata for automation from private images

[DSAWimi]

With our new K8s Installation (1.19.4) and Flux (1.21.0) we have problems fetching metadata from docker hub for private images. In the Flux log we can see, that flux find the correct secret and try to use it, but the call ends with access denied. Inside the Deployment, we use the same secret as imagePullSecret and that works fine. The automation for public images works also correct.

Setting

• Kubernetes: 1.19.4
• Helm: 3.4.1
• Flux: 1.21.0
• Git provider: Bitbucket
• Container Registry Provider: docker.io

Create Secret
kubectl create secret docker-registry docker-hub-cred --namespace=demo \ --docker-server=docker.io --docker-username=ro_user --docker-password='ro_pass' --docker-email='dockerhub.ro@mail.de'

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test1
  namespace: demo
  annotations:
    fluxcd.io/automated: "true"
    fluxcd.io/tag.test1: semver:~0.3.1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test1
  template:
    metadata:
      labels:
        app: test1
    spec:
      containers:
      - name: test1
        image: reg1/flux_test:0.3.1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
      imagePullSecrets:
        - name: docker-hub-cred

HELM Install

helm upgrade -i flux fluxcd/flux \
   --set git.user=Gituser \
   --set git.email=Gitmail \
   --set git.url=git@bitbucket.org:repo1 \
   --set git.path="namespaces\,workloads" \
   --set git.branch=master \
   --set syncGarbageCollection.enabled=true \
   --namespace flux

Flux-Log
ts=2021-01-27T09:13:27.3284659Z caller=warming.go:180 component=warmer canonical_name=index.docker.io/reg1/flux_test auth="{map[docker.io:<registry creds for ro_user@docker.io,cred>]}" err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"

We have already read other issues with similar problems. Like inside #2589 (comment) recommended we patched the default service account to use the imagepullsecret, but with no positive effect.
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "docker-hub-cred"}]}' -n demo

[DSAWimi]

The problem is solved. While the normal kubectl deployment accepts docker.io as docker server in the secrect, flux needs a complete URI
kubectl create secret docker-registry docker-hub-cred --namespace=demo \ --docker-server=https://index.docker.io/v1/ --docker-username=ro_user --docker-password='ro_pass' --docker-email='dockerhub.ro@mail.de'