bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 #766

w13915984028 · 2024-07-12T09:12:49Z

Problem:

Bump the rancher-monitoring chart to 103.1.1+up45.1.1 and fix CVEs.

Bump rancher-logging to 103.1.0+up4.4.0

Solution:

Bump rancher-monitoring charts
Bump rancher-logging charts
Bump shell image (0.1.26) to reduce CVE
Bump eventrouter image (not available yet)

Upstream has fixed the rancher-monitoring-crd RBAC issue, the local patch is removed.

The image docker.io/rancher/shell:v0.1.23 is used by Rancher. The rancher-monitoring and rancher-logging is verified not using the v0.1.23 but v0.1.26.

Depends on PR

Bump rancher-monitoring and rancher-logging chart version addons#5

Related Issue:
harvester/harvester#6166

Test plan:

(1) Install a new cluster
(2) Enable rancher-monitoring addon, it should work
(3) Enable rancher-logging addon, it should work

The upgrade processing will be in another PR

local test:

$ cat /oem/harvester.config
...
runtimeversion: v1.27.13+rke2r1
rancherversion: v2.8.3
harvesterchartversion: 0.0.0-master-8c709bac
monitoringchartversion: 103.1.1+up45.31.1
..
loggingchartversion: 103.1.0+up4.4.0

pods:

harv21:/home/rancher # kubectl get pods -n cattle-monitoring-system
NAME                                                     READY   STATUS      RESTARTS   AGE
alertmanager-rancher-monitoring-alertmanager-0           2/2     Running     0          32m
helm-install-rancher-monitoring-xzwq9                    0/1     Completed   0          32m
prometheus-rancher-monitoring-prometheus-0               3/3     Running     0          32m
rancher-monitoring-grafana-d6f466988-vcc6x               4/4     Running     0          32m
rancher-monitoring-kube-state-metrics-7889dccd84-x9fpj   1/1     Running     0          32m
rancher-monitoring-operator-7545bff858-js5nh             1/1     Running     0          32m
rancher-monitoring-prometheus-adapter-55dc9ccd5d-mj7db   1/1     Running     0          32m
rancher-monitoring-prometheus-node-exporter-7h6pg        1/1     Running     0          32m
harv21:/home/rancher # kubectl get pods -n cattle-logging-system
NAME                                                      READY   STATUS      RESTARTS   AGE
harvester-default-event-tailer-0                          1/1     Running     0          32m
helm-install-rancher-logging-8mgvk                        0/1     Completed   0          32m
rancher-logging-68cb99b5bd-rpg8d                          1/1     Running     0          33m
rancher-logging-kube-audit-fluentbit-22bm5                1/1     Running     0          31m
rancher-logging-kube-audit-fluentd-0                      2/2     Running     0          31m
rancher-logging-kube-audit-fluentd-configcheck-ac2d4553   0/1     Completed   0          31m
rancher-logging-rke2-journald-aggregator-xzw6s            1/1     Running     0          33m
rancher-logging-root-fluentbit-lqpk7                      1/1     Running     0          32m
rancher-logging-root-fluentd-0                            2/2     Running     0          32m
rancher-logging-root-fluentd-configcheck-ac2d4553         0/1     Completed   0          32m
harv21:/home/rancher #

build pulling log:

Pulling images...
...
docker.io/rancher/rancher:v2.8.3
docker.io/rancher/shell:v0.1.26
docker.io/rancher/shell:v0.1.23
docker.io/rancher/system-agent:v0.3.6-suc

w13915984028 · 2024-07-17T08:01:47Z

@mergify backport v1.4

mergify · 2024-07-17T08:01:54Z

backport v1.4

✅ Backports have been created

#782 bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 (backport #766) has been created for branch v1.4

w13915984028 · 2024-07-17T09:45:48Z

About image docker.io/rancher/shell:v0.1.23, it is used by Rancher.

$ kubectl get pods -n cattle-system -n cattle-system
NAME                                         READY   STATUS      RESTARTS   AGE
harvester-cluster-repo-6d9fb84bb4-7ntxt      1/1     Running     0          26m
helm-operation-6t8dp                         0/2     Completed   0          26m
helm-operation-h7dns                         0/2     Completed   0          27m
helm-operation-k7pqw                         0/2     Completed   0          23m
helm-operation-tnq6g                         0/2     Completed   0          25m
helm-operation-wdg7w                         0/2     Completed   0          27m
helm-operation-wswnv                         0/2     Completed   0          27m
rancher-7f7df858d-fwn5l                      1/1     Running     0          25m
rancher-webhook-795765957f-f5rpw             1/1     Running     0          27m
system-upgrade-controller-78cfb99bb7-qlk8b   1/1     Running     0          26m

helm-operation tasks which uses shell v0.1.23

$ kubeclt get pods -n cattle-system -n cattle-system helm-operation-k7pqw -oyaml 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: fcbf90656626745a5db157954383d1d00b58f7136cd5c1bb8928e6bf4ff59b98
    cni.projectcalico.org/podIP: ""
    cni.projectcalico.org/podIPs: ""
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "k8s-pod-network",
          "ips": [
              "10.52.0.77"
          ],
          "default": true,
          "dns": {}
      }]
    pod-impersonation.cattle.io/cluster-role: pod-impersonation-helm-op-gdwjf
  creationTimestamp: "2024-07-17T09:10:29Z"
  generateName: helm-operation-
  labels:
    pod-impersonation.cattle.io/token: qp2lk4qz28jsj6xfpj2dcxlcjl5qbw4w6dzpvrv49cvsf7cwhzttb9
  name: helm-operation-k7pqw
  namespace: cattle-system
  ownerReferences:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    name: pod-impersonation-helm-op-gdwjf
    uid: ac17afbf-da41-49f2-954c-f4c77aba281d
  resourceVersion: "7914"
  uid: 5f8231e5-5175-4f73-aa72-31fa60126df7
spec:
  automountServiceAccountToken: false
  containers:
  - command:
    - helm-cmd
    env:
    - name: KUBECONFIG
      value: /home/shell/.kube/config
    image: rancher/shell:v0.1.23
    imagePullPolicy: IfNotPresent
    name: helm
    resources: {}
    stdin: true
    stdinOnce: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /home/shell/helm
      name: data
      readOnly: true
    - mountPath: /home/shell/.kube/config
      name: user-kubeconfig
      readOnly: true
      subPath: config
    workingDir: /home/shell/helm
  - command:
    - sh
    - -c
    - kubectl proxy --disable-filter || true
    env:
    - name: KUBECONFIG
      value: /root/.kube/config
    image: rancher/shell:v0.1.23
    imagePullPolicy: IfNotPresent
    name: proxy
    resources: {}
    securityContext:
      readOnlyRootFilesystem: true
      runAsGroup: 0
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /root/.kube/config
      name: admin-kubeconfig
      readOnly: true
      subPath: config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: pod-impersonation-helm-op-shbkn-token
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: harv21
  nodeSelector:
    kubernetes.io/os: linux
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoSchedule
    key: cattle.io/os
    operator: Equal
    value: linux
  - effect: NoSchedule
    key: node-role.kubernetes.io/controlplane
    operator: Equal
    value: "true"
  - effect: NoExecute
    key: node-role.kubernetes.io/etcd
    operator: Equal
    value: "true"
  - effect: NoSchedule
    key: node.cloudprovider.kubernetes.io/uninitialized
    operator: Equal
    value: "true"
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    secret:
      defaultMode: 420
      secretName: helm-operation-gngkt
  - configMap:
      defaultMode: 420
      name: impersonation-helm-op-admin-kubeconfig-b8p8d
    name: admin-kubeconfig
  - configMap:
      defaultMode: 420
      name: impersonation-helm-op-user-kubeconfig-m69q2
    name: user-kubeconfig
  - name: pod-impersonation-helm-op-shbkn-token
    secret:
      defaultMode: 420
      secretName: pod-impersonation-helm-op-shbkn-token
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-07-17T09:10:29Z"
    reason: PodCompleted
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-07-17T09:10:39Z"
    reason: PodCompleted
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-07-17T09:10:39Z"
    reason: PodCompleted
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-07-17T09:10:29Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://a502e1348fac6d20c23de63feb0efa3cc19979bc0a614eae19079cce08f65ce0
    image: docker.io/rancher/shell:v0.1.23
    imageID: docker.io/rancher/shell@sha256:5f8ed4770080893bd7a03d8627acfa476d7c4b8d10f533bdf32cfaf059bc85a4
    lastState: {}
    name: helm
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://a502e1348fac6d20c23de63feb0efa3cc19979bc0a614eae19079cce08f65ce0
        exitCode: 0
        finishedAt: "2024-07-17T09:10:38Z"
        reason: Completed
        startedAt: "2024-07-17T09:10:32Z"
  - containerID: containerd://ffed02b97a53c206192371657585e445256879fc7bf18e88c56c93dae263b316
    image: docker.io/rancher/shell:v0.1.23
    imageID: docker.io/rancher/shell@sha256:5f8ed4770080893bd7a03d8627acfa476d7c4b8d10f533bdf32cfaf059bc85a4
    lastState: {}
    name: proxy
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://ffed02b97a53c206192371657585e445256879fc7bf18e88c56c93dae263b316
        exitCode: 0
        finishedAt: "2024-07-17T09:10:39Z"
        reason: Completed
        startedAt: "2024-07-17T09:10:32Z"
  hostIP: 192.168.122.131
  phase: Succeeded
  podIP: 10.52.0.77
  podIPs:
  - ip: 10.52.0.77
  qosClass: BestEffort
  startTime: "2024-07-17T09:10:29Z"

…up4.4.0 Signed-off-by: Jian Wang <jian.wang@suse.com>

w13915984028 · 2024-07-25T09:23:10Z

@ibrokethecloud @tserong
The PR has been rebased and shell image was updated to v0.1.26; with a local test, everything works as expected, the PR is safe to go, thanks.

Below are some pods listed from the newly installed cluster:

docker.io/rancher/shell                                                     v0.1.23                                     cf4efe61147d5       396MB
docker.io/rancher/shell                                                     v0.1.26                                     33dc949c57f81       304MB

docker.io/rancher/mirrored-banzaicloud-fluentd                              v1.14.6-alpine-5                            9aae16d37878a       221MB
docker.io/rancher/mirrored-fluent-fluent-bit                                2.2.0                                       cd5b76149224a       87.4MB

docker.io/rancher/mirrored-kube-logging-logging-operator                    4.4.0                                       aceed871957bf       61.3MB

ibrokethecloud

lgtm. thanks.

mallardduck · 2024-07-31T19:55:30Z

From my understanding Harvester considers the last 2 minors as the maintained versions; so currently 1.3 and 1.2 - but when 1.4.0 releases it will be 1.3 and 1.4. If this is correct I am wondering if we can backport (at least just the shell changes) to Harvester 1.3 as well for security benefits?

This same version of Shell can be used in both Rancher 2.7 and 2.8 versions, so if the Harvester 1.3 branch uses similar k8s/rancher versions, then this can be safe there too.

Additionally, if the version of shell that Harvester embedded Rancher uses could be updated too that would be good. Not sure if there's a mechanism on the installer to do that, but changing the shell version that rancher uses can be done via:

kubectl patch settings.management.cattle.io shell-image --type merge -p '{"value":"rancher/shell:v0.1.26"}'

bk201 · 2024-08-07T04:28:36Z

@w13915984028 Please help backport to v1.3. Do we need to handle the upgrade path?

w13915984028 · 2024-08-07T05:30:30Z

@bk201 The upgrade PR was also ready harvester/harvester#6187 .

I will check whether to bump the whole chart or only the shell version to v1.3. thanks.

w13915984028 · 2024-08-07T06:00:36Z

cc @bk201 @mallardduck

Logged issue harvester/harvester#6283 and PR #790 (maybe more) for v1.3.2

The PR #790 is to bump shell version to v0.1.26.

How to patch Rancher shell, I will check

kubectl patch settings.management.cattle.io shell-image --type merge -p '{"value":"rancher/shell:v0.1.26"}'

w13915984028 changed the title ~~bump rancher-monitoring to 103.1.1+up45.1.1~~ bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 Jul 12, 2024

w13915984028 mentioned this pull request Jul 17, 2024

Bump rancher-monitoring and rancher-logging chart version harvester/addons#5

Merged

2 tasks

w13915984028 force-pushed the bumpmon branch from 230965a to 2d1abd7 Compare July 17, 2024 07:56

w13915984028 requested review from tserong and ibrokethecloud July 17, 2024 08:03

This was referenced Jul 17, 2024

Upgrade processing of monitoring and logging harvester/harvester#6187

Merged

[ENHANCEMENT v1.4.0] Bump rancher-monitoring version to 103.1.1+up45.31.1, rancher-logging to 103.1.0+up4.4.0 harvester/harvester#6166

Open

w13915984028 force-pushed the bumpmon branch 2 times, most recently from 4dd33d5 to c92c922 Compare July 25, 2024 07:45

Bump rancher-monitoring to 103.1.1+up45.1.1, bump logging to 103.1.0+…

e45e1a7

…up4.4.0 Signed-off-by: Jian Wang <jian.wang@suse.com>

w13915984028 force-pushed the bumpmon branch from c92c922 to e45e1a7 Compare July 25, 2024 07:47

ibrokethecloud approved these changes Jul 25, 2024

View reviewed changes

w13915984028 requested a review from bk201 July 25, 2024 13:58

tserong approved these changes Jul 26, 2024

View reviewed changes

bk201 merged commit 73f24af into harvester:master Jul 26, 2024
6 checks passed

w13915984028 mentioned this pull request Jul 26, 2024

bump monitoring charts shell version #780

Closed

mergify bot mentioned this pull request Aug 2, 2024

Bump rancher-monitoring and rancher-logging chart version (backport #5) harvester/addons#11

Merged

2 tasks

starbops mentioned this pull request Aug 6, 2024

[BUG] UpgradeLog can't be ready cause of fluentd PVC doesn't give access mode harvester/harvester#6272

Closed

w13915984028 mentioned this pull request Aug 7, 2024

[ENHANCEMENT] [v1.3.2] minor updates harvester/harvester#6283

Closed

3 tasks

w13915984028 mentioned this pull request Aug 7, 2024

Patch shell version to v0.1.26 #790

Merged

mergify bot mentioned this pull request Aug 9, 2024

Upgrade processing of monitoring and logging (backport #6187) harvester/harvester#6308

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 #766

bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 #766

w13915984028 commented Jul 12, 2024 •

edited

Loading

w13915984028 commented Jul 17, 2024

mergify bot commented Jul 17, 2024 •

edited

Loading

w13915984028 commented Jul 17, 2024

w13915984028 commented Jul 25, 2024 •

edited

Loading

ibrokethecloud left a comment

mallardduck commented Jul 31, 2024

bk201 commented Aug 7, 2024 •

edited

Loading

w13915984028 commented Aug 7, 2024

w13915984028 commented Aug 7, 2024

bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 #766

bump rancher-monitoring to 103.1.1+up45.1.1 rancher-logging to 103.1.0+up4.4.0 #766

Conversation

w13915984028 commented Jul 12, 2024 • edited Loading

w13915984028 commented Jul 17, 2024

mergify bot commented Jul 17, 2024 • edited Loading

✅ Backports have been created

w13915984028 commented Jul 17, 2024

w13915984028 commented Jul 25, 2024 • edited Loading

ibrokethecloud left a comment

Choose a reason for hiding this comment

mallardduck commented Jul 31, 2024

bk201 commented Aug 7, 2024 • edited Loading

w13915984028 commented Aug 7, 2024

w13915984028 commented Aug 7, 2024

w13915984028 commented Jul 12, 2024 •

edited

Loading

mergify bot commented Jul 17, 2024 •

edited

Loading

w13915984028 commented Jul 25, 2024 •

edited

Loading

bk201 commented Aug 7, 2024 •

edited

Loading