rebase: update ceph to reef #4030

Madhu-1 · 2023-08-02T14:37:58Z

Testing Reef and seeing everything works.

Madhu-1 · 2023-08-02T14:46:43Z

/test ci/centos/mini-e2e/k8s-1.27

nixpanic · 2023-08-03T07:21:54Z

/test ci/centos/mini-e2e/k8s-1.27

nixpanic · 2023-08-03T08:28:41Z

The CI job tries to pull this:

Trying to pull registry-ceph-csi.apps.ocp.cloud.ci.centos.org/ceph/ceph:v18...

The mirror contains this image:

going to copy docker://quay.io/ceph/ceph:v18 to docker://registry-ceph-csi.apps.ocp.cloud.ci.centos.org/quay.io/ceph/ceph:v18
Getting image source signatures
Copying blob sha256:66286c37fba202f94836d18ed3ea1efa02816e6db632ec78297328694bed0b94
Copying blob sha256:a20960ea1b02448618d7cdee5e657e6b83404689ca4ce4555251aeada11b5a7a
Copying config sha256:ff9c8d7de4ca37ca64a34818fa8d6e7b4e6be435e4f8f54ed2e00827f8fa3d31
Writing manifest to image destination
Storing signatures

The mirror contains quay.io in the image name, but it seems to have been stripped by the CI job.

PR #4031 should probably not include the quay.io/ part in the 2nd column with "short names".

Updates: ceph#4030 Signed-off-by: Niels de Vos <ndevos@ibm.com>

CI jobs pull the ceph/ceph:v18 image (without `quay.io`). If the mirroring includes the registry host, the image can not be found. Updates: ceph#4030 Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic · 2023-08-03T08:47:58Z

#4032 has been created, the mirroring runs nightly, so after merging the PR, a new /test ... can be done to see if the Ceph Reef image can be used in the CI job.

Madhu-1 · 2023-08-04T06:29:44Z

/test ci/centos/mini-e2e/k8s-1.27

CI jobs pull the ceph/ceph:v18 image (without `quay.io`). If the mirroring includes the registry host, the image can not be found. Updates: #4030 Signed-off-by: Niels de Vos <ndevos@ibm.com>

Madhu-1 · 2023-08-07T07:36:29Z

/test ci/centos/mini-e2e/k8s-1.27

Madhu-1 · 2023-08-07T07:59:06Z

  Aug  7 07:49:45.556: INFO: ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-757999d6c7-c4v5d/exec?command=%2Fbin%2Fsh&command=-c&command=ceph+osd+pool+create+.nfs+128&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true)

  Aug  7 07:49:46.063: INFO: stdErr occurred: pool names beginning with . are not allowed



  Aug  7 07:49:46.068: INFO: ExecWithOptions {Command:[/bin/sh -c ceph osd pool set .nfs size 1 --yes-i-really-mean-it] Namespace:rook-ceph PodName:rook-ceph-tools-757999d6c7-c4v5d ContainerName:rook-ceph-tools Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}

  Aug  7 07:49:46.068: INFO: >>> kubeConfig: /root/.kube/config

  Aug  7 07:49:46.068: INFO: ExecWithOptions: Clientset creation

  Aug  7 07:49:46.068: INFO: ExecWithOptions: execute(POST https://192.168.49.2:8443/api/v1/namespaces/rook-ceph/pods/rook-ceph-tools-757999d6c7-c4v5d/exec?command=%2Fbin%2Fsh&command=-c&command=ceph+osd+pool+set+.nfs+size+1+--yes-i-really-mean-it&container=rook-ceph-tools&container=rook-ceph-tools&stderr=true&stdout=true)

  Aug  7 07:49:46.572: INFO: failed to execute command: command terminated with exit code 2

  Aug  7 07:49:46.572: INFO: stdErr occurred: Error ENOENT: unrecognized pool '.nfs'

@nixpanic NFS pool creation failing, anything changed in Ceph Reef version?

Madhu-1 · 2023-08-07T08:10:34Z

https://docs.ceph.com/en/reef/rados/operations/pools/#pool-names mentions its reserved and we should not use it

Madhu-1 · 2023-08-07T08:17:39Z

/test ci/centos/mini-e2e/k8s-1.27

nixpanic · 2023-08-07T08:34:46Z

@nixpanic NFS pool creation failing, anything changed in Ceph Reef version?

It seems like Reef has some changes with respect to pool creation... Rook usually creates the .nfs pool, I think. But it requires redundancy. I do not know how Rook creates the .nfs pool in Reef, maybe we need to pass an extra option to the ceph osd command?

https://github.com/rook/rook/blob/7aaf46e11243b21df4d4f088fc80b949019559b0/pkg/operator/ceph/nfs/nfs.go#L319

Madhu-1 · 2023-08-07T09:54:30Z

@nixpanic NFS pool creation failing, anything changed in Ceph Reef version?

It seems like Reef has some changes with respect to pool creation... Rook usually creates the .nfs pool, I think. But it requires redundancy. I do not know how Rook creates the .nfs pool in Reef, maybe we need to pass an extra option to the ceph osd command?

https://github.com/rook/rook/blob/7aaf46e11243b21df4d4f088fc80b949019559b0/pkg/operator/ceph/nfs/nfs.go#L319

@nixpanic Yes you are correct we were missing extra flag to create pool in cephcsi e2e, added missing flag now.

Madhu-1 · 2023-08-07T09:56:00Z

/test ci/centos/mini-e2e/k8s-1.27

Madhu-1 · 2023-08-08T07:58:38Z

#7 45.98   Running scriptlet: nfs-ganesha-5.5-1.el8s.x86_64                        11/12 
#7 45.99 error: nfs-ganesha-selinux-5.4-1.el8s.noarch: erase skipped
#7 45.99 /var/tmp/rpm-tmp.FjPzqW: line 2: /etc/selinux/config: No such file or directory
#7 45.99 warning: %posttrans(nfs-ganesha-5.5-1.el8s.x86_64) scriptlet failed, exit status 1
#7 45.99 
#7 45.99 Error in POSTTRANS scriptlet in rpm package nfs-ganesha
#7 45.99   Running scriptlet: nfs-ganesha-5.4-1.el8s.x86_64                        11/12 
#7 46.11   Verifying        : nfs-ganesha-5.5-1.el8s.x86_64                         1/12 
#7 46.11   Verifying        : nfs-ganesha-5.4-1.el8s.x86_64                         2/12 
#7 46.11   Verifying        : nfs-ganesha-ceph-5.5-1.el8s.x86_64                    3/12 
#7 46.11   Verifying        : nfs-ganesha-ceph-5.4-1.el8s.x86_64                    4/12 
#7 46.11   Verifying        : nfs-ganesha-rados-grace-5.5-1.el8s.x86_64             5/12 
#7 46.11   Verifying        : nfs-ganesha-rados-grace-5.4-1.el8s.x86_64             6/12 
#7 46.11   Verifying        : nfs-ganesha-rados-urls-5.5-1.el8s.x86_64              7/12 
#7 46.11   Verifying        : nfs-ganesha-rados-urls-5.4-1.el8s.x86_64              8/12 
#7 46.11   Verifying        : nfs-ganesha-rgw-5.5-1.el8s.x86_64                     9/12 
#7 46.11   Verifying        : nfs-ganesha-rgw-5.4-1.el8s.x86_64                    10/12 
#7 46.11   Verifying        : nfs-ganesha-selinux-5.5-1.el8s.noarch                11/12 
#7 46.11   Verifying        : nfs-ganesha-selinux-5.4-1.el8s.noarch                12/12Error: Transaction failed
#7 46.51  
#7 46.51 
#7 46.51 Upgraded:
#7 46.51   nfs-ganesha-5.5-1.el8s.x86_64                                                 
#7 46.51   nfs-ganesha-ceph-5.5-1.el8s.x86_64                                            
#7 46.51   nfs-ganesha-rados-grace-5.5-1.el8s.x86_64                                     
#7 46.51   nfs-ganesha-rados-urls-5.5-1.el8s.x86_64                                      
#7 46.51   nfs-ganesha-rgw-5.5-1.el8s.x86_64                                             
#7 46.51 Failed:
#7 46.51   nfs-ganesha-selinux-5.4-1.el8s.noarch  nfs-ganesha-selinux-5.5-1.el8s.noarch 
#7 46.51 
#7 ERROR: process "/bin/sh -c dnf -y update        && dnf clean all        && rm -rf /var/cache/yum" did not complete successfully: exit code: 1
------
 > [updated_base 3/3] RUN dnf -y update        && dnf clean all        && rm -rf /var/cache/yum:
46.51 
46.51 Upgraded:
46.51   nfs-ganesha-5.5-1.el8s.x86_64                                                 
46.51   nfs-ganesha-ceph-5.5-1.el8s.x86_64                                            
46.51   nfs-ganesha-rados-grace-5.5-1.el8s.x86_64                                     
46.51   nfs-ganesha-rados-urls-5.5-1.el8s.x86_64                                      
46.51   nfs-ganesha-rgw-5.5-1.el8s.x86_64                                             
46.51 Failed:
46.51   nfs-ganesha-selinux-5.4-1.el8s.noarch  nfs-ganesha-selinux-5.5-1.el8s.noarch 
46.51

it looks like ceph v18 image is broken today and not able to build cephcsi image like yesterday 🤕

Madhu-1 · 2023-08-15T14:12:04Z

/test ci/centos/mini-e2e/k8s-1.27

Madhu-1 · 2023-08-21T10:37:10Z

/test ci/centos/mini-e2e/k8s-1.27

Madhu-1 · 2023-08-21T10:50:00Z

/test ci/centos/mini-e2e/k8s-1.27

nixpanic · 2023-08-23T08:17:54Z

Both CI jobs failed with:

  [FAILED] failed to resize filesystem PVC: context deadline exceeded
  In [It] at: /go/src/github.com/ceph/ceph-csi/e2e/rbd.go:852 @ 08/23/23 08:11:07.519

This isn't a test that I have seen sporadically fail, so it may point to a change related to the Ceph version.

Madhu-1 · 2023-08-23T08:30:12Z

Both CI jobs failed with:
  [FAILED] failed to resize filesystem PVC: context deadline exceeded
  In [It] at: /go/src/github.com/ceph/ceph-csi/e2e/rbd.go:852 @ 08/23/23 08:11:07.519
This isn't a test that I have seen sporadically fail, so it may point to a change related to the Ceph version.

CI passed on 1.25,1.26. and 1.27 not sure is it something to do with ceph version, let me check on that one

nixpanic · 2023-08-24T07:12:47Z

CI passed on 1.25,1.26. and 1.27 not sure is it something to do with ceph version, let me check on that one

Other PRs seem to fail in the same way with RBD resizing on Kubernetes 1.28. It is not only this PR 😢

nixpanic · 2023-08-25T15:01:15Z

@Mergifyio rebase

Updating ceph base image to Reef as its the latest ceph release. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

update Rook to the latest v12 release. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

in recent ceph version .nfs pool creation is failing, as we are sure about creating the pools in the e2e tests, tring to create the pool with required extra agruments to make it successful. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

disable ganesha as dnf update is failing on Reef ceph version. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

As we dont need to enable nfs modules in ceph v16.2.8 onwards skipping this one. Because of this one we have a regression in nfs export Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

specify from what ceph versions we need to run nfs service commands and when to skip it. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

mergify · 2023-08-25T15:01:31Z

rebase

✅ Branch has been successfully rebased

ceph-csi-bot · 2023-08-25T15:03:33Z

/test ci/centos/k8s-e2e-external-storage/1.27

ceph-csi-bot · 2023-08-25T15:03:33Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2023-08-25T15:03:34Z

/test ci/centos/mini-e2e-helm/k8s-1.27

ceph-csi-bot · 2023-08-25T15:03:34Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2023-08-25T15:03:35Z

/test ci/centos/mini-e2e/k8s-1.27

ceph-csi-bot · 2023-08-25T15:03:36Z

/test ci/centos/k8s-e2e-external-storage/1.26

ceph-csi-bot · 2023-08-25T15:03:36Z

/test ci/centos/k8s-e2e-external-storage/1.28

ceph-csi-bot · 2023-08-25T15:03:37Z

/test ci/centos/mini-e2e-helm/k8s-1.26

ceph-csi-bot · 2023-08-25T15:03:37Z

/test ci/centos/mini-e2e-helm/k8s-1.28

ceph-csi-bot · 2023-08-25T15:03:38Z

/test ci/centos/mini-e2e/k8s-1.26

ceph-csi-bot · 2023-08-25T15:03:38Z

/test ci/centos/mini-e2e/k8s-1.28

nixpanic · 2023-08-25T15:06:20Z

/test ci/centos/mini-e2e/k8s-1.25

nixpanic · 2023-08-25T15:06:31Z

/test ci/centos/mini-e2e-helm/k8s-1.25

nixpanic · 2023-08-25T15:06:39Z

/test ci/centos/k8s-e2e-external-storage/1.25

Madhu-1 added DNM DO NOT MERGE component/testing Additional test cases or CI work labels Aug 2, 2023

mergify bot added the rebase update the version of an external component label Aug 2, 2023

nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Aug 3, 2023

ci: strip quay.io from mirrored ceph/ceph:v18 image

4e6a398

Updates: ceph#4030 Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic mentioned this pull request Aug 3, 2023

ci: strip quay.io from mirrored ceph/ceph:v18 image #4032

Merged

Madhu-1 force-pushed the test-reef branch from 042d340 to c3da1f2 Compare August 7, 2023 09:53

Madhu-1 mentioned this pull request Aug 9, 2023

dnf update in ceph v18 container image is failing ceph/ceph-container#2141

Closed

Madhu-1 force-pushed the test-reef branch from c3da1f2 to 497c043 Compare August 15, 2023 14:11

Madhu-1 force-pushed the test-reef branch from 497c043 to 7023cd7 Compare August 21, 2023 10:36

Madhu-1 force-pushed the test-reef branch from 7023cd7 to 43ae986 Compare August 21, 2023 10:49

Madhu-1 requested review from a team August 21, 2023 13:23

nixpanic mentioned this pull request Aug 24, 2023

E2E constantly failing on kube 1.28 while resizing RBD PVC #4073

Closed

Madhu-1 added 6 commits August 25, 2023 15:01

rebase: update ceph to reef

f93a7b2

Updating ceph base image to Reef as its the latest ceph release. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

rebase: update Rook to the latest release

4cec304

update Rook to the latest v12 release. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

build: disable ganesha package in Dockerfile

b06e328

disable ganesha as dnf update is failing on Reef ceph version. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

ci: skip enabling nfs modules in e2e

c9a7115

As we dont need to enable nfs modules in ceph v16.2.8 onwards skipping this one. Because of this one we have a regression in nfs export Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

doc: set ceph version for NFS service

84240b5

specify from what ceph versions we need to run nfs service commands and when to skip it. Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

nixpanic force-pushed the test-reef branch from 92aa289 to 84240b5 Compare August 25, 2023 15:01

nixpanic added the ok-to-test Label to trigger E2E tests label Aug 25, 2023

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Aug 25, 2023

mergify bot merged commit 461effe into ceph:devel Aug 25, 2023
24 checks passed

rebase: update ceph to reef #4030

rebase: update ceph to reef #4030

Conversation

Madhu-1 commented Aug 2, 2023

Madhu-1 commented Aug 2, 2023

nixpanic commented Aug 3, 2023

nixpanic commented Aug 3, 2023

nixpanic commented Aug 3, 2023

Madhu-1 commented Aug 4, 2023

Madhu-1 commented Aug 7, 2023

Madhu-1 commented Aug 7, 2023

Madhu-1 commented Aug 7, 2023

Madhu-1 commented Aug 7, 2023

nixpanic commented Aug 7, 2023 • edited Loading

Madhu-1 commented Aug 7, 2023

Madhu-1 commented Aug 7, 2023

Madhu-1 commented Aug 8, 2023

Madhu-1 commented Aug 15, 2023

Madhu-1 commented Aug 21, 2023

Madhu-1 commented Aug 21, 2023

nixpanic commented Aug 23, 2023

Madhu-1 commented Aug 23, 2023

nixpanic commented Aug 24, 2023

nixpanic commented Aug 25, 2023

mergify bot commented Aug 25, 2023

✅ Branch has been successfully rebased

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

ceph-csi-bot commented Aug 25, 2023

nixpanic commented Aug 25, 2023

nixpanic commented Aug 25, 2023

nixpanic commented Aug 25, 2023

nixpanic commented Aug 7, 2023 •

edited

Loading