Bump Kubernetes dependencies to v1.27.3 #3730

RainbowMango · 2023-06-28T08:36:49Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR updates Kubernetes dependencies to v1.27.3 as well as transitive dependencies and adoption:

Update Kubernetes dependencies to v1.27.3
Update controller-runtime to v0.15.0 (this is required during the Kubernetes dependencies update)
Update cluster-api to v1.5.0-rc.0 (this is required during the controller-runtime update)
Adopted Kubernetes changes Set singular names for core types to pass to discovery kubernetes/kubernetes#113542
Adopted controller-runtime breaking change: ⚠️ make *http.Client configurable and use/ share the same http.Client in the default configuration kubernetes-sigs/controller-runtime#2122
Adopted controller-runtime breaking change: ⚠️ Use and provide a single dynamic rest mapper kubernetes-sigs/controller-runtime#2296
Adopted controller-runtime breaking change: ⚠ Add context to EventHandler(s) kubernetes-sigs/controller-runtime#2139
Adopted controller-runtime breaking change: ⚠ Improve builder's capabilities and general UX kubernetes-sigs/controller-runtime#2135
Adopted Kubernetes breaking change: update lease controller kubernetes/kubernetes#116218
Adopted controller-runtime breaking change: ⚠️ Deprecate pkg/config/v1alpha1 types kubernetes-sigs/controller-runtime#2149
Adopted controller-runtime breaking change: ⚠ Remove DelegatedClient, move Options in client.New kubernetes-sigs/controller-runtime#2150
Adopted controller-runtime breaking change: ⚠️ Remove pkg/inject kubernetes-sigs/controller-runtime#2134
Adopted controller-runtime breaking change: ⚠️ Allow passing a custom webhook server kubernetes-sigs/controller-runtime#2293
Adopted controller-runtime breaking change: ⚠️ Add constructor for conversion webhooks, rm admission.GetDecoder() kubernetes-sigs/controller-runtime#2144
Adopted Kubernetes breaking change: Add API for watch list kubernetes/kubernetes#115402
Adopted Kubernetes breaking change: Move genericclioptions.IOStreams to genericiooptions.IOStreams kubernetes/kubernetes#117102
Adopted controller-runtime breaking change: ⚠ Update fake client deletionTimestamp behavior kubernetes-sigs/controller-runtime#2316
Adopted controller-runtime breaking change: ⚠️ Fakeclient: Fix status handling kubernetes-sigs/controller-runtime#2259

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Karmada is now built with Kubernetes v1.27.3 dependencies.

RainbowMango · 2023-06-28T08:38:54Z

Hi @lonelyCZ, need your help here.
Now I'm blocked by updating Swagger docs:

# hack/update-swagger-docs.sh 
W0628 16:27:02.787558 1826111 recommended.go:152] Neither kubeconfig is provided nor service-account is mounted, so APIPriorityAndFairness will be disabled
W0628 16:27:02.788751 1826111 authentication.go:49] Authentication is disabled
F0628 16:27:02.791130 1826111 render.go:113] unable to install api resources: unable to setup API &{[cluster.karmada.io/v1alpha1] map[v1alpha1:map[clusters:0xc000419380 clusters/proxy:0x3d66230 clusters/status:0xc000669e00]] v1 <nil>r
exit status 255

I'm looking into it, but don't have a clue yet.

lonelyCZ · 2023-06-28T08:53:32Z

Hi @lonelyCZ, need your help here. Now I'm blocked by updating Swagger docs:

# hack/update-swagger-docs.sh 
W0628 16:27:02.787558 1826111 recommended.go:152] Neither kubeconfig is provided nor service-account is mounted, so APIPriorityAndFairness will be disabled
W0628 16:27:02.788751 1826111 authentication.go:49] Authentication is disabled
F0628 16:27:02.791130 1826111 render.go:113] unable to install api resources: unable to setup API &{[cluster.karmada.io/v1alpha1] map[v1alpha1:map[clusters:0xc000419380 clusters/proxy:0x3d66230 clusters/status:0xc000669e00]] v1 <nil>r
exit status 255

I'm looking into it, but don't have a clue yet.

I will look it today!

lonelyCZ · 2023-06-28T18:21:34Z

I just inspected it, and I found this commit at kubernetes codebase .

We can resolve it by implementing this interface. Like here cc7494c

The testing result is that

RainbowMango · 2023-06-29T01:58:59Z

Thank you so much for working on this issue late into the night! Take care of yourself.
I followed your advice and it's working now. Thank you.

RainbowMango · 2023-07-15T10:22:28Z

The next breaking change we need to adopt is kubernetes-sigs/controller-runtime#2296 which removed the WithCustomMapper option that relied at

karmada/pkg/util/restmapper/restmapper.go

Line 92 in 9a03a10

option := apiutil.WithCustomMapper(func() (meta.RESTMapper, error) {

-bash-5.0# make karmada-controller-manager
BUILD_PLATFORMS=linux/amd64 hack/build.sh karmada-controller-manager
!!! Building karmada-controller-manager for linux/amd64:
+ CGO_ENABLED=0
+ GOOS=linux
+ GOARCH=amd64
+ go build -ldflags '-X github.com/karmada-io/karmada/pkg/version.gitVersion=v1.7.0-alpha.2-43-g3a07a1ed -X github.com/karmada-io/karmada/pkg/version.gitCommit=3a07a1edbb60728755afde7a55f835ae62d9324d -X github.com/karmada-io/karmadar
# github.com/karmada-io/karmada/pkg/util/restmapper
pkg/util/restmapper/restmapper.go:92:20: undefined: apiutil.WithCustomMapper
make: *** [Makefile:43: karmada-controller-manager] Error 1

References:

the reason why introduced CachedRESTMapper: performance improvement for karmada-controller-manager to use DeferredDiscoveryRESTMapper #2105

RainbowMango · 2023-07-26T06:55:25Z

Current blocker is that karmada-metrics-adapter failed to start:

-bash-5.0# kubectl logs -n karmada-system karmada-metrics-adapter-6474fb596-2ppkh 
I0726 06:46:58.638131       1 options.go:97] karmada-metrics-adapter version: version.Info{GitVersion:"v1.7.0-alpha.2-85-g4184b562", GitCommit:"4184b562b4b7f80e1484389608941f534a8c8f03", GitTreeState:"clean", BuildDate:"2023-07-26T06:41:19Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
I0726 06:46:58.869444       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
E0726 06:46:59.600734       1 options.go:88] Unable to install resource metrics adapter: unable to get openapi models: cannot find model definition for k8s.io/metrics/pkg/apis/metrics/v1beta1.NodeMetrics. If you added a new type, you may need to add +k8s:openapi-gen=true to the package or type and run code-gen again
E0726 06:46:59.600766       1 run.go:74] "command failed" err="unable to get openapi models: cannot find model definition for k8s.io/metrics/pkg/apis/metrics/v1beta1.NodeMetrics. If you added a new type, you may need to add +k8s:openapi-gen=true to the package or type and run code-gen again"

edit: Fixed.

RainbowMango · 2023-07-28T01:41:01Z

Hi @ikaven1024 Need your help here.

The TestMultiClusterCache_Watch and TestMultiClusterCache_Watch_Namespaced is failing after updating the Kubernetes dependencies, note that, the E2E tests work fine.

# go test ./pkg/search/proxy/store/... -run=TestMultiClusterCache_Watch 
I0728 09:35:19.163117 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster2
I0728 09:35:19.163160 1701082 cluster_cache.go:70] Add cache for cluster2 /v1, Resource=pods
I0728 09:35:19.163478 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster1
I0728 09:35:19.163504 1701082 cluster_cache.go:70] Add cache for cluster1 /v1, Resource=pods
I0728 09:35:27.165983 1701082 resource_cache.go:36] Stop store for cluster2 /v1, Resource=pods
I0728 09:35:27.166022 1701082 resource_cache.go:36] Stop store for cluster1 /v1, Resource=pods
--- FAIL: TestMultiClusterCache_Watch (8.00s)
    --- FAIL: TestMultiClusterCache_Watch/resource_version_is_empty (5.00s)
        multi_cluster_cache_test.go:994: timeout
    --- FAIL: TestMultiClusterCache_Watch/resource_version_of_cluster2_is_empty (1.00s)
        multi_cluster_cache_test.go:1000: Watch() got = map[pod13:{}], but want = map[pod13:{} pod21:{} pod22:{} pod23:{}]
I0728 09:35:27.166242 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster1
I0728 09:35:27.166269 1701082 cluster_cache.go:70] Add cache for cluster1 /v1, Resource=pods
I0728 09:35:27.166344 1701082 multi_cluster_cache.go:84] Add cache for cluster cluster2
I0728 09:35:27.166375 1701082 cluster_cache.go:70] Add cache for cluster2 /v1, Resource=pods
I0728 09:35:38.169094 1701082 resource_cache.go:36] Stop store for cluster1 /v1, Resource=pods
I0728 09:35:38.169118 1701082 resource_cache.go:36] Stop store for cluster2 /v1, Resource=pods
--- FAIL: TestMultiClusterCache_Watch_Namespaced (11.00s)
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_all_namespaces (5.00s)
        multi_cluster_cache_test.go:1127: timeout
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_ns1_namespace (5.00s)
        multi_cluster_cache_test.go:1127: timeout
    --- FAIL: TestMultiClusterCache_Watch_Namespaced/request_ns2_namespace (0.00s)
        multi_cluster_cache_test.go:1132: Watch() got = map[], but want = map[pod22:{}]
FAIL
FAIL	github.com/karmada-io/karmada/pkg/search/proxy/store	19.026s
FAIL

Update controller-runtime to v0.15.0 Update cluster-api to v1.5.0-rc.0 Update sigs.k8s.io/metrics-server to v0.6.1-0.20230706083104-796fc0f832c1 Update sigs.k8s.io/custom-metrics-apiserver to v1.27.0 Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>

…ames for core types Adopt breaking change of contriller-runtime PR 2122 Adopt breaking change of contriller-runtime PR 2296 Adopt breaking change of controller-runtime PR 2139 Adopt breaking change of controller-runtime PR 2135 Adopt breaking change of Kubernetes PR 116218 Adopt breaking change of controller-runtime PR 2149 Adopt breaking change of controller-runtime PR 2150 Adopt breaking change of controller-runtime PR 2134 Adopt breaking change of controller-runtime PR 2293 Adopt breaking change of controller-runtime PR 2144 Adopt k8s feature: watch list. PR 115402 Adopt breaking change of Kubernetes PR 117102 Adopt Kubernetes change: disable staticcheck Fix unit tests that failure due to fake-client behavior changes. Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>

Update auto-generated files Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>

codecov-commenter · 2023-07-29T09:06:39Z

Codecov Report

Merging #3730 (3f5c907) into master (04d2ef1) will decrease coverage by 0.26%.
Report is 2 commits behind head on master.
The diff coverage is 15.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #3730      +/-   ##
==========================================
- Coverage   55.18%   54.93%   -0.26%     
==========================================
  Files         227      227              
  Lines       21671    21676       +5     
==========================================
- Hits        11960    11907      -53     
- Misses       9074     9130      +56     
- Partials      637      639       +2

Flag	Coverage Δ
unittests	`54.93% <15.00%> (-0.26%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ers/binding/cluster_resource_binding_controller.go	`0.00% <0.00%> (ø)`
pkg/controllers/status/crb_status_controller.go	`48.33% <0.00%> (-5.00%)`	⬇️
pkg/controllers/status/rb_status_controller.go	`46.77% <0.00%> (-4.84%)`	⬇️
...controllers/unifiedauth/unified_auth_controller.go	`25.11% <0.00%> (ø)`
...ependenciesdistributor/dependencies_distributor.go	`3.02% <ø> (ø)`
pkg/registry/cluster/storage/storage.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/cache.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/proxy.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/resourceregistry.go	`0.00% <0.00%> (ø)`
pkg/registry/search/storage/search.go	`0.00% <0.00%> (ø)`
... and 9 more

... and 3 files with indirect coverage changes

RainbowMango · 2023-07-29T09:57:46Z

Great thanks to @lonelyCZ @ikaven1024 @XiShanYongYe-Chang @jwcesign @zishen @Poor12 !
I couldn't deal with so many breaking changes without your help!

XiShanYongYe-Chang

/lgtm

karmada-bot · 2023-07-31T04:29:22Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Jun 28, 2023

karmada-bot requested review from Garrybest, lonelyCZ and pidb June 28, 2023 08:37

karmada-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 28, 2023

RainbowMango force-pushed the pr_update_k8s_v127 branch from c016ba3 to afb1a69 Compare June 29, 2023 01:50

RainbowMango force-pushed the pr_update_k8s_v127 branch from afb1a69 to 6f4f3b4 Compare July 15, 2023 09:09

RainbowMango force-pushed the pr_update_k8s_v127 branch from 3a07a1e to 2d7e2f9 Compare July 17, 2023 08:34

RainbowMango mentioned this pull request Jul 20, 2023

build cached restmapper based on Kubernetes restmapper #3819

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch 2 times, most recently from 9977684 to 453fdab Compare July 24, 2023 08:23

RainbowMango mentioned this pull request Jul 24, 2023

Make some cleanup in dependencies distributor #3827

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch 3 times, most recently from 139af7a to 2ff237c Compare July 25, 2023 09:02

RainbowMango mentioned this pull request Jul 25, 2023

Cleanup deprecated methods in wait package #3835

Closed

4 tasks

RainbowMango force-pushed the pr_update_k8s_v127 branch from 6dfab1f to 4184b56 Compare July 25, 2023 11:10

RainbowMango force-pushed the pr_update_k8s_v127 branch from 4184b56 to 1150e36 Compare July 27, 2023 02:45

ikaven1024 mentioned this pull request Jul 29, 2023

fix ut in TestMultiClusterCache watch #3862

Merged

RainbowMango force-pushed the pr_update_k8s_v127 branch from 1150e36 to 01e788c Compare July 29, 2023 07:15

Update vendor

3f5c907

Update auto-generated files Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>

RainbowMango force-pushed the pr_update_k8s_v127 branch from 01e788c to 3f5c907 Compare July 29, 2023 08:48

RainbowMango marked this pull request as ready for review July 29, 2023 09:08

karmada-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2023

XiShanYongYe-Chang reviewed Jul 31, 2023

View reviewed changes

karmada-bot assigned XiShanYongYe-Chang Jul 31, 2023

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2023

RainbowMango added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2023

karmada-bot merged commit 7381933 into karmada-io:master Jul 31, 2023
12 checks passed

RainbowMango mentioned this pull request Aug 8, 2023

Fix SSA being silently disabled by lack of OpenAPI config kubernetes/kubernetes#114998

Merged

This was referenced Aug 16, 2023

repair the cluster_status_controller_test #3948

Merged

Test ci zishen/karmada#1

Closed

RainbowMango added this to the v1.7 milestone Oct 21, 2023

RainbowMango mentioned this pull request Oct 21, 2023

Cherrypick #4144: fix cannot access ResourceRegistry with its singularName #4152

Closed

3 tasks

RainbowMango deleted the pr_update_k8s_v127 branch March 30, 2024 08:20

zhzhuang-zju mentioned this pull request May 16, 2024

migrate wait.PollUntil to wait.PollUntilContextCancel #4947

Merged

RainbowMango mentioned this pull request May 28, 2024

Enable deprecation check and bump golangci-lint to v1.59.0 #4992

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump Kubernetes dependencies to v1.27.3 #3730

Bump Kubernetes dependencies to v1.27.3 #3730

RainbowMango commented Jun 28, 2023 •

edited

Loading

RainbowMango commented Jun 28, 2023

lonelyCZ commented Jun 28, 2023

lonelyCZ commented Jun 28, 2023

RainbowMango commented Jun 29, 2023

RainbowMango commented Jul 15, 2023 •

edited

Loading

RainbowMango commented Jul 26, 2023 •

edited

Loading

RainbowMango commented Jul 28, 2023

codecov-commenter commented Jul 29, 2023

RainbowMango commented Jul 29, 2023

XiShanYongYe-Chang left a comment

karmada-bot commented Jul 31, 2023

Bump Kubernetes dependencies to v1.27.3 #3730

Bump Kubernetes dependencies to v1.27.3 #3730

Conversation

RainbowMango commented Jun 28, 2023 • edited Loading

RainbowMango commented Jun 28, 2023

lonelyCZ commented Jun 28, 2023

lonelyCZ commented Jun 28, 2023

RainbowMango commented Jun 29, 2023

RainbowMango commented Jul 15, 2023 • edited Loading

RainbowMango commented Jul 26, 2023 • edited Loading

RainbowMango commented Jul 28, 2023

codecov-commenter commented Jul 29, 2023

Codecov Report

RainbowMango commented Jul 29, 2023

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

karmada-bot commented Jul 31, 2023

RainbowMango commented Jun 28, 2023 •

edited

Loading

RainbowMango commented Jul 15, 2023 •

edited

Loading

RainbowMango commented Jul 26, 2023 •

edited

Loading