Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource schema #139

Merged
merged 4 commits into from
Feb 28, 2024
Merged

Resource schema #139

merged 4 commits into from
Feb 28, 2024

Conversation

MbolotSuse
Copy link
Contributor

@MbolotSuse MbolotSuse commented Dec 15, 2023

Overview

Related to rancher/rancher#41301. Also see the previous pr #62 for some earlier discussion and context.

Currently, the steve schemas endpoint produces an extremely large amount of data. Since the UI needs this endpoint for a variety of reasons and frequently fetches the entirety of it, this produces a large performance impact where pages will need to wait a significant amount of time, purely for the schemas to load.

This PR reduces the amount of schemas, and causes (most of) the schemas to have no resourceField. In addition, it introduces a new schemaDefinitions endpoint which provides all fields/types/descriptions for a single schema.

Solution Detail

  • Changes the schemas to be only top-level resources. For example, standalone resources in k8s like a Pod would be included, but there is no longer a schema for the PodSpec.
  • Removes the resourceFields from the schemas.
  • Introduces a new schemaDefinitions endpoint which allows users to specify a single schema (i.e. Pod) and provides the full resource fields for that schema.
    • It does not support listing one or more schemas (i.e. /v1/schemaDefinitions will return an error). This was done to encourage use of only a specific resource definition at a time - the full openapi schema for k8s is still quite large, and this could cause large performance drops if a list was supported.
    • This provides a fully recursive definition. For example, since Pod includes PodSpec, this endpoint will return the top-level fields for both Pod and PodSpec.
    • This endpoint aims to use the same access control as the schemas endpoint. In short, if you can see the schema for an object, you can get it's schemaDefinition as well.
  • Reduces the openapiv2 part of the schema retrieval to only add description (previously this was used for resource fields).
  • Removes the adding of resource fields from the CRD part of schema processing.
  • Adds a new schemaFieldVisitor which translates from an openapi schema to a schema definition.

Notes

  • There are some schemas which don't represent resources, but are rather custom steve types. These will still have resourceFields.

@MbolotSuse MbolotSuse force-pushed the resource-schema branch 2 times, most recently from 3cf0900 to e683831 Compare December 15, 2023 21:22
pkg/schema/converter/description.go Outdated Show resolved Hide resolved
pkg/schema/converter/crd.go Show resolved Hide resolved
pkg/schema/converter/crd.go Show resolved Hide resolved
pkg/schema/converter/description.go Outdated Show resolved Hide resolved
pkg/schema/converter/description.go Outdated Show resolved Hide resolved
pkg/schema/definitions/handler.go Outdated Show resolved Hide resolved
pkg/schema/definitions/handler.go Outdated Show resolved Hide resolved
pkg/schema/definitions/schema.go Outdated Show resolved Hide resolved
pkg/schema/definitions/visitor.go Outdated Show resolved Hide resolved
pkg/schema/definitions/visitor.go Outdated Show resolved Hide resolved
@MbolotSuse MbolotSuse requested review from moio and rmweir January 10, 2024 14:04
@MbolotSuse MbolotSuse force-pushed the resource-schema branch 2 times, most recently from 108d104 to d121dfe Compare January 10, 2024 20:43
Copy link
Contributor

@moio moio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @MbolotSuse, tons of great work here - I feel confident this will help significantly in the performance of cases we've observed.

I put a couple hours understanding context and code - I admit this is not a very deep review, as I lacked and still lack a lot of insight. Hopefully comments will still help in some way.

Looking forward to see this tested in the UI!

pkg/schema/converter/discovery.go Outdated Show resolved Hide resolved
pkg/schema/converter/k8stonorman.go Outdated Show resolved Hide resolved
pkg/schema/converter/crd.go Show resolved Hide resolved
pkg/schema/definitions/handler.go Show resolved Hide resolved
pkg/schema/definitions/schema.go Show resolved Hide resolved
pkg/schema/definitions/handler.go Show resolved Hide resolved
pkg/schema/definitions/handler.go Show resolved Hide resolved
pkg/schema/definitions/visitor.go Outdated Show resolved Hide resolved
pkg/schema/definitions/visitor.go Outdated Show resolved Hide resolved
pkg/schema/definitions/visitor_test.go Show resolved Hide resolved
@MbolotSuse MbolotSuse force-pushed the resource-schema branch 2 times, most recently from 697db4f to 0e49474 Compare January 19, 2024 14:45
@MbolotSuse MbolotSuse requested a review from moio January 19, 2024 22:34
pkg/schema/definitions/schema.go Show resolved Hide resolved
pkg/schema/definitions/schema.go Show resolved Hide resolved
pkg/schema/definitions/openapi_test.go Show resolved Hide resolved
pkg/schema/definitions/handler_test.go Show resolved Hide resolved

addBaseSchema("management.cattle.io.globalrole", "management.cattle.io.missingfrommodel")

tests := []struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could have a couple more tests wrt. resource version:

  • resource.Version == ""
  • resource.Version == group.Version
  • resource.Version != group.Version

to make sure we properly handle these cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can look into adding this as a part of MbolotSuse#1

pkg/schema/converter/crd.go Show resolved Hide resolved
Changes steve to only return top level schema definitions rather
than a full defined schema for every field. Also adds basic docs
and fixes a bug with schema generation on versioned crds
Introduces new schema definitions endpoint, allowing the caller to get
the fields/types/descriptions for a given kubernetes resource.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants