Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add indexer ADR #132

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add indexer ADR #132

wants to merge 1 commit into from

Conversation

jpellizzari
Copy link
Contributor

ADR to explain the complications of labels queries

Copy link
Contributor

@enekofb enekofb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for framing the issue. added a potential suggestion to continue the discussion.

// result will have "podinfo-a", "podinfo-b", but not "podinfo-d" since it is not in namespace-a
```

This works for enumerable fields, but there may be cases where we need to query against fields for which we do not know the key/value pairs. For example, if we want to query for labels on an object we run into a couple issues:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you expand on what you mean with enumerable fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Enumerable" in this context means: "Fields we know exist ahead of time". We can predict that an object will have a namespace, but we cannot predict that it has any labels or what those label keys might be.

}
},
}
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my mind, i was actually thinking in a different direction where the object would look instead of

{
        Cluster:    "management",
        Name:       "podinfo-a",
        Namespace:  "namespace-a",
        Kind:       "HelmChart",
        APIGroup:   "apps",
        APIVersion: "v1",
        Labels: map[string]string{
            "weave.works/templateType": "application",
        }
    },

something like

{
        Cluster:    "management",
        Name:       "podinfo-a",
        Namespace:  "namespace-a",
        Kind:       "HelmChart",
        APIGroup:   "apps",
        APIVersion: "v1",
       TemplateType: "application",
    },

and the query would be

q := &query{
    filters: []string{
        "kind:/(HelmChart|HelmRepository)/",
        "templateType: application",
    },
}

to have that type of querying just via the indexer or unstructured.

do you see whether this could be feasible or what would be the major impediments for it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking is that it would be weird to have a HelmRelease with a TemplateType key, for example. The only object with a TemplateType that is actually valid is a GitOpsTemplate.

Adding all these fields to the normalized object would take quite a bit more maintenance and involvement for someone wanting to add their new kind.

May be we can add it solely to the indexed object on .Add(). I will give that a try.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i reminder for @enekofb is to refresh the picture on the indexes that we currently have ... cause that is also something that we could look at.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doing some discovery in this pr weaveworks/weave-gitops-enterprise#3537

@enekofb
Copy link
Contributor

enekofb commented Oct 20, 2023

⚠️ Disclaimer ⚠️ : I am still in the knowledge-building phase so you might find some nonsense on some of the following :)

I started to look here with the following hypothesis to test out of reading https://blevesearch.com/docs/Index-Mapping/

  1. We have more than one document mapping:
  • a default document mapping for normalized objects
  • X specific document mappings for denormalized objects
  1. our normalized object schema is more like a standard kube resources + weave gitops specific fields (like cluster, tenant, etc ..) + unstructured
  • searches: via the indexed fields
  • visualization: via the unstructured
  1. labels belong to the structured schema the same as annotations therefore to the default mapping

When do we need mapping extensions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants