-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add indexer ADR #132
base: main
Are you sure you want to change the base?
Add indexer ADR #132
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# 22. Query Service Indexer | ||
|
||
Date: 2023-08-18 | ||
|
||
## Status | ||
|
||
Draft | ||
|
||
## Context | ||
|
||
Our solution for cross-cluster querying has evolved to support some powerful querying patterns (complex AND/OR logic, full-text search), but may be deficient for the next level of capabilities that our (internal) users are asking for. | ||
|
||
At present, we can do fairly complex query logic with **normalized fields**: | ||
|
||
```golang | ||
|
||
// Given these objects | ||
objects := []models.Object{ | ||
{ | ||
Cluster: "management", | ||
Name: "podinfo-a", | ||
Namespace: "namespace-a", | ||
Kind: "HelmChart", | ||
APIGroup: "apps", | ||
APIVersion: "v1", | ||
}, | ||
{ | ||
Cluster: "management", | ||
Name: "podinfo-b", | ||
Namespace: "namespace-a", | ||
Kind: "HelmRepository", | ||
APIGroup: "apps", | ||
APIVersion: "v1", | ||
}, | ||
{ | ||
Cluster: "management", | ||
Name: "podinfo-d", | ||
Namespace: "namespace-b", | ||
Kind: "HelmRepository", | ||
APIGroup: "apps", | ||
APIVersion: "v1", | ||
}, | ||
} | ||
|
||
|
||
qs := NewQueryService(...) | ||
|
||
|
||
q := &query{filters: []string{ | ||
// Regex-style query where we can add the OR logic within a field. | ||
"kind:/(HelmChart|HelmRepository)/", | ||
// Top-level fields are AND'd together. | ||
"namespace:namespace-a", | ||
}} | ||
|
||
|
||
result := qs.RunQuery(q) | ||
// result will have "podinfo-a", "podinfo-b", but not "podinfo-d" since it is not in namespace-a | ||
``` | ||
|
||
This works for enumerable fields, but there may be cases where we need to query against fields for which we do not know the key/value pairs. For example, if we want to query for labels on an object we run into a couple issues: | ||
|
||
- We don't know the fields for a given object's labels before adding them to the indexer | ||
- Each object's labels have their own key/value pairs that are only relevant in the context of one object, or more realistically, one `kind` of object | ||
|
||
An example of two normalized objects with different label pairs: | ||
|
||
```golang | ||
objects := []models.Object{ | ||
{ | ||
Cluster: "management", | ||
Name: "podinfo-a", | ||
Namespace: "namespace-a", | ||
Kind: "HelmChart", | ||
APIGroup: "apps", | ||
APIVersion: "v1", | ||
Labels: map[string]string{ | ||
"weave.works/template": "true", | ||
} | ||
}, | ||
{ | ||
Cluster: "management", | ||
Name: "podinfo-b", | ||
Namespace: "namespace-a", | ||
Kind: "HelmRepository", | ||
APIGroup: "apps", | ||
APIVersion: "v1", | ||
Labels: map[string]string{ | ||
"other.org.com/somekey": "othervalue", | ||
} | ||
}, | ||
} | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. in my mind, i was actually thinking in a different direction where the object would look instead of {
Cluster: "management",
Name: "podinfo-a",
Namespace: "namespace-a",
Kind: "HelmChart",
APIGroup: "apps",
APIVersion: "v1",
Labels: map[string]string{
"weave.works/templateType": "application",
}
}, something like {
Cluster: "management",
Name: "podinfo-a",
Namespace: "namespace-a",
Kind: "HelmChart",
APIGroup: "apps",
APIVersion: "v1",
TemplateType: "application",
}, and the query would be
to have that type of querying just via the indexer or unstructured. do you see whether this could be feasible or what would be the major impediments for it? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My thinking is that it would be weird to have a Adding all these fields to the normalized object would take quite a bit more maintenance and involvement for someone wanting to add their new May be we can add it solely to the indexed object on There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i reminder for @enekofb is to refresh the picture on the indexes that we currently have ... cause that is also something that we could look at. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. doing some discovery in this pr weaveworks/weave-gitops-enterprise#3537 |
||
|
||
Given this data ^^, here is the type of query we would like to support: | ||
|
||
```golang | ||
q := &query{ | ||
labels: map[string]string{ | ||
"weave.works/template": "true", | ||
} | ||
filters: []string{ | ||
"kind:/(HelmChart|HelmRepository)/", | ||
}, | ||
} | ||
``` | ||
|
||
The main issue with this query is that [bleve](https://blevesearch.com/), our current indexing/querying solution is not capable of doing this type of query, or if it is, the documentation available does not explain how it can be done. | ||
|
||
## Decision | ||
|
||
## Consequences |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you expand on what you mean with
enumerable fields
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Enumerable" in this context means: "Fields we know exist ahead of time". We can predict that an object will have a
namespace
, but we cannot predict that it has anylabels
or what those label keys might be.