Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add indexer ADR #132

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions docs/adrs/0022-query-service-indexer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# 22. Query Service Indexer

Date: 2023-08-18

## Status

Draft

## Context

Our solution for cross-cluster querying has evolved to support some powerful querying patterns (complex AND/OR logic, full-text search), but may be deficient for the next level of capabilities that our (internal) users are asking for.

At present, we can do fairly complex query logic with **normalized fields**:

```golang

// Given these objects
objects := []models.Object{
{
Cluster: "management",
Name: "podinfo-a",
Namespace: "namespace-a",
Kind: "HelmChart",
APIGroup: "apps",
APIVersion: "v1",
},
{
Cluster: "management",
Name: "podinfo-b",
Namespace: "namespace-a",
Kind: "HelmRepository",
APIGroup: "apps",
APIVersion: "v1",
},
{
Cluster: "management",
Name: "podinfo-d",
Namespace: "namespace-b",
Kind: "HelmRepository",
APIGroup: "apps",
APIVersion: "v1",
},
}


qs := NewQueryService(...)


q := &query{filters: []string{
// Regex-style query where we can add the OR logic within a field.
"kind:/(HelmChart|HelmRepository)/",
// Top-level fields are AND'd together.
"namespace:namespace-a",
}}


result := qs.RunQuery(q)
// result will have "podinfo-a", "podinfo-b", but not "podinfo-d" since it is not in namespace-a
```

This works for enumerable fields, but there may be cases where we need to query against fields for which we do not know the key/value pairs. For example, if we want to query for labels on an object we run into a couple issues:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you expand on what you mean with enumerable fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Enumerable" in this context means: "Fields we know exist ahead of time". We can predict that an object will have a namespace, but we cannot predict that it has any labels or what those label keys might be.


- We don't know the fields for a given object's labels before adding them to the indexer
- Each object's labels have their own key/value pairs that are only relevant in the context of one object, or more realistically, one `kind` of object

An example of two normalized objects with different label pairs:

```golang
objects := []models.Object{
{
Cluster: "management",
Name: "podinfo-a",
Namespace: "namespace-a",
Kind: "HelmChart",
APIGroup: "apps",
APIVersion: "v1",
Labels: map[string]string{
"weave.works/template": "true",
}
},
{
Cluster: "management",
Name: "podinfo-b",
Namespace: "namespace-a",
Kind: "HelmRepository",
APIGroup: "apps",
APIVersion: "v1",
Labels: map[string]string{
"other.org.com/somekey": "othervalue",
}
},
}
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my mind, i was actually thinking in a different direction where the object would look instead of

{
        Cluster:    "management",
        Name:       "podinfo-a",
        Namespace:  "namespace-a",
        Kind:       "HelmChart",
        APIGroup:   "apps",
        APIVersion: "v1",
        Labels: map[string]string{
            "weave.works/templateType": "application",
        }
    },

something like

{
        Cluster:    "management",
        Name:       "podinfo-a",
        Namespace:  "namespace-a",
        Kind:       "HelmChart",
        APIGroup:   "apps",
        APIVersion: "v1",
       TemplateType: "application",
    },

and the query would be

q := &query{
    filters: []string{
        "kind:/(HelmChart|HelmRepository)/",
        "templateType: application",
    },
}

to have that type of querying just via the indexer or unstructured.

do you see whether this could be feasible or what would be the major impediments for it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking is that it would be weird to have a HelmRelease with a TemplateType key, for example. The only object with a TemplateType that is actually valid is a GitOpsTemplate.

Adding all these fields to the normalized object would take quite a bit more maintenance and involvement for someone wanting to add their new kind.

May be we can add it solely to the indexed object on .Add(). I will give that a try.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i reminder for @enekofb is to refresh the picture on the indexes that we currently have ... cause that is also something that we could look at.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doing some discovery in this pr weaveworks/weave-gitops-enterprise#3537


Given this data ^^, here is the type of query we would like to support:

```golang
q := &query{
labels: map[string]string{
"weave.works/template": "true",
}
filters: []string{
"kind:/(HelmChart|HelmRepository)/",
},
}
```

The main issue with this query is that [bleve](https://blevesearch.com/), our current indexing/querying solution is not capable of doing this type of query, or if it is, the documentation available does not explain how it can be done.

## Decision

## Consequences