thanos-query segv panic on labels regexp #7676
Comments
|
Hey there, @gberche-orange! I'm here to help you with any bugs, questions, or contributions you might have. Let's see if we can tackle this issue together! I found a similar closed issue: receive: Query failure on Seg fault which was last updated on January 27, 2023. This issue was referenced by several other discussions and pull requests, including:
The suggested solution from the duplicate issue discussion points to a possible culprit related to gogo/protobuf#651. The recommended fix is to move away from the unmaintained Additionally, the // LabelSetsMatch returns false if all label-set do not match the matchers (aka: OR is between all label-sets).
func LabelSetsMatch(matchers []*labels.Matcher, lset ...labels.Labels) bool {
if len(lset) == 0 {
return true
}
for _, ls := range lset {
notMatched := false
for _, m := range matchers {
if lv := ls.Get(m.Name); ls.Has(m.Name) && !m.Matches(lv) {
notMatched = true;
break;
}
}
if !notMatched {
return true;
}
}
return false;
}This function checks if any of the provided label sets match the given matchers [2]. To address the panic, you might want to investigate the |
|
Do you have any components that talk to the Query component directly through the StoreAPI? |
|
There was a bug in the distributed query engine where we didn't propagate matchers correctly. I think that should be fixed in this but just to make sure... Is this the distributed engine? If not does it happen for the Prometheus engine too? |
|
Thanks for your responses
No, our thanos instance is only queried by Grafana through the query front-end AFAIK
No. We don't use distributed engine (parameter not set --query.engine=distributed on the query pod, since we were having issues with it as mentioned in #7328 ) |
|
Possible that this is related to the problem I'm experiencing in #7844? Does reverting to Thanos 0.35.1 make the problem go away? |
Thanks @chris-barbour-as for the heads up! Yes, reverting to Thanos 0.35.1 (through bitnami helm chart version https://artifacthub.io/packages/helm/bitnami/thanos/15.7.15) resolved the issue for me. |
|
@gberche-orange I am looking at the code and it seems related to the store matchers. |
|
@gberche-orange This should be fixed by #7903 |
Thanos, Prometheus and Golang version used:
Object Storage Provider:
Scality
What happened:
thanos-query segv panic apparently on labels regexp according to stack trace.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
I am not yet able to identify the query that triggers this behavior.
Full logs to relevant components:
Anything else we need to know:
The text was updated successfully, but these errors were encountered: