Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] EsqlSpecIT test {ints.MultipleWarnings} failing #100164

Closed
costin opened this issue Oct 2, 2023 · 5 comments · Fixed by #100724
Closed

[CI] EsqlSpecIT test {ints.MultipleWarnings} failing #100164

costin opened this issue Oct 2, 2023 · 5 comments · Fixed by #100724
Assignees
Labels
:Analytics/ES|QL AKA ESQL Team:QL (Deprecated) Meta label for query languages team >test-failure Triaged test failures from CI

Comments

@costin
Copy link
Member

costin commented Oct 2, 2023

Build scan:
https://gradle-enterprise.elastic.co/s/juee6bgkgo6na/tests/:x-pack:plugin:esql:qa:server:multi-node:javaRestTest/org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT/test%20%7Bints.MultipleWarnings%7D
Reproduction line:

./gradlew ':x-pack:plugin:esql:qa:server:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT" -Dtests.method="test {ints.MultipleWarnings}" -Dtests.seed=CAB3ECC97601B05C -Dtests.locale=zh-HK -Dtests.timezone=Asia/Brunei -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT&tests.test=test%20%7Bints.MultipleWarnings%7D
Failure excerpt:

java.lang.AssertionError: Expected a list containing
 0: "Line 1:76: evaluation of [to_int(x)] failed, treating result as null. Only first 20 failures recorded."
 1: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100010000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100680000000] out of [integer] range"
 2: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100020000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100690000000] out of [integer] range"
 3: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100030000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100700000000] out of [integer] range"
 4: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100040000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100710000000] out of [integer] range"
 5: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100050000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100720000000] out of [integer] range"
 6: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100060000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100730000000] out of [integer] range"
 7: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100070000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100740000000] out of [integer] range"
 8: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100080000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100750000000] out of [integer] range"
 9: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100090000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100760000000] out of [integer] range"
10: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100100000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100770000000] out of [integer] range"
11: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100110000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100780000000] out of [integer] range"
12: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100120000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100790000000] out of [integer] range"
13: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100130000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100800000000] out of [integer] range"
14: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100140000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100810000000] out of [integer] range"
15: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100150000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100820000000] out of [integer] range"
16: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100160000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100830000000] out of [integer] range"
17: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100170000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100840000000] out of [integer] range"
18: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100180000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100850000000] out of [integer] range"
19: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100190000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100860000000] out of [integer] range"
20: expected "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100200000000] out of [integer] range" but was "Line 1:76: org.elasticsearch.xpack.ql.QlIllegalArgumentException: [100870000000] out of [integer] range"

  at __randomizedtesting.SeedInfo.seed([CAB3ECC97601B05C:42E7D313D8FDDDA4]:0)
  at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:82)
  at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:65)
  at org.elasticsearch.xpack.esql.qa.rest.RestEsqlTestCase.performRequest(RestEsqlTestCase.java:498)
  at org.elasticsearch.xpack.esql.qa.rest.RestEsqlTestCase.runEsql(RestEsqlTestCase.java:447)
  at org.elasticsearch.xpack.esql.qa.rest.EsqlSpecTestCase.doTest(EsqlSpecTestCase.java:102)
  at org.elasticsearch.xpack.esql.qa.rest.EsqlSpecTestCase.test(EsqlSpecTestCase.java:94)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

@costin costin added :Analytics/ES|QL AKA ESQL >test-failure Triaged test failures from CI labels Oct 2, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine elasticsearchmachine added the Team:QL (Deprecated) Meta label for query languages team label Oct 2, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

@bpintea
Copy link
Contributor

bpintea commented Oct 9, 2023

@dnhatn CC
This test seems to occasionally still loose the headers. Maybe also relevant for #99927.

@dnhatn
Copy link
Member

dnhatn commented Oct 9, 2023

@bpintea Sure, I will have a look.

@dnhatn
Copy link
Member

dnhatn commented Oct 12, 2023

Response headers can be lost when a driver is yielded and rescheduled. I've opened #100724.

dnhatn added a commit that referenced this issue Oct 18, 2023
Today, we have a hierarchy of tasks in ESQL designed to leverage the 
task framework for reporting status and cancellation.

```mermaid
flowchart
 RESTLayer -->|  EsqlQueryRequest indices:data/read/esql  | ComputeService
 ComputeService -->| DriverRequest indices:data/read/esql/compute | Driver
 ComputeService -->| DataNodeRequest indices:data/read/esql/data | DataNode
 DataNode -->| DriverRequest indices:data/read/esql/compute | Driver
 Driver -->| LookupRequest indices:data/read/esql/lookup | EnrichLookupService
```

The primary issue here is that `DriverRequest` is neither 
`IndicesRequest` nor `CompositeIndicesRequest`. Consequently, the Driver
is executed within the context of the system user, leading to access
indices with the system user.

To address this issue, this PR makes `DriverRequest` a 
`CompositeIndicesRequest` and ensures that the Driver executes within
the user's context. With this fix we can now properly capture the
response headers when a Driver is yielded and rescheduled.

Relates #100707
Relates #99646

Relates #99926
Closes #100164
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Oct 18, 2023
Today, we have a hierarchy of tasks in ESQL designed to leverage the 
task framework for reporting status and cancellation.

```mermaid
flowchart
 RESTLayer -->|  EsqlQueryRequest indices:data/read/esql  | ComputeService
 ComputeService -->| DriverRequest indices:data/read/esql/compute | Driver
 ComputeService -->| DataNodeRequest indices:data/read/esql/data | DataNode
 DataNode -->| DriverRequest indices:data/read/esql/compute | Driver
 Driver -->| LookupRequest indices:data/read/esql/lookup | EnrichLookupService
```

The primary issue here is that `DriverRequest` is neither 
`IndicesRequest` nor `CompositeIndicesRequest`. Consequently, the Driver
is executed within the context of the system user, leading to access
indices with the system user.

To address this issue, this PR makes `DriverRequest` a 
`CompositeIndicesRequest` and ensures that the Driver executes within
the user's context. With this fix we can now properly capture the
response headers when a Driver is yielded and rescheduled.

Relates elastic#100707
Relates elastic#99646

Relates elastic#99926
Closes elastic#100164
elasticsearchmachine pushed a commit that referenced this issue Oct 18, 2023
Today, we have a hierarchy of tasks in ESQL designed to leverage the 
task framework for reporting status and cancellation.

```mermaid
flowchart
 RESTLayer -->|  EsqlQueryRequest indices:data/read/esql  | ComputeService
 ComputeService -->| DriverRequest indices:data/read/esql/compute | Driver
 ComputeService -->| DataNodeRequest indices:data/read/esql/data | DataNode
 DataNode -->| DriverRequest indices:data/read/esql/compute | Driver
 Driver -->| LookupRequest indices:data/read/esql/lookup | EnrichLookupService
```

The primary issue here is that `DriverRequest` is neither 
`IndicesRequest` nor `CompositeIndicesRequest`. Consequently, the Driver
is executed within the context of the system user, leading to access
indices with the system user.

To address this issue, this PR makes `DriverRequest` a 
`CompositeIndicesRequest` and ensures that the Driver executes within
the user's context. With this fix we can now properly capture the
response headers when a Driver is yielded and rescheduled.

Relates #100707
Relates #99646

Relates #99926
Closes #100164
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL Team:QL (Deprecated) Meta label for query languages team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants