[BUG] ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' fails #2334

andrross · 2022-03-03T23:53:48Z

Running the following test seems to fail consistently on main:

./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest'

Failures:

REPRODUCE WITH: ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' --tests "org.opensearch.upgrades.FullClusterRestartIT.testRecoveryWithTranslogRetentionDisabled" -Dtests.seed=4BBBB3F7DB8DEDF8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb -Dtests.timezone=Asia/Tomsk -Druntime.java=17

org.opensearch.upgrades.FullClusterRestartIT > testRecoveryWithTranslogRetentionDisabled FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://[::1]:43897], URI [/testrecoverywithtranslogretentiondisabled], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testrecoverywithtranslogretentiondisabled"}
        at __randomizedtesting.SeedInfo.seed([4BBBB3F7DB8DEDF8:36AD1B03BAABB8F4]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.FullClusterRestartIT.testRecoveryWithTranslogRetentionDisabled(FullClusterRestartIT.java:1352)

REPRODUCE WITH: ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' --tests "org.opensearch.upgrades.FullClusterRestartIT.testEnableSoftDeletesOnRestore" -Dtests.seed=4BBBB3F7DB8DEDF8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb -Dtests.timezone=Asia/Tomsk -Druntime.java=17

org.opensearch.upgrades.FullClusterRestartIT > testEnableSoftDeletesOnRestore FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:33817], URI [/testenablesoftdeletesonrestore], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testenablesoftdeletesonrestore"}
        at __randomizedtesting.SeedInfo.seed([4BBBB3F7DB8DEDF8:7FE4775E071E53C3]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.FullClusterRestartIT.testEnableSoftDeletesOnRestore(FullClusterRestartIT.java:1551)

REPRODUCE WITH: ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' --tests "org.opensearch.upgrades.FullClusterRestartIT.testOperationBasedRecovery" -Dtests.seed=4BBBB3F7DB8DEDF8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb -Dtests.timezone=Asia/Tomsk -Druntime.java=17

org.opensearch.upgrades.FullClusterRestartIT > testOperationBasedRecovery FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:36513], URI [/testoperationbasedrecovery], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testoperationbasedrecovery"}
        at __randomizedtesting.SeedInfo.seed([4BBBB3F7DB8DEDF8:5657BFD2DD0B178F]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.FullClusterRestartIT.testOperationBasedRecovery(FullClusterRestartIT.java:1289)

REPRODUCE WITH: ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' --tests "org.opensearch.upgrades.FullClusterRestartIT.testSnapshotRestore" -Dtests.seed=4BBBB3F7DB8DEDF8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb -Dtests.timezone=Asia/Tomsk -Druntime.java=17

org.opensearch.upgrades.FullClusterRestartIT > testSnapshotRestore FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://[::1]:43897], URI [/testsnapshotrestore], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testsnapshotrestore"}
        at __randomizedtesting.SeedInfo.seed([4BBBB3F7DB8DEDF8:74DE590E1130EA87]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.FullClusterRestartIT.testSnapshotRestore(FullClusterRestartIT.java:810)

The text was updated successfully, but these errors were encountered:

andrross · 2022-03-04T00:04:18Z

The tests pass using this 1.x commit:

./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' -Dbwc.refspec.1.x=13bb6cb46a1e2069c084775a3946a48772254132

but fail using this commit:

./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' -Dbwc.refspec.1.x=6004f051402ba61f9d5225a9c799fd4bbd20f679

looks like the backport PR #2315 might be the cause. @VachaShah care to take a look?

VachaShah · 2022-03-04T00:10:03Z

Sure I will take a look

VachaShah · 2022-03-04T00:26:35Z

@andrross Looks like this failure is on main, the original PR #1660 for backport #2315 has been merged since a few months to main.

andrross · 2022-03-04T00:30:11Z

@VachaShah The ':qa:full-cluster-restart:v1.3.0#oldClusterTest' test task checks out the latest commit on the 1.x branch to run some backward compatibility tests. That test fails when running on main. The following command tells the test to use an older commit from 1.x (the commit before your backport commit), and then the tests pass:

./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' -Dbwc.refspec.1.x=13bb6cb46a1e2069c084775a3946a48772254132

If you point the test at the commit of your backport PR then it fails again. That's why I suspect the backport PR to 1.x is causing the backward compatibility test failure on main.

VachaShah · 2022-03-04T00:38:45Z

Ah got it! That makes sense. I jumped to the comment too fast 🤦🏼‍♀️ . I will take a look at why this is failing bwc. Should I revert the commit till then?

andrross · 2022-03-04T00:40:21Z

It looks like there's a problem with an unexpected warning. If it is a quick fix then we can just go ahead with the fix. If you can't figure it out quickly then a revert would be better.

VachaShah · 2022-03-04T00:41:37Z

Sounds good, on it.

dreamer-89 · 2022-03-04T06:49:38Z

@VachaShah : I started seeing testRecoverReplica more frequently in main gradle check failures. Is this also covered as part of fix here ?

REPRODUCE WITH: ./gradlew ':qa:translog-policy:v1.3.0#Step1OldClusterTest' --tests "org.opensearch.upgrades.TranslogPolicyIT.testRecoverReplica" -Dtests.seed=A7F39AB3B5FF3CF9 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mt -Dtests.timezone=Chile/Continental -Druntime.java=17

org.opensearch.upgrades.TranslogPolicyIT > testRecoverReplica FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:33619], URI [/testrecoverreplica], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testrecoverreplica"}
        at __randomizedtesting.SeedInfo.seed([A7F39AB3B5FF3CF9:5D6A55174026FDF4]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.TranslogPolicyIT.testRecoverReplica(TranslogPolicyIT.java:129)

Sample failure: #2335 (comment)

VachaShah · 2022-03-04T07:39:49Z

@VachaShah : I started seeing testRecoverReplica more frequently in main gradle check failures. Is this also covered as part of fix here ?

REPRODUCE WITH: ./gradlew ':qa:translog-policy:v1.3.0#Step1OldClusterTest' --tests "org.opensearch.upgrades.TranslogPolicyIT.testRecoverReplica" -Dtests.seed=A7F39AB3B5FF3CF9 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mt -Dtests.timezone=Chile/Continental -Druntime.java=17

org.opensearch.upgrades.TranslogPolicyIT > testRecoverReplica FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:33619], URI [/testrecoverreplica], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testrecoverreplica"}
        at __randomizedtesting.SeedInfo.seed([A7F39AB3B5FF3CF9:5D6A55174026FDF4]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.TranslogPolicyIT.testRecoverReplica(TranslogPolicyIT.java:129)

Sample failure: #2335 (comment)

Yes it will fix the soft deletes warning that are causing this error.

dblock · 2022-03-04T15:09:02Z

Closed via #2339

andrross added bug Something isn't working untriaged labels Mar 3, 2022

andrross mentioned this issue Mar 3, 2022

Updated the url for docker distribution #2325

Merged

5 tasks

VachaShah mentioned this issue Mar 4, 2022

[Remove] types from PutMappingRequest #2335

Merged

VachaShah mentioned this issue Mar 4, 2022

Fixing soft deletes deprecation warning #2339

Merged

5 tasks

dblock closed this as completed Mar 4, 2022

nknize mentioned this issue Mar 4, 2022

Log warning in rest client response handler #2338

Closed

This was referenced Mar 4, 2022

Replace exclusionary words whitelist and blacklist in the places that won't impact backwards compatibility #2178

Merged

[BUG] 1.x changes can break builds on main #2350

Open

reta mentioned this issue Mar 7, 2022

Update the BWC versions (post 1.x backport) #2390

Merged

5 tasks

kartg removed the untriaged label Mar 8, 2022

tlfeng mentioned this issue Mar 31, 2022

[AUTO] [1.x] Added bwc version 1.3.2. #2673

Merged

reta mentioned this issue Jun 15, 2022

[Backport 1.x] Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider #3581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' fails #2334

[BUG] ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' fails #2334

andrross commented Mar 3, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

VachaShah commented Mar 4, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

dreamer-89 commented Mar 4, 2022

VachaShah commented Mar 4, 2022

dblock commented Mar 4, 2022

[BUG] ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' fails #2334

[BUG] ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' fails #2334

Comments

andrross commented Mar 3, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

VachaShah commented Mar 4, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

andrross commented Mar 4, 2022

VachaShah commented Mar 4, 2022

dreamer-89 commented Mar 4, 2022

VachaShah commented Mar 4, 2022

dblock commented Mar 4, 2022