[BUG] Flaky test org.opensearch.search.fetch.subphase.highlight.HighlighterSearchIT.testHighlightQueryRewriteDatesWithNow

[peternied]

Failure:

java.lang.AssertionError: 
Expected: an empty iterable
     but: [<MapperParsingException[failed to parse field [d] of type [date] in document with id '1'. Preview of field's value: '2023-10-05T17:30:34.967Z']; nested: IllegalArgumentException[failed to parse date field [2023-10-05T17:30:34.967Z] with format [strict_date_time_no_millis||strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];>]

Repro with

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.fetch.subphase.highlight.HighlighterSearchIT" -Dtests.method="testHighlightQueryRewriteDatesWithNow {p0={"search.concurrent_segment_search.enabled":"false"}}" -Dtests.seed=5A3F88D61C8FD270 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-CL -Dtests.timezone=SystemV/HST10 -Druntime.java=20

Additional context:

          ### Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27137/
• CommitID: 02b92ab
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Originally posted by @github-actions[bot] in #10261 (comment)

[reta]

CC @CaptainDredge

[CaptainDredge]

@reta can you please close this? Not able to repro it

[dblock]

@CaptainDredge It clearly failed when this issue was opened, that's why it's flaky. It mostly passes, except when otherwise. No repro is not a good reason to close ;)

[reta]

Replied in #10474 (comment) but the issue was addressed (not ignored or disappeared by itself)