-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix failures in BulkProcessorIT#testGlobalParametersAndBulkProcessor. #38129
Conversation
This PR fixes a couple test issues: * It narrows an assertWarnings call that was too broad, and wasn't always applicable with certain random sequences. * Previously, we could send a typeless bulk request containing '_type: 'null'. Now we omit the _type key altogether for typeless requests.
Pinging @elastic/es-search |
These issues originally came to light because @atorok saw this test fail an intake build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1702/testReport/junit/org.elasticsearch.client/BulkProcessorIT/testGlobalParametersAndBulkProcessor/ |
I can't see the original failure now. Do you have the reproduce line?
I don't see that broader previous |
Oops, I didn't realize that link would expire. Here was the reproduce line it contained:
The |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
There was clearly an issue with serializing null.
I added a comment signalling my unease about the warnings checks we have here - we're not really cleanly testing an API's contract here. We're just squashing some awkward side effects of reusing server-side POJOs on the client - they're setting up warnings in thread locals that have no business appearing in client-side processes but we're having to acknowledge them anyway just to make tests pass. It's ugly (no discredit to you intended) but I guess we have to live with it until we fully divorce HLRC from server-side POJOs.
@@ -448,33 +448,41 @@ private static MultiGetRequest indexDocs(BulkProcessor processor, int numDocs, S | |||
} else { | |||
BytesArray data = bytesBulkRequest(localIndex, localType, i); | |||
processor.add(data, globalIndex, globalType, globalPipeline, null, XContentType.JSON); | |||
|
|||
if (localType != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line troubled me. Bulk requests can pass a type on each request line (localType
var here) or pass a default fallback choice which is inherited by any request lines that don't declare a localType (the fallback is called globalType
).
So here, I assumed the test's logic should ideally be:
if (localType !=null || globalType != null) {
These are the conditions under which a user has sent a choice of custom type to the server.
Sadly this doesn't work. It looks like in the server-side BulkRequest
that only types expressed at the line level are flagged. It assumes any "global" references to type are already flagged by the main RestBulkAction which we are not exercising in this test. What we are squashing with these assertWarnings
checks in BulkProcessorIT are the ThreadContext values normally set up on the server side by the server-side objects like BulkRequest which we are (questionably) re-using in HLRC.
These warning checks are not really the same as simply checking that the response from a _bulk REST interaction has the expected warning if you use happen to use global or local type references in the request. That contract is exercised in BulkRequestWithGlobalParametersIT
so I think we're good there.
Thanks @markharwood for the review. Off the top of my head, it seems like if we had an |
* master: (23 commits) Lift retention lease expiration to index shard (elastic#38380) Make Ccr recovery file chunk size configurable (elastic#38370) Prevent CCR recovery from missing documents (elastic#38237) re-enables awaitsfixed datemath tests (elastic#38376) Types removal fix FullClusterRestartIT warnings (elastic#38445) Make sure to reject mappings with type _doc when include_type_name is false. (elastic#38270) Updates the grok patterns to be consistent with logstash (elastic#27181) Ignore type-removal warnings in XPackRestTestHelper (elastic#38431) testHlrcFromXContent() should respect assertToXContentEquivalence() (elastic#38232) add basic REST test for geohash_grid (elastic#37996) Remove DiscoveryPlugin#getDiscoveryTypes (elastic#38414) Fix the clock resolution to millis in GetWatchResponseTests (elastic#38405) Throw AssertionError when no master (elastic#38432) `if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API (elastic#38411) Enable CronEvalToolTest.testEnsureDateIsShownInRootLocale (elastic#38394) Fix failures in BulkProcessorIT#testGlobalParametersAndBulkProcessor. (elastic#38129) SQL: Implement CURRENT_DATE (elastic#38175) Mute testReadRequestsReturnLatestMappingVersion (elastic#38438) [ML] Report index unavailable instead of waiting for lazy node (elastic#38423) Update Rollup Caps to allow unknown fields (elastic#38339) ...
This PR fixes a couple test issues:
applicable with certain random sequences.
Now we omit the _type key altogether for typeless requests.