CCR: Aborted document is exposed in Lucene changes

[dnhatn]

The CCR branch started failing frequently after merging #31007. Some CI instances:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+ccr+feature-branch-periodic/1002/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+ccr+feature-branch-periodic/1000/console

These failures can be explained as follows:

1. A user issues an indexing which will throw an exception in the analyzing phase. Since the IndexingChain fails to process a document, the DocumentsWriterPerThread will hard-delete that document internally in Lucene on the primary.

[2018-07-20T18:04:15,095][DEBUG][o.e.a.b.TransportShardBulkAction] [test][0] failed to execute bulk item (index) BulkShardRequest [[test][0]] containing [index {[test][test][2], source[{"suggest_context":{"input":"foo"}}]}]
java.lang.IllegalArgumentException: Contexts are mandatory in context enabled completion field [suggest_context]

2. On ES, we make all docs live when reading Lucene changes history (in fact, we can not distinguish between hard-deletes and soft-deletes). If a recovering replica reads the aborted document, it will fail to index that document. In fact, the replica will never be able to complete its recovery.

[2018-07-20T18:04:15,148][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [elasticsearch[node-1][generic][T#4]], exiting
java.lang.AssertionError: unexpected failure while replicating translog entry: java.lang.IllegalArgumentException: Contexts are mandatory in context enabled completion field [suggest_context]
    at org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:401) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:458) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT]
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:448)

The problem is that we read aborted documents which should never be exposed. This might be a critical problem in CCR and Lucene rollbacks.

/cc @s1monw and @bleskes

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

I see a solution that brings a deleted doc to live iff it was soft-deleted. However, this approach may be fragile for stale documents (soft-deleted before indexing) because it relies on the order of processing fields. If the soft-deletes field of a doc is processed then that doc is aborted, we can't exclude that document from liveDocs. Another option is to expose hard liveDocs in Lucene.

[s1monw]

I think we can expose the actual hard deletes on a SegmentReader level. that is the best solution IMO.

[s1monw]

as a workaround we can fix this specific issue by loading the original livedocs if there are any deletions. This is safe in this case since we write deletes to disk on flush so if there is an aborted doc we will have at least 1 hard deleted doc. We can then do this for the segment reader in question:

SegmentReader reader = ...;
SegmentCommitInfo si = reader.getSegmentInfo();
Bits hardLiveDocs = si.getDelCount() != 0 ?  si.info.getCodec().liveDocsFormat().readLiveDocs(si.info.dir, si, IOContext.READONCE) : null;

@dnhatn WDYT?

[dnhatn]

@s1monw Thanks for the hint. I opened #32333.

[dnhatn]

Fixed by #32333.