Disk usage has increased in the nightly Logging benchmark

[ChrisHegarty]

Disk usage has increased in the nightly Logging benchmark [1] (on 2023-12-04), from 182GB to 190GB. A little more than 4%. This lines up with the bump to Lucene 9.9.0 in Elasticsearch.

Lucene 9.9.0 has changed postings back to using FOR (from PFOR)[2] . This is the likely change that has resulted in the additional usage.

[1] https://elasticsearch-benchmarks.elastic.co/#tracks/logging/nightly/default/30d - look at nightly-elastic/logs-io
[2] apache/lucene#12741

Note from the Lucene 9.9.0 changelog: "GITHUB#12696: Change Postings back to using FOR in Lucene99PostingsFormat. Freqs, positions and offset keep using PFOR."

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[ChrisHegarty]

Elasticsearch can add back PFOR encoding for posting, in our own codec.

[martijnvg]

Maybe a heuristic for using PFOR would be if an index is backing index of a data stream? (mapperService.mappingLookup().isDataStreamTimestampFieldEnabled() would do the trick)