Use dataset size instead of on-disk size for data stream stats (#103342…

…) (#103345)

Previously we were using the size of the shard that was on local disk,
however, in some cases (for object storage) this would return 0b, which
is very confusing. This commit changes it to use the dataset size, which
should be calculated correctly regardless of where the data resides.

docs/changelog/103342.yaml

@@ -0,0 +1,5 @@
+pr: 103342
+summary: Use dataset size instead of on-disk size for data stream stats
+area: Data streams
+type: bug
+issues: []

modules/data-streams/src/main/java/org/elasticsearch/datastreams/action/DataStreamsStatsTransportAction.java

@@ -229,11 +229,11 @@ public DataStreamsStatsAction.Response newResponse(
                 assert dataStream != null;
 
                 // Aggregate global stats
-                totalStoreSizeBytes += shardStat.getStoreStats().sizeInBytes();
+                totalStoreSizeBytes += shardStat.getStoreStats().totalDataSetSizeInBytes();
 
                 // Aggregate data stream stats
                 AggregatedStats stats = aggregatedDataStreamsStats.computeIfAbsent(dataStream.getName(), s -> new AggregatedStats());
-                stats.storageBytes += shardStat.getStoreStats().sizeInBytes();
+                stats.storageBytes += shardStat.getStoreStats().totalDataSetSizeInBytes();
                 stats.maxTimestamp = Math.max(stats.maxTimestamp, shardStat.getMaxTimestamp());
             }