[Reporting] Add counter for total size of exported reports

[yaronp68]

users are asking for the size of exported reports as it is another load on IO and on the cluster in general. They would like it to be in Kibana monitoring information.

@ycombinator @cachedout

[elasticmachine]

Pinging @elastic/stack-monitoring

[AlonaNadler]

adding @joelgriffith @tsullivan

[joelgriffith]

Good timing! We just had this land in the reporting index as we were experiencing issues when ES had a lower body-size limit vs kibana. Here's the initial PR: #26667. I'm not sure what work would be required to pull into monitoring, or if they're able to pull this info in already?

[tsullivan]

Kibana's usage collection services will let us define a "stats" collector. Most people are familiar with the usage collectors that get instantiated for telemetry, but there's a more general class that Reporting can use to provide stats from Reporting.

Report size doesn't seem particularly difficult, for the reason Joel brought up.

[cachedout]

Hi all. I'm trying to understand how this would be expressed in monitoring in a way that would be useful to the users in question. @yaronp68 could you please describe a little bit more how you would imagine users interacting with this data?

[joelgriffith]

@yaronp68 this issue has been hanging around for a few days... is this still something you see as valuable or should we close this?

[yaronp68]

@cachedout we have global counters for operations and bytes for many metrics. We can accumulate the total size bytes exported in a global counter in kibana and report it to monitoring and telemetry. Like search and indexing it gives an indication of load. In this case on kibana IO as well

does this make sense.
@joelgriffith I would ask @cachedout and @ycombinator to see if what you expose is what they need to collect using the metricbeat kibana module and in this case we can close this one

[cachedout]

@yaronp68 I'm still struggling a bit to see how we would display this type of data in Stack Monitoring right now in a way that's going to be helpful to the user.

Do we have any test data that indicates how close the relationship is between document size and the load on Kibana? Do we know how long does it typically take to render these sorts of large documents? In the event that a very large document spikes the load of Kibana, do we know if it's even able to respond to API requests during that period?

I'm not sure if the idea here is to display the document-size statistics in a time-series graph? Could you clarify how you envisioned presenting this data in the UI?

What I'm wondering here is whether or not it makes sense to poll for this type of data or whether to log it (as appears to happen now). I tend to think that we might be more successful with this type of problem if we consider upping the log-level in Kibana to something above "debug" versus polling the Kibana API for that data with Metricbeat.

[tsullivan]

The logging on report size is only something that happens on a per-report basis, at the time of report generation. There are a few diagnostic benefits from logging the report size.

Monitoring how the total size changes over time has benefits for curating the reporting data.

Since we want to express how much disk space is taken by the reporting indices, we have to keep in mind that in Lucene, a deleted document still exists in the shard, it is just marked as deleted. The bytes from deleted documents are reclaimed when Lucene segments are merged.

[tsullivan]

@joelgriffith and I are at a point where having metrics on Reporting visible in a UI would be greatly beneficial to our dev cycles. How can we help!?!?!?

[cachedout]

@tsullivan I'd love to hear more specifics about what you're looking for and how you'd like to see the data presented. We could talk next week or you could put more details in this PR. Either way, I'm sure we can find a way to make this happen. Thanks!

[tsullivan]

@cachedout We've recently gone through some refactoring and performance improvement fixes to Reporting, but without measuring the baseline and comparing our changes, we don't have the best idea on how much help the work is doing.

The problem of measuring performance gets harder with each release because Kibana's growing size also impacts the end-to-end time of reports.

We think that a lot of reporting performance has to do with Kibana UI performance, but we don't really know how much. @Bamieh is working on capturing UI performance metrics, which I think will help us gather more clues: #46851 (comment)

In general, we want to answer the question of how much server-side load and client-side load happens when running a reporting job. I see it as the need to just capture more on performance all-around.

Here is the main issue on Monitoring of Reporting: #29281 - it covers the need to measure CPU and memory utilization of the headless browser child process Kibana spawns to run a report. We can get the process PID of the browser, but we need some ideas on where to go from there.

[tsullivan]

Coming back to this issue,

users are asking for the size of exported reports as it is another load on IO and on the cluster in general. They would like it to be in Kibana monitoring information.

We can easily report the size of reports for Monitoring consumption. I think this is just a matter of getting it on the roadmap. I just don't think the size of exported reports is one of the biggest factors that put load into a cluster. If it refers to the size of the Elasticsearch indices, that information is already shown in ES Monitoring.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

Replacing with #29281