Check fields/body size not to exceed data limits of Azure Monitor Data Collection API

[yokawasa]

Enhancement

There are data limits of Azure Monitor Data Collection API. An enhancement is to check fields/body size not to exceed data limits of Azure Monitor Data Collection API.

Data Limits

There are some constraints around the data posted to the Azure Monitor Data collection API.

Maximum of 30 MB per post to Azure Monitor Data Collector API. This is a size limit for a single post. If the data from a single post that exceeds 30 MB, you should split the data up to smaller sized chunks and send them concurrently.
Maximum of 32 KB limit for field values. If the field value is greater than 32 KB, the data will be truncated.
Recommended maximum number of fields for a given type is 50. This is a practical limit from a usability and search experience perspective.
A table in a Log Analytics workspace only supports up to 500 columns (referred to as a field in this article).
The maximum number of characters for the column name is 500.

• https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api

relevant issues

• question: Is plugin prepared for LogAnalytics data limit? #15

[marcosflobo]

@yokawasa I see this issue closed, does it mean that this feature is released in version 0.6.0? 😃

[yokawasa]

@marcosflobo
Thanks for the comment. The release is not for this feature.
I'm releasing new version for this feature shortly.

[yokawasa]

@marcosflobo
I've just published fluent-plugin-azure-loganalytics-0.7.0 that change base azure-loganalytics-datacollector-api to ">= 0.5.0". azure-log-analytics-data-collector-0.5.0 added check body size not to exceed data limit of 30MB in Azure Monitor Data Collection API

• https://github.com/yokawasa/fluent-plugin-azure-loganalytics/releases/tag/v0.7.0
• https://rubygems.org/gems/fluent-plugin-azure-loganalytics/versions/0.7.0

I didn't add each fields size check as I thought it may cause some performance issue if the num of field is too large. Instead, I added some description on data limits on README.md.

• https://github.com/yokawasa/fluent-plugin-azure-loganalytics#data-limits

[yokawasa]

@marcosflobo I'm closing this issue
Please open an issue if you have any other suggestions!
Thanks again for your suggestion.

[marcosflobo]

Thank you very much @yokawasa for your work for the open-source community!! We'll upgrade our version in production very soon thanks to you!

[xrstf]

Are there plans to support

A table in a Log Analytics workspace only supports up to 500 columns (referred to as a field in this article).

out of the box or should users use the record_transformer to remove unwanted/overhanging fields themselves?