fluentd-otc-comparison.md

Comparison of Fluentd and Opentelemetry Collector functionality

• Sumologic supported Fluentd plugins
  • Sumologic Output Plugin
  • fluent-plugin-datapoint
  • fluent-plugin-protobuf
  • fluent-plugin-prometheus-format
  • fluent-plugin-kubernetes-sumologic
    • sanitized pod name
  • fluent-plugin-kubernetes-metadata-filter
  • fluent-plugin-enhance-k8s-metadata
  • fluent-plugin-events
• Configuration by pipelines
  • Events
  • Metrics
  • Logs
  • Other

Sumologic supported Fluentd plugins

Sumologic Output Plugin

Fluentd configuration option	Opentelemetry Collector
data_type	Defined by pipeline type (service.pipelines). See basic configuration documentation
endpoint	exporters.sumologic.endpoint
verify_ssl	exporters.sumologic.tls.insecure_skip_verify
source_category	processors.source.source_category. Placeholders format changed to %{}.
source_name	processors.source.source_name. Placeholders format changed to %{}.
source_name_key	Not supported. Use predefined attribute in processors.source.source_name. For example %{_sourceName}
source_host	processors.source.source_host. Placeholders format changed to %{}.
log_format	exporters.sumologic.log_format (json_merge and fields formats are not supported)
log_key	exporters.sumologic.json_logs.log_key
open_timeout	exporters.sumologic.timeout (doesn't differentiate between open and send)
send_timeout	exporters.sumologic.timeout (doesn't differentiate between open and send)
add_timestamp	exporters.sumologic.json_logs.add_timestamp
timestamp_key	exporters.sumologic.json_logs.timestamp_key
proxy_uri	environment variables
metric_data_format	exporters.sumologic.metric_format
disable_cookies	Cookies are not used in Opentelemetry Collector
compress	exporters.sumologic.compress_encoding set to ""
compress_encoding	exporters.sumologic.compress_encoding
custom_fields	Resource processor
custom_dimensions	Resource processor

Additional behavior:

Description	Opentelemetry Collector
record[_sumo_metadata][source_name] taking precedence over source_name	Can be achieved by separate pipelines
record[_sumo_metadata][source_host] taking precedence over source_host	Can be achieved by separate pipelines
record[_sumo_metadata][source_category] taking precedence over source_category	Can be achieved by separate pipelines
record[_sumo_metadata][fields] being base for fields	Can be achieved using resource processor and separate pipelines

fluent-plugin-datapoint

In order to receive prometheus data and for their initial processing telegrafreceiver is being used. It should cover fluent-plugin-datapoint functionality and more.

fluent-plugin-protobuf

In order to receive prometheus data and for their initial processing telegrafreceiver is being used. It should cover fluent_plugin_protobuf functionality and more.

fluent-plugin-prometheus-format

Fluentd configuration option	Opentelemetry Collector
relabel	Use groupbyattrs and resourceprocessor to relabel metrics
inclusions	Use filter processor
strict_inclusions	Use filter processor
exclusions	Use filter processor
strict_exclusions	Use filter processor

fluent-plugin-kubernetes-sumologic

Fluentd configuration option	Opentelemetry Collector
source_category	processors.source.source_category along with exporters.sumologic.sourceCategory: '%{_sourceCategory}'
source_category_replace_dash	processors.source.source_category_replace_dash along with exporters.sumologic.sourceCategory: '%{_sourceCategory}'
source_category_prefix	processors.source.source_category_prefix along with exporters.sumologic.sourceCategory: '%{_sourceCategory}
source_name	processors.source.source_name along with exporters.sumologic.sourceName: '%{_sourceName}'
log_format	N/A
source_host	processors.source.source_host along with exporters.sumologic.sourceHost: '%{_sourceHost}'
exclude_container_regex	processors.source.exclude
exclude_facility_regex	processors.source.exclude
exclude_host_regex	processors.source.exclude
exclude_namespace_regex	processors.source.exclude
exclude_pod_regex	processors.source.exclude
exclude_priority_regex	processors.source.exclude
exclude_unit_regex	processors.source.exclude
per_container_annotations_enabled	processors.source.container_annotations.enabled
per_container_annotation_prefixes	processors.source.container_annotations.prefixes

Additional behavior:

Description	Opentelemetry Collector
Allow to use sanitized pod name in source templates	processors.source.pod_name_key
Using _sumo_metadata to propagate configuration to output plugin	Use _sourceName, _sourceCategory and _sourceHost attributes in source templates in exporter
Setting host to record["_HOSTNAME"] if record['_SYSTEMD_UNIT'] exists	processors.source.source_host: "%{_HOSTNAME}"
Using namespace, pod, pod_name, pod_id, container, source_host, labels, namespace_labels in source templates	Support all resource attributes plus matching pod_name_key
Setting undefined for non-existing field in source templates	Supported by source processor, not documented
Filtering out records with annotations["sumologic.com/exclude"] set to true	Supported by source processor
Filtering in records with annotations["sumologic.com/include"] set to true	Supported by source processor
Ignoring exclude configuration if annotations["sumologic.com/include"] is set to true	Supported by source processor
Setting log_format to annotations["sumologic.com/format"]	log_format is configured statically in sumologic exporter
Setting sourceHost to annotations["sumologic.com/sourceHost"]	Not supported by source processor
Setting sourceName to annotations["sumologic.com/sourceName"]	Not supported by source processor
Support for per container sourceCategory (precedence over annotations described below)	Supported by source processor
Setting sourceCategory as sourceCategoryPrefix + sourceCategory and replacing - with sourceCategoryReplaceDash	Supported by source processor
overwrite sourceCategory by annotations["sumologic.com/sourceCategory"]	Supported by source processor
overwrite sourceCategoryPrefix by annotations["sumologic.com/sourceCategoryPrefix"]	Supported by source processor
overwrite sourceCategoryReplaceDash by annotations["sumologic.com/sourceCategoryReplaceDash"]	Supported by source processor

sanitized pod name

Sanitized pod name is name portion of the pod. Please consider following examples:

• for a daemonset pod named dset-otelcol-sumo-xa314 it's going to be dset-otelcol-sumo
• for a deployment pod named dep-otelcol-sumo-75675f5861-qasd2 it's going to be dep-otelcol-sumo
• for a statefulset pod named st-otelcol-sumo-0 it's going to be st-otelcol-sumo

fluent-plugin-kubernetes-metadata-filter

Fluentd configuration option	Opentelemetry Kubernetes Processor
annotation_match	processors.k8s_tagger.extract.annotations
de_dot	Behaves like false
watch	Behaves like true
ca_file	No direct translation. Please use processors.k8s_tagger.auth_type: kubeconfig
verify_ssl	No direct translation. Please use processors.k8s_tagger.auth_type: kubeconfig
client_cert	No direct translation. Please use processors.k8s_tagger.auth_type: kubeconfig
client_key	No direct translation. Please use processors.k8s_tagger.auth_type: kubeconfig
bearer_token_file	No direct translation. Please use processors.k8s_tagger.auth_type: kubeconfig
cache_size	N/A
cache_ttl	N/A

fluent-plugin-enhance-k8s-metadata

Fluentd configuration option	Opentelemetry Kubernetes Processor
cache_size	N/A
cache_ttl	N/A
cache_refresh	N/A
cache_refresh_variation	N/A
in_namespace_path	Uses processors.k8s_tagger.pod_association: [{from: build_hostname}]
in_pod_path	Uses processors.k8s_tagger.pod_association: [{from: build_hostname}]
core_api_versions	Supports all
api_groups	Supports all
data_type	N/A

fluent-plugin-events

There is no replacement for fluent-plugin-events in Opentelemetry Collector for now.

Configuration by pipelines

Events

Events are not supported by Opentelemetry Collector

Metrics

Configuration path	Opentelemetry Collector
sumologic.collectionMonitoring	processors.filter
sumologic.metrics.enabled	Respected
sumologic.metrics.metadata.provider	Respected
fluentd.metrics.statefulset.podAntiAffinity	metadata.metrics.statefulset.podAntiAffinity
fluentd.metrics.statefulset.replicaCount	metadata.metrics.statefulset.replicaCount
fluentd.metrics.statefulset.resources	metadata.metrics.statefulset.resources
fluentd.metrics.statefulset.priorityClassName	metadata.metrics.statefulset.priorityClassName
fluentd.metrics.autoscaling.enabled	metadata.metrics.autoscaling.enabled
fluentd.metrics.autoscaling.minReplicas	metadata.metrics.autoscaling.minReplicas
fluentd.metrics.autoscaling.maxReplicas	metadata.metrics.autoscaling.maxReplicas
fluentd.metrics.autoscaling.targetCPUUtilizationPercentage	metadata.metrics.autoscaling.targetCPUUtilizationPercentage
fluentd.metrics.autoscaling.targetMemoryUtilizationPercentage	metadata.metrics.autoscaling.targetMemoryUtilizationPercentage
fluentd.metrics.podDisruptionBudget	metadata.metrics.podDisruptionBudget
fluentd.metrics.enabled	metadata.metrics.enabled
sumologic.collector.sources.metrics	default source is used for all metrics ingestion
fluentd.metrics.extraEnvVars	metadata.metrics.statefulset.extraEnvVars
fluentd.metrics.extraVolumes	metadata.metrics.statefulset.extraVolumes
fluentd.metrics.extraVolumeMounts	metadata.metrics.statefulset.extraVolumeMounts
fluentd.persistence.enabled	metadata.persistence.enabled
fluentd.persistence.storageClass	metadata.persistence.storageClass
fluentd.persistence.accessMode	metadata.persistence.accessMode
fluentd.persistence.size	metadata.persistence.size
fluentd.image.repository	metadata.image.repository
fluentd.image.tag	metadata.image.tag
fluentd.image.pullPolicy	metadata.image.pullPolicy
fluentd.logLevel	metadata.metrics.logLevel
fluentd.logLevelFilter	Not supported. Own logs are being ingested
fluentd.verifySsl	metadata.metrics.config.exporters.sumologic.tls.insecure_skip_verify
fluentd.proxyUri	metadata.metrics.statefulset.extraEnvVars
fluentd.compression.enabled	metadata.metrics.config.exporters.sumologic.compress_encoding
fluentd.compression.encoding	metadata.metrics.config.exporters.sumologic.compress_encoding
fluentd.securityContext.fsGroup	metadata.securityContext.fsGroup
fluentd.buffer	Persistent Queue with File Storage extension is in use, configuration under metadata.metrics.config
otelcol.metrics.enabled	Respected
fluentd.monitoring.input	otelcol.metrics.enabled
fluentd.monitoring.output	otelcol.metrics.enabled
fluentd.metadata.cacheSize	Not supported
fluentd.metadata.cacheTtl	Not supported
fluentd.metadata.cacheRefresh	Not supported
fluentd.metadata.cacheRefreshVariation	Not supported
fluentd.metadata.pluginLogLevel	Not supported
fluentd.metadata.coreApiVersions	Not supported
fluentd.metadata.apiGroups	Not supported
sumologic.collectorName	metadata.metrics.config.processors.source.collector
sumologic.clusterName	metadata.metrics.config.processors.source.collector

Logs

Configuration path	Opentelemetry Collector
sumologic.collectionMonitoring	Respected
sumologic.logs.metadata.provider	Respected
sumologic.collector.sources.logs	Respected
fluentd.persistence.enabled	metadata.persistence.enabled
fluentd.persistence.storageClass	metadata.persistence.storageClass
fluentd.persistence.accessMode	metadata.persistence.accessMode
fluentd.persistence.size	metadata.persistence.size
fluentd.image.repository	metadata.image.repository
fluentd.image.tag	metadata.image.tag
fluentd.image.pullPolicy	metadata.image.pullPolicy
fluentd.logLevel	metadata.logs.logLevel
fluentd.logLevelFilter	Not supported. Own logs are being ingested
fluentd.verifySsl	metadata.logs.config.merge.exporters.sumologic/containers.tls.insecure_skip_verify, metadata.logs.config.merge.exporters.sumologic/systemd.tls.insecure_skip_verify
fluentd.proxyUri	metadata.metrics.statefulset.extraEnvVars
fluentd.compression.enabled	metadata.logs.config.merge.exporters.sumologic/containers.compress_encoding, metadata.logs.config.merge.exporters.sumologic/systemd.compress_encoding
fluentd.compression.encoding	metadata.logs.config.merge.exporters.sumologic/containers.compress_encoding, metadata.logs.config.merge.exporters.sumologic/systemd.compress_encoding
fluentd.securityContext.fsGroup	metadata.securityContext.fsGroup
fluentd.buffer	Persistent Queue with File Storage extension is in use, configuration under metadata.logs.config.merge
otelcol.metrics.enabled	Respected
fluentd.monitoring.input	otelcol.metrics.enabled
fluentd.monitoring.output	otelcol.metrics.enabled
fluentd.metadata.annotation_match	metadata.logs.config.merge.processors.k8s_tagger.extract.annotations
fluentd.metadata.cacheSize	Not supported
fluentd.metadata.cacheTtl	Not supported
fluentd.metadata.cacheRefresh	Not supported
fluentd.metadata.cacheRefreshVariation	Not supported
fluentd.metadata.pluginLogLevel	Not supported
fluentd.metadata.coreApiVersions	Not supported
fluentd.metadata.apiGroups	Not supported
fluentd.logs.enabled	metadata.logs.enabled
fluentd.logs.statefulset.podAntiAffinity	metadata.logs.statefulset.podAntiAffinity
fluentd.logs.statefulset.replicaCount	metadata.logs.statefulset.replicaCount
fluentd.logs.statefulset.resources	metadata.logs.statefulset.resources
fluentd.logs.statefulset.priorityClassName	metadata.logs.statefulset.priorityClassName
fluentd.logs.autoscaling.enabled	metadata.logs.autoscaling.enabled
fluentd.logs.autoscaling.minReplicas	metadata.logs.autoscaling.minReplicas
fluentd.logs.autoscaling.maxReplicas	metadata.logs.autoscaling.maxReplicas
fluentd.logs.autoscaling.targetCPUUtilizationPercentage	metadata.logs.autoscaling.targetCPUUtilizationPercentage
fluentd.logs.autoscaling.targetMemoryUtilizationPercentage	metadata.logs.autoscaling.targetMemoryUtilizationPercentage
fluentd.logs.podDisruptionBudget	metadata.logs.podDisruptionBudget
fluentd.logs.rawConfig	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.input.forwardExtraConf	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.output.addTimestamp	metadata.logs.config.merge.exporters.sumologic/containers.json_logs.add_timestamp, metadata.logs.config.merge.exporters.sumologic/systemd.json_logs.add_timestamp
fluentd.logs.output.timestampKey	metadata.logs.config.merge.exporters.sumologic/containers.json_logs.timestamp_key, metadata.logs.config.merge.exporters.sumologic/systemd.json_logs.timestamp_key
fluentd.logs.output.pluginLogLevel	Not supported
fluentd.logs.output.extraConf	metadata.logs.config.merge.exporters.sumologic/containers, metadata.logs.config.merge.exporters.sumologic/systemd
fluentd.logs.extraLogs	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.containers.overrideRawConfig	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.containers.outputConf	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.containers.overrideOutputConf	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
sumologic.collectorName	metadata.logs.config.merge.processors.source/containers.collector, metadata.logs.config.merge.processors.source/systemd.collector, metadata.logs.config.merge.processors.source/kubelet.collector
sumologic.clusterName	metadata.logs.config.merge.processors.source/containers.collector, metadata.logs.config.merge.processors.source/systemd.collector, metadata.logs.config.merge.processors.source/kubelet.collector
fluentd.logs.containers.k8sMetadataFilter.watch	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.k8sMetadataFilter.caFile	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.k8sMetadataFilter.verifySsl	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.k8sMetadataFilter.clientCert	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.k8sMetadataFilter.clientKey	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.k8sMetadataFilter.bearerTokenFile	metadata.logs.config.merge.processors.k8s_tagger.auth_type
fluentd.logs.containers.extraFilterPluginConf	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null
fluentd.logs.containers.extraOutputPluginConf	metadata.logs.config.merge; mind that configuration is going to be merged unless you use null

Other

Configuration path	Opentelemetry Collector
otelcol.logLevelFilter	Not respected
otelcol.metrics.enabled	Respected