Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] TSDB: need metric_type mapping for dynamic templates #155004

Closed
lalit-satapathy opened this issue Apr 17, 2023 · 54 comments · Fixed by #159789 or #160417
Closed

[Fleet] TSDB: need metric_type mapping for dynamic templates #155004

lalit-satapathy opened this issue Apr 17, 2023 · 54 comments · Fixed by #159789 or #160417
Assignees
Labels
QA:Validated Issue has been validated by QA Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@lalit-satapathy
Copy link

lalit-satapathy commented Apr 17, 2023

We have multiple integrations which use the dynamic templates for field mappings. The Fleet generated mapping for the dynamic templates is well described in the issue here.

In TSDB to annotate, metric_type the requirement from elasticsearch is to:

To mark a field as a metric, you must specify a metric type using the time_series_metric mapping parameter.

This mapping is currently done via metric_type in fields.yml file.

The issue is enabling the same for dynamic templates. How would the metric_type work for the fields with dynamic templates? Based on the elastic search documentation, above I am assuming, an explicit metric_type mapping is needed for dynamic templates. Can Fleet does the mapping for the same?

For example, Fleet can respects a (metric_type: counter) as part of the dynamic templates or provide any similar alternative usage.

- name: cockroachdb.status.*.counter
  type: object
  object_type: double
  metric_type: counter
  object_type_mapping_type: "*"
  description: >
    Prometheus counter metric

This requirement impacts o11y TSDB migration multiple Prometheus style packages. Short-term testing solution for testing is to add this manually in a custom template like below:

{
  "dynamic_templates": [
    {
      "cockroachdb.status.*.counter": {
        "path_match": "cockroachdb.status.*.counter",
        "mapping": {
          "time_series_metric": "counter",
          "type": "double"
        }
      }
    }
  ]
}

Packages affected:

  • Istio
  • System
  • Prometheus

@ruflin @andresrc @joshdover @kpollich

@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 17, 2023
@ruflin ruflin added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 17, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@jsoriano
Copy link
Member

- name: cockroachdb.status.*.counter
  type: object
  object_type: double
  metric_type: counter
  object_type_mapping_type: "*"
  description: >
    Prometheus counter metric

This syntax may not be valid according to the spec after elastic/package-spec#525.

Alternatively, this syntax might work from the spec side:

- name: cockroachdb.status.*.counter
  type: double
  metric_type: counter
  description: >
    Prometheus counter metric

@juliaElastic
Copy link
Contributor

@mrodm Is a package-spec change needed for this? I see you created a draft pr.

@juliaElastic
Copy link
Contributor

juliaElastic commented May 24, 2023

If I understand correctly, the kibana change needed is to extend the logic for dynamic mappings here, to replace metric_type with time_series_metric like it is done for regular fields here.

@lalit-satapathy Could you confirm if we can use these steps to verify the kibana change in this issue:

  1. Add metric_type: gauge to the Prometheus gauge metric here
  2. Build a new version of the prometheus package
  3. Install the new package in fleet (upload zip)
  4. Check the created component template mappings (metrics-prometheus.collector@package) that it contains "time_series_metric": "gauge" in dynamic_templates
{
  "dynamic_templates": [
    {
      "prometheus.*.value": {
        "path_match": "prometheus.*.value",
        "mapping": {
          "time_series_metric": "gauge",
          "type": "double"
        }
      }
    }
  ]
}

@mrodm
Copy link
Contributor

mrodm commented May 24, 2023

@mrodm Is a package-spec change needed for this? I see you created a draft pr.

@juliaElastic That PR is an attempt to add more validation checks for those keys, but it should not be required for this issue. Currently, it should be possible to define the fields as detailed in #155004 (comment)

@constanca-m
Copy link

constanca-m commented Jun 14, 2023

Alternatively, this syntax might work from the spec side:

- name: cockroachdb.status.*.counter
  type: double
  metric_type: counter
  description: >
    Prometheus counter metric

Hey @jsoriano, I applied your solution:

- name: istio.istiod.metrics.*.counter
  type: double
  metric_type: counter
  description: >
    Istiod counter metric

But the field name in vertical axis disappears:
image
Any idea why?

The mapping for the field is with TSDB enabled is:

"dynamic_templates": [
      {
        "istio.istiod.metrics.*.counter": {
          "path_match": "istio.istiod.metrics.*.counter",
          "match_mapping_type": "double",
          "mapping": {
            "type": "double"
          }
        }
      },
     ...
  ],
  ...
              "metrics": {
                "properties": {
                  "pilot_inbound_updates": {
                    "properties": {
                      "counter": {
                        "type": "long"
                      },
 ...

For TSDB disabled is:

      {
        "istio.istiod.metrics.*.counter": {
          "path_match": "istio.istiod.metrics.*.counter",
          "mapping": {
            "type": "double"
          }
        }
      },
...
                  "pilot_inbound_updates": {
                    "properties": {
                      "counter": {
                        "type": "double"
                      },

Conclusion: the "match_mapping_type": "double" was adding with the change, metric_type was not. Visualization works, but vertical axis field name disappears.

Also applied @juliaElastic suggestion, and it did not work. Mapping for the field stayed the same as for TSDB disabled:

- name: istio.istiod.metrics.*.counter
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: counter
  description: >
    Istiod counter metric

Resulting mapping:

      {
        "istio.istiod.metrics.*.counter": {
          "path_match": "istio.istiod.metrics.*.counter",
          "mapping": {
            "type": "double"
          }
        }
      },
...
                  "pilot_inbound_updates": {
                    "properties": {
                      "counter": {
                        "type": "double"
                      },

@mlunadia
Copy link

@jlind23 We have a few packages (Prometheus server, Istio, Airflow, CockroachDB, Redisenterprise) that are currently blocked for migration to TSDB due to this issue, any support you can provide is much welcomed

@juliaElastic
Copy link
Contributor

@constanca-m We have to make a change in Kibana/Fleet to apply the metric_type mapping in dynamic templates.

The issue is in our current sprint and not started yet.
@criamico Do you think you will have time to work on this? It should be a small effort.

@tetianakravchenko
Copy link
Contributor

@mrodm

@mrodm Is a package-spec change needed for this? I see you created a draft pr.

@juliaElastic That PR is an attempt to add more validation checks for those keys, but it should not be required for this issue. Currently, it should be possible to define the fields as detailed in #155004 (comment)

@constanca-m tested suggested syntax, but it doesn't work and the visualization that use metric defined with the new syntax is broken. Should there be added a support for metric_type on the package-spec side?

@jlind23
Copy link
Contributor

jlind23 commented Jun 14, 2023

After discussing with @criamico, she agreed on taking a stab at it as soon as possible

@criamico
Copy link
Contributor

Currently working on this, I have a question about the change described in #155004 (comment).

Should this mapping only be added if tsdb is enabled? I'm going to replace metric_type with time_series_metric but I'm guessing it should happen only when tsdb is enabled, but I'm looking for confirmation.

@tetianakravchenko
Copy link
Contributor

@criamico

Should this mapping only be added if tsdb is enabled? I'm going to replace metric_type with time_series_metric but I'm guessing it should happen only when tsdb is enabled, but I'm looking for confirmation.

Yes, similar to the metric_type for other field type that do not use dynamic_template - time_series_metric will be added only in case tsdb is enabled

criamico added a commit that referenced this issue Jun 20, 2023
Closes #155004

## Summary
Add dynamic mapping for tsdb `time_series_metric`. The logic is
replacing `metric_type` with `time_series_metric` in dynamic fields that
have a wildcard in the name.

<img width="2599" alt="Screenshot 2023-06-20 at 15 33 37"
src="https://github.com/elastic/kibana/assets/16084106/3265ae5c-3416-4e16-a017-9c3cd3abb4ac">



### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@juliaElastic juliaElastic added the QA:Needs Validation Issue needs to be validated by QA label Jun 21, 2023
@juliaElastic
Copy link
Contributor

@tetianakravchenko @lalit-satapathy The Fleet implementation for this was merged yesterday, could you do testing on your side? It is available in the latest kibana 8.9.0-SNAPSHOT.

@constanca-m
Copy link

@juliaElastic I added this to the mapping of an object:

- name: istio.istiod.metrics.*.counter
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: counter
  description: >
    Istiod counter metric

And used version 8.9.0 Snapshot just now. The mapping is:

      {
        "istio.istiod.metrics.*.counter": {
          "path_match": "istio.istiod.metrics.*.counter",
          "mapping": {
            "type": "double"
          }
        }
      },

So we can't see the time series metric.

@juliaElastic
Copy link
Contributor

The fix for group type is merged and backported to 8.9, should be available in the next snapshot.

@juliaElastic
Copy link
Contributor

@tetianakravchenko @lalit-satapathy Are all the issues resolved with metric_type based on your testing, can we close the issue?

@juliaElastic
Copy link
Contributor

@amolnater-qasource To test this feature:

  • install prometheus-1.6.0 package
  • Go to Stack Management, Index Templates, look at metrics-prometheus.collector
  • Go to Preview tab and verify that the dynamic_templates section contains double fields with time_series_metric in mapping.
image

@amolnater-qasource
Copy link

Thank you for the details @juliaElastic

@lalit-satapathy
Copy link
Author

@tetianakravchenko @lalit-satapathy Are all the issues resolved with metric_type based on your testing, can we close the issue?

@tetianakravchenko Please comment/close this issue if dynamic mapping is resolved in Prometheus.

@tetianakravchenko
Copy link
Contributor

tetianakravchenko commented Aug 4, 2023

@tetianakravchenko @lalit-satapathy Are all the issues resolved with metric_type based on your testing, can we close the issue?

@tetianakravchenko Please comment/close this issue if dynamic mapping is resolved in Prometheus.

@juliaElastic @lalit-satapathy
The only issue I am still facing is regarding the dynamic_templates itself:

  • dynamic template seems to be added correctly:
"dynamic_templates": [
...
      {
        "prometheus.*.counter": {
          "path_match": "prometheus.*.counter",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "counter",
            "type": "double"
          }
        }
      },
...

but in the prometheus properties I see that not all metrics include expected mapping configuration (time_series_metric and correct type), some metrics does not include time_series_metric and use type: long instead of double:
Screenshot 2023-08-04 at 15 04 09

full mapping:

{
  "mappings": {
    "_meta": {
      "managed_by": "fleet",
      "managed": true,
      "package": {
        "name": "prometheus"
      }
    },
    "_data_stream_timestamp": {
      "enabled": true
    },
    "dynamic_templates": [
      {
        "container.labels": {
          "path_match": "container.labels.*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "prometheus.labels.*": {
          "path_match": "prometheus.labels.*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "prometheus.metrics.*": {
          "path_match": "prometheus.metrics.*",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "gauge",
            "type": "double"
          }
        }
      },
      {
        "prometheus.*.value": {
          "path_match": "prometheus.*.value",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "gauge",
            "type": "double"
          }
        }
      },
      {
        "prometheus.*.counter": {
          "path_match": "prometheus.*.counter",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "counter",
            "type": "double"
          }
        }
      },
      {
        "prometheus.*.rate": {
          "path_match": "prometheus.*.rate",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "gauge",
            "type": "double"
          }
        }
      },
      {
        "prometheus.*.histogram": {
          "path_match": "prometheus.*.histogram",
          "mapping": {
            "type": "histogram"
          }
        }
      },
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "ephemeral_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "cloud": {
        "properties": {
          "account": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              }
            }
          },
          "availability_zone": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "image": {
            "properties": {
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "instance": {
            "properties": {
              "id": {
                "type": "keyword",
                "time_series_dimension": true
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "machine": {
            "properties": {
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "project": {
            "properties": {
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "provider": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "region": {
            "type": "keyword",
            "time_series_dimension": true
          }
        }
      },
      "container": {
        "properties": {
          "id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "image": {
            "properties": {
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword",
            "value": "prometheus.collector"
          },
          "namespace": {
            "type": "constant_keyword",
            "value": "default"
          },
          "type": {
            "type": "constant_keyword",
            "value": "metrics"
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "elastic_agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "snapshot": {
            "type": "boolean"
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "agent_id_status": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "dataset": {
            "type": "constant_keyword",
            "value": "prometheus.collector"
          },
          "duration": {
            "type": "long"
          },
          "ingested": {
            "type": "date",
            "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
          },
          "module": {
            "type": "constant_keyword",
            "value": "prometheus"
          }
        }
      },
      "host": {
        "properties": {
          "architecture": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "containerized": {
            "type": "boolean"
          },
          "domain": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          },
          "mac": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "os": {
            "properties": {
              "build": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "codename": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "family": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "kernel": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024,
                "fields": {
                  "text": {
                    "type": "text"
                  }
                }
              },
              "platform": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "metricset": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "period": {
            "type": "long"
          }
        }
      },
      "prometheus": {
        "properties": {
          "go_gc_duration_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_gc_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_gc_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_goroutines": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "go_info": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "go_memstats_alloc_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_alloc_bytes_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_memstats_buck_hash_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_frees_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_memstats_gc_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_heap_alloc_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_heap_idle_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_heap_inuse_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_heap_objects": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "go_memstats_heap_released_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_heap_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_last_gc_time_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_lookups_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_memstats_mallocs_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "go_memstats_mcache_inuse_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "go_memstats_mcache_sys_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "go_memstats_mspan_inuse_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_mspan_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_next_gc_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_other_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_stack_inuse_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_stack_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_memstats_sys_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "go_threads": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "labels": {
            "properties": {
              "alertmanager": {
                "type": "keyword"
              },
              "branch": {
                "type": "keyword"
              },
              "call": {
                "type": "keyword"
              },
              "code": {
                "type": "keyword"
              },
              "config": {
                "type": "keyword"
              },
              "dialer_name": {
                "type": "keyword"
              },
              "endpoint": {
                "type": "keyword"
              },
              "event": {
                "type": "keyword"
              },
              "goarch": {
                "type": "keyword"
              },
              "goos": {
                "type": "keyword"
              },
              "goversion": {
                "type": "keyword"
              },
              "handler": {
                "type": "keyword"
              },
              "instance": {
                "type": "keyword"
              },
              "interval": {
                "type": "keyword"
              },
              "job": {
                "type": "keyword"
              },
              "listener_name": {
                "type": "keyword"
              },
              "name": {
                "type": "keyword"
              },
              "quantile": {
                "type": "keyword"
              },
              "queue_name": {
                "type": "keyword"
              },
              "reason": {
                "type": "keyword"
              },
              "revision": {
                "type": "keyword"
              },
              "role": {
                "type": "keyword"
              },
              "scrape_job": {
                "type": "keyword"
              },
              "slice": {
                "type": "keyword"
              },
              "status_code": {
                "type": "keyword"
              },
              "tags": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              },
              "version": {
                "type": "keyword"
              }
            }
          },
          "labels_id": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "net_conntrack_dialer_conn_attempted_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "net_conntrack_dialer_conn_closed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "net_conntrack_dialer_conn_established_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "net_conntrack_dialer_conn_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "net_conntrack_listener_conn_accepted_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "net_conntrack_listener_conn_closed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "process_cpu_seconds_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "process_max_fds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "process_open_fds": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "process_resident_memory_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "process_start_time_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "process_virtual_memory_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "process_virtual_memory_max_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_api_remote_read_queries": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_build_info": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_config_last_reload_success_timestamp_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_config_last_reload_successful": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_queries": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_queries_concurrent_max": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_query_duration_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_engine_query_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_query_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_query_log_enabled": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_engine_query_log_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_http_request_duration_seconds": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_http_requests_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_http_response_size_bytes": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_notifications_alertmanagers_discovered": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_notifications_dropped_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_notifications_errors_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_notifications_queue_capacity": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_notifications_queue_length": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_notifications_sent_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_ready": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_remote_storage_exemplars_in_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_remote_storage_highest_timestamp_in_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_remote_storage_histograms_in_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_remote_storage_samples_in_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_remote_storage_string_interner_zero_reference_releases_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_rule_evaluation_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_rule_evaluation_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_rule_group_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_rule_group_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_azure_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_consul_rpc_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_consul_rpc_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_consul_rpc_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_discovered_targets": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_dns_lookup_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_dns_lookups_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_failed_configs": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_file_read_errors_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_file_scan_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_file_scan_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_file_watcher_errors_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_http_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_events_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_http_request_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_http_request_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_http_request_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_depth": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_items_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_latency_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_latency_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_longest_running_processor_seconds": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_unfinished_work_seconds": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_work_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kubernetes_workqueue_work_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kuma_fetch_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kuma_fetch_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kuma_fetch_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_kuma_fetch_skipped_updates_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_linode_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_nomad_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_received_updates_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_sd_updates_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_interval_length_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_target_interval_length_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_interval_length_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_metadata_cache_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_target_metadata_cache_entries": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_exceeded_label_limits_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_exceeded_target_limit_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_reloads_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_reloads_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_sync_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_target_limit": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pool_targets": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pools_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrape_pools_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_cache_flush_forced_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_exceeded_body_size_limit_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_exceeded_sample_limit_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_exemplar_out_of_order_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_sample_duplicate_timestamp_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_sample_out_of_bounds_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_scrapes_sample_out_of_order_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_sync_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_sync_length_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_target_sync_length_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_target_sync_length_seconds_sum": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_template_text_expansion_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_template_text_expansions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_treecache_watcher_goroutines": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_treecache_zookeeper_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_blocks_loaded": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_checkpoint_creations_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_checkpoint_creations_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_checkpoint_deletions_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_checkpoint_deletions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_clean_start": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_compaction_chunk_range_seconds": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_tsdb_compaction_chunk_samples": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_tsdb_compaction_chunk_size_bytes": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_tsdb_compaction_duration_seconds": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_tsdb_compaction_populating_block": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_compactions_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_compactions_skipped_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_compactions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_compactions_triggered_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_data_replay_duration_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_exemplar_exemplars_appended_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_exemplar_exemplars_in_storage": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_exemplar_max_exemplars": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_exemplar_out_of_order_exemplars_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_exemplar_series_with_exemplars_in_storage": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_active_appenders": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_chunks": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_chunks_created_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_chunks_removed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_chunks_storage_size_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_head_gc_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_gc_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_max_time": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_head_max_time_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_head_min_time": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_head_min_time_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_head_out_of_order_samples_appended_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_samples_appended_total": {
            "properties": {
              "counter": {
                "type": "double",
                "time_series_metric": "counter"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_series": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_series_created_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_series_not_found_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_series_removed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_truncations_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_head_truncations_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_isolation_high_watermark": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_isolation_low_watermark": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_lowest_timestamp": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_lowest_timestamp_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_mmap_chunk_corruptions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_out_of_bound_samples_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_out_of_order_samples_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_reloads_failures_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_reloads_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_retention_limit_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_size_retentions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_snapshot_replay_error_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_storage_blocks_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_symbol_table_size_bytes": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_time_retentions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_tombstone_cleanup_seconds": {
            "properties": {
              "histogram": {
                "type": "histogram"
              }
            }
          },
          "prometheus_tsdb_too_old_samples_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_vertical_compactions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_completed_pages_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_corruptions_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_fsync_duration_seconds": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_wal_fsync_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_fsync_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_page_flushes_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_segment_current": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_storage_size_bytes": {
            "properties": {
              "value": {
                "type": "double",
                "time_series_metric": "gauge"
              }
            }
          },
          "prometheus_tsdb_wal_truncate_duration_seconds_count": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_truncate_duration_seconds_sum": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_truncations_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_truncations_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_tsdb_wal_writes_failed_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_web_federation_errors_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "prometheus_web_federation_warnings_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "promhttp_metric_handler_requests_in_flight": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          },
          "promhttp_metric_handler_requests_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },
          "up": {
            "properties": {
              "value": {
                "type": "long"
              }
            }
          }
        }
      },
      "service": {
        "properties": {
          "address": {
            "type": "keyword",
            "time_series_dimension": true
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}

I am not sure if this issue is related to the fleet or elasticsearch itself

@joshdover
Copy link
Contributor

@tetianakravchenko I think you're missing a dynamic_template that matches on type long?

@tetianakravchenko
Copy link
Contributor

@joshdover there is no dynamic_template that has long type. In my previous message I've added full mapping under Details

@lalit-satapathy
Copy link
Author

@tetianakravchenko are you referring to the metric in the mapping given below or some other metric?

 "go_memstats_lookups_total": {
            "properties": {
              "counter": {
                "type": "long"
              },
              "rate": {
                "type": "long"
              }
            }
          },

Where this mapping comes from?

@tetianakravchenko
Copy link
Contributor

tetianakravchenko commented Aug 7, 2023

@lalit-satapathy

@tetianakravchenko are you referring to the metric in the mapping given below or some other metric?

yes, this is one of the metric, that has not expected mapping. In my comment above, you can find full mapping and notice that not all metrics unde mappings.properties.prometheus have the same (expected) mapping

Where this mapping comes from?

I believe it defined dynamically, since default configuration is dynamic: true

@juliaElastic
Copy link
Contributor

juliaElastic commented Aug 7, 2023

@tetianakravchenko Could you help document the steps to reproduce the issue? I'm not seeing those go_memstats* fields in index template mappings, are they created when we ingest data?

@tetianakravchenko
Copy link
Contributor

@juliaElastic

@tetianakravchenko Could you help document the steps to reproduce the issue? I'm not seeing those go_memstats* fields in index template mappings, are they created when we ingest data?

yes, those fields are added when the data is ingested. Those metrics are scraped from the prometheus server

To reproduce it, you need to install prometheus server and use it as a host for the prometheus.collector datastream configuration

@tetianakravchenko
Copy link
Contributor

@juliaElastic I think I found the issue.
initial field definition:

- name: prometheus.*.counter
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: counter
  description: >
    Prometheus gauge metric

this configuration would generate dynamic_template:

{
        "prometheus.*.counter": {
          "path_match": "prometheus.*.counter",
          "match_mapping_type": "*",
          "mapping": {
            "time_series_metric": "counter",
            "type": "double"
          }
        }
      },

but the configuration above is not supported by the elastic-package, so we switched to this field configuration:

- name: prometheus.*.counter
  type: double
  metric_type: gauge
  description: |
    Prometheus metric

in the end it generate this dynamic_template:

{
        "prometheus.*.counter": {
          "path_match": "prometheus.*.counter",
          "match_mapping_type": "double",
          "mapping": {
            "time_series_metric": "counter",
            "type": "double"
          }
        }
      },

Note the difference of match_mapping_type

So it is expected that at dynamic_template is not applied for all fields.

@lalit-satapathy I think we should switch back to the format:

- name: prometheus.*.counter
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: counter
  description: >
    Prometheus gauge metric

@tetianakravchenko
Copy link
Contributor

As was discussed with @lalit-satapathy:

  • we agreed to switch back to the previous format:
- name: prometheus.*.counter
  type: object
  object_type: double
  object_type_mapping_type: "*"
  metric_type: counter
  description: >
    Prometheus gauge metric
  • this is not yet supported by the latest elastic-package version (0.84.0) and it fails with error:
file "/Users/tetianakravchenko/go/src/github.com/elastic/integrations/build/packages/prometheus-1.8.0.zip/data_stream/collector/fields/fields.yml" is invalid: field 1.type: 1.type must be one of the following: "histogram", "aggregate_metric_double", "long", "integer", "short", "byte", "double", "float", "half_float", "scaled_float", "unsigned_long"

@harshitgupta-qasource
Copy link

Hi Team,

We have executed 05 testcases under the Feature test run for the 8.10.0 release at the link:

Status:

  • PASS: 05

Build details:
VERSION: 8.10.0 BC4
BUILD: 66329
COMMIT: 913dd8b

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks

@harshitgupta-qasource harshitgupta-qasource added QA:Validated Issue has been validated by QA and removed QA:Needs Validation Issue needs to be validated by QA labels Sep 6, 2023
@lalit-satapathy
Copy link
Author

Please let us know if anything else is required from our end.

We have not seen any new issues on this either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
QA:Validated Issue has been validated by QA Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet