Several of the new collectors fail with incorrect data types #818

sclu1034 · 2023-06-22T08:40:44Z

What did you do?
Updated to v0.13.0 and removed queries.yaml in favor of the built-in collectors.

What did you expect to see?
An error-free log and the same metrics to be reported as before.

What did you see instead? Under which circumstances?

Repeating entries of the logs shown below and affected metrics, such as pg_postmaster_start_time_seconds, stopped being reported.
No special circumstances, I just started the tool via quay.io/prometheuscommunity/postgres-exporter:latest with the exact same setup as the previous image.

Environment

postgres_exporter version:

postgres_exporter, version 0.13.0 (branch: HEAD, revision: f550f8f8ad77e054f12eb3d3f6605edc037573f5)
  build user:       root@de207ed6b751
  build date:       20230622-07:05:45
  go version:       go1.20.5
  platform:         linux/amd64
  tags:             netgo static_build

postgres_exporter flags: Default Docker CMD line
PostgreSQL version: 15.3
Logs:

ts=2023-06-22T08:25:13.132Z caller=proc.go:250 msg="Excluded databases" databases=[]
ts=2023-06-22T08:25:13.142Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9187
ts=2023-06-22T08:25:13.142Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9187
ts=2023-06-22T08:25:17.855Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=postgres:5432
ts=2023-06-22T08:25:17.860Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.004954557 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"
ts=2023-06-22T08:25:17.861Z caller=postgres_exporter.go:622 level=info msg="Semantic version changed" server=postgres:5432 from=0.0.0 to=15.3.0
ts=2023-06-22T08:25:17.864Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.008552172 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:17.865Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.009474908 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"
ts=2023-06-22T08:25:17.867Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.011338687 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:32.855Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.000977326 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:32.857Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.002956973 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"
ts=2023-06-22T08:25:32.859Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.00465972 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:32.859Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.005185534 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"
ts=2023-06-22T08:25:47.856Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.001654959 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:47.857Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.002426217 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-19 13:22:47.713957 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-22T08:25:47.857Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.00275669 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"
ts=2023-06-22T08:25:47.859Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.004420991 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"

The text was updated successfully, but these errors were encountered:

SuperQ · 2023-06-22T10:29:14Z

CC @Sticksman

reneeckstein · 2023-06-22T15:53:55Z

can confirm the same issue on PostgreSQL 14.7 (AWS Aurora Postgres)

Art-Sel · 2023-06-23T14:23:50Z

same problem on 14.2

postgres_exporter, version 0.13.0 (branch: master, revision: f550f8f)
build user: user@lenovo
build date: 20230623-13:42:32
go version: go1.18.3
platform: linux/amd64
tags: netgo static_build

ts=2023-06-23T14:19:50.408Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/run/postgresql/:5432
ts=2023-06-23T14:19:50.412Z caller=postgres_exporter.go:622 level=info msg="Semantic version changed" server=/var/run/postgresql/:5432 from=0.0.0 to=14.2.0
ts=2023-06-23T14:19:50.421Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.012660037 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"
ts=2023-06-23T14:19:50.426Z caller=collector.go:190 level=error msg="collector failed" name=statio_user_tables duration_seconds=0.017781942 err="sql: Scan error on column index 5, name \"idx_blks_read\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:19:50.426Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.018095918 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:19:50.430Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.021661276 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"
ts=2023-06-23T14:19:50.432Z caller=collector.go:190 level=error msg="collector failed" name=stat_user_tables duration_seconds=0.024134202 err="sql: Scan error on column index 5, name \"idx_scan\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:19:50.434Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.025337841 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:19:51.237Z caller=collector.go:190 level=error msg="collector failed" name=statio_user_tables duration_seconds=0.012568834 err="sql: Scan error on column index 5, name \"idx_blks_read\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:19:51.239Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.014648358 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"
ts=2023-06-23T14:19:51.242Z caller=collector.go:190 level=error msg="collector failed" name=stat_user_tables duration_seconds=0.017622653 err="sql: Scan error on column index 5, name \"idx_scan\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:19:51.243Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.017891161 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:19:51.244Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.019340401 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:19:51.245Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.020069416 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"
ts=2023-06-23T14:20:39.361Z caller=collector.go:190 level=error msg="collector failed" name=statio_user_tables duration_seconds=0.012915492 err="sql: Scan error on column index 5, name \"idx_blks_read\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:20:39.363Z caller=collector.go:190 level=error msg="collector failed" name=process_idle duration_seconds=0.014780197 err="sql: Scan error on column index 3, name \"seconds\": unsupported Scan, storing driver.Value type []uint8 into type *[]int64"
ts=2023-06-23T14:20:39.363Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.015138929 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:20:39.365Z caller=collector.go:190 level=error msg="collector failed" name=stat_user_tables duration_seconds=0.017022539 err="sql: Scan error on column index 5, name \"idx_scan\": converting NULL to int64 is unsupported"
ts=2023-06-23T14:20:39.367Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.01827618 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-23 16:48:04.349714 +0300 MSK\") to a float64: invalid syntax"
ts=2023-06-23T14:20:39.369Z caller=collector.go:190 level=error msg="collector failed" name=stat_database duration_seconds=0.021057372 err="sql: Scan error on column index 1, name \"datname\": converting NULL to string is unsupported"

Sticksman · 2023-06-24T20:11:00Z

Taking a look

moonchildua · 2023-06-26T11:44:34Z

same problem with 12.5

SuperQ · 2023-06-26T11:52:08Z

If anyone wants to test these build artifacts from the #823 that would help.

sclu1034 · 2023-06-27T13:58:52Z

This is not fixed!

Just updated to v0.13.1. There are less failures, but some of the previous ones still exist.
Specifically, ones that have no relation to NULL values:

ts=2023-06-27T13:56:02.460Z caller=proc.go:250 msg="Excluded databases" databases=[]
ts=2023-06-27T13:56:02.471Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9187
ts=2023-06-27T13:56:02.471Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9187
ts=2023-06-27T13:56:02.860Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=postgres:5432
ts=2023-06-27T13:56:02.868Z caller=postgres_exporter.go:622 level=info msg="Semantic version changed" server=postgres:5432 from=0.0.0 to=15.3.0
ts=2023-06-27T13:56:02.876Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.015816516 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-27T13:56:02.878Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.01693146 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-27T13:56:17.863Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.002038289 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-27T13:56:17.864Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.003329763 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-27T13:56:32.863Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.002265238 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"
ts=2023-06-27T13:56:32.865Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.004408598 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-27 11:39:22.518996 +0200 CEST\") to a float64: invalid syntax"

mrjones-plip · 2023-06-29T21:37:02Z

@sclu1034 - if you try running on master, is it fixed? I found that running on latest (v0.13.1) it wasn't fixed but that on master it was. I run in docker compose, so for me this was image: prometheuscommunity/postgres-exporter:master

I wonder if v0.13.1 didn't have the fix included in the PR somehow 🤷

Sticksman · 2023-06-30T07:07:13Z

Yeah I think we got the next set of fixes in after 0.13.1 got released

sclu1034 · 2023-06-30T07:09:48Z

I wonder if v0.13.1 didn't have the fix included in the PR somehow 🤷

@mrjones-plip The PR was opened several hours after that version was released, so, logically, it is not part of that tag.
The other errors you mention at medic/cht-watchdog#70 were fixed in an earlier PR: #823. And as you can tell from the changelog, that's the one that did make it into v1.13.1.

PriyankaKotwal2019 · 2023-07-17T08:11:19Z

Is this issue really fixed? If I use image: prometheuscommunity/postgres-exporter:master I am getting below error for postgress 14.7

ts=2023-07-17T08:07:08.498Z caller=postgres_exporter.go:613 level=info msg="Semantic version changed" server=matchbox-db.qa.paymentandmatching.services.otto-payments.de:5432 from=0.0.0 to=14.7.0 ts=2023-07-17T08:07:08.499Z caller=collector.go:190 level=error msg="collector failed" name=wal duration_seconds=0.021305146 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2023-07-17T08:07:12.898Z caller=collector.go:190 level=error msg="collector failed" name=wal duration_seconds=0.021232623 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2023-07-17T08:07:38.477Z caller=collector.go:190 level=error msg="collector failed" name=wal duration_seconds=0.001553542 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2023-07-17T08:07:42.897Z caller=collector.go:190 level=error msg="collector failed" name=wal duration_seconds=0.020496939 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported"

and if I use latest tag as image:prometheuscommunity/postgres-exporter:v0.13.1 then getting following error

ts=2023-07-17T08:09:51.241Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.027859128 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-13 00:23:42.991597 +0000 UTC\") to a float64: invalid syntax" ts=2023-07-17T08:09:51.245Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.031713436 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-13 00:23:42.991597 +0000 UTC\") to a float64: invalid syntax" ts=2023-07-17T08:09:51.247Z caller=postgres_exporter.go:622 level=info msg="Semantic version changed" server=matchbox-db.qa.paymentandmatching.services.otto-payments.de:5432 from=0.0.0 to=14.7.0 ts=2023-07-17T08:09:58.418Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.002188862 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-13 00:23:42.991597 +0000 UTC\") to a float64: invalid syntax" ts=2023-07-17T08:09:58.420Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.003781537 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-06-13 00:23:42.991597 +0000 UTC\") to a float64: invalid syntax"

craigvanaman · 2023-07-17T08:57:31Z

I've still seeing this on postgres-exporter:v0.13.1.

ts=2023-07-17T08:53:35.791Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.007044365 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-07-13 13:58:54.054636 +0000 UTC\") to a float64: invalid syntax"

reneeckstein · 2023-07-17T16:13:57Z

someone needs to release a newer version

osmanbv · 2023-07-19T15:00:43Z

we have still seeing issue. If you find any solution or release new version, can you update here

sysadmind · 2023-07-24T00:14:13Z

0.13.2 has been released

steklopod · 2023-08-07T12:46:36Z

I can confirm that this works:

  postgres_exporter:
    image: prometheuscommunity/postgres-exporter:v0.13.2

marekaf · 2023-11-15T08:21:29Z

I'm still seeing this in v0.15.0

mazinas · 2024-06-18T05:59:43Z

Still seeing this in v0.15.0

ts=2024-06-17T14:37:24.076Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.036868834 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2024-06-17T14:37:50.903Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.002791467 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2024-06-17T14:37:53.758Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.021771771 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2024-06-17T14:52:43.817Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.031722307 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2024-06-18T05:10:40.257Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.039104393 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported" ts=2024-06-18T05:12:17.698Z caller=collector.go:202 level=error msg="collector failed" name=wal duration_seconds=0.01223086 err="sql: Scan error on column index 1, name \"size\": converting NULL to uint64 is unsupported"

Electronn · 2024-09-23T20:26:44Z

Had the same errors with AWS PostgreSQL cluster. Found a way of fixing. Bugfix in PR - #1070

Sticksman mentioned this issue Jun 24, 2023

Bug fix: Make collector not fail on null values #823

Merged

SuperQ closed this as completed in #823 Jun 27, 2023

sysadmind reopened this Jun 27, 2023

Sticksman mentioned this issue Jun 27, 2023

Bug Fix: Fix lingering type issues #828

Merged

SuperQ closed this as completed in #828 Jun 27, 2023

mrjones-plip mentioned this issue Jun 29, 2023

postgres/prometheus couch2pg errors: 'converting NULL to int64 is unsupported' and more medic/cht-watchdog#70

Closed

yqlu mentioned this issue Jul 19, 2023

Updated the PostgreSQL Prometheus documentation and dashboard GoogleCloudPlatform/monitoring-dashboard-samples#581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Several of the new collectors fail with incorrect data types #818

Several of the new collectors fail with incorrect data types #818

sclu1034 commented Jun 22, 2023 •

edited

Loading

SuperQ commented Jun 22, 2023

reneeckstein commented Jun 22, 2023

Art-Sel commented Jun 23, 2023 •

edited

Loading

Sticksman commented Jun 24, 2023

moonchildua commented Jun 26, 2023

SuperQ commented Jun 26, 2023

sclu1034 commented Jun 27, 2023 •

edited

Loading

mrjones-plip commented Jun 29, 2023

Sticksman commented Jun 30, 2023

sclu1034 commented Jun 30, 2023

PriyankaKotwal2019 commented Jul 17, 2023

craigvanaman commented Jul 17, 2023

reneeckstein commented Jul 17, 2023

osmanbv commented Jul 19, 2023

sysadmind commented Jul 24, 2023

steklopod commented Aug 7, 2023

marekaf commented Nov 15, 2023

mazinas commented Jun 18, 2024

Electronn commented Sep 23, 2024 •

edited

Loading

Several of the new collectors fail with incorrect data types #818

Several of the new collectors fail with incorrect data types #818

Comments

sclu1034 commented Jun 22, 2023 • edited Loading

SuperQ commented Jun 22, 2023

reneeckstein commented Jun 22, 2023

Art-Sel commented Jun 23, 2023 • edited Loading

Sticksman commented Jun 24, 2023

moonchildua commented Jun 26, 2023

SuperQ commented Jun 26, 2023

sclu1034 commented Jun 27, 2023 • edited Loading

mrjones-plip commented Jun 29, 2023

Sticksman commented Jun 30, 2023

sclu1034 commented Jun 30, 2023

PriyankaKotwal2019 commented Jul 17, 2023

craigvanaman commented Jul 17, 2023

reneeckstein commented Jul 17, 2023

osmanbv commented Jul 19, 2023

sysadmind commented Jul 24, 2023

steklopod commented Aug 7, 2023

marekaf commented Nov 15, 2023

mazinas commented Jun 18, 2024

Electronn commented Sep 23, 2024 • edited Loading

sclu1034 commented Jun 22, 2023 •

edited

Loading

Art-Sel commented Jun 23, 2023 •

edited

Loading

sclu1034 commented Jun 27, 2023 •

edited

Loading

Electronn commented Sep 23, 2024 •

edited

Loading