bloom-builder: "panic: duplicate metrics collector registration attempted"

[diranged]

Describe the bug
Trying to use the new bloom-builder and bloom-planner components introduced by @chaudum in #14003 - but even after we create the /var/loki volume (see #14082), we are running into the builder crashing on startup with this error:

level=info ts=2024-09-09T17:23:51.681576259Z caller=main.go:126 msg="Starting Loki" version="(version=release-3.1.x-89fe788, branch=release-3.1.x, revision=89fe788d)"
level=info ts=2024-09-09T17:23:51.681628661Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2024-09-09T17:23:51.681644678Z caller=modules.go:748 component=bloomstore msg="no metas cache configured"
level=info ts=2024-09-09T17:23:51.681730499Z caller=blockscache.go:420 component=bloomstore msg="run ttl evict job"
level=info ts=2024-09-09T17:23:51.681753203Z caller=blockscache.go:380 component=bloomstore msg="run lru evict job"
level=info ts=2024-09-09T17:23:51.681816379Z caller=blockscache.go:365 component=bloomstore msg="run metrics collect job"
level=info ts=2024-09-09T17:23:51.686655187Z caller=server.go:352 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x4737440, {0x4000f04610?, 0x0?, 0x0?})
	/src/loki/vendor/github.com/prometheus/client_golang/prometheus/registry.go:405 +0x78
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewCounter({{0x2fadad0?, 0x4737440?}}, {{0x25ded97, 0x4}, {0x0, 0x0}, {0x260e2dc, 0x14}, {0x261d67e, 0x18}, ...})
	/src/loki/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:265 +0x128
github.com/grafana/loki/v3/pkg/storage/bloom/v1.NewMetrics({0x2fadad0, 0x4737440})
	/src/loki/pkg/storage/bloom/v1/metrics.go:62 +0x7c
github.com/grafana/loki/v3/pkg/bloombuild/builder.New({{0x6400000, 0x6400000, {0x0, 0x0}, 0x0, 0x0, 0x0, {0x5f5e100, 0x2540be400, 0xa}, ...}, ...}, ...)
	/src/loki/pkg/bloombuild/builder/builder.go:65 +0x154
github.com/grafana/loki/v3/pkg/loki.(*Loki).initBloomBuilder(0x4000fe3008)
	/src/loki/pkg/loki/modules.go:1586 +0x2b4
github.com/grafana/dskit/modules.(*Manager).initModule(0x40000d88e8, {0xffffc8fbdc0a, 0xd}, 0x4001b78fe8, 0x4000b72b70)
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x194
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0x40000d88e8, {0x4000c5cb10, 0x1, 0x1?})
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xb0
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run(0x4000fe3008, {0x0?, {0x4?, 0x2?, 0x4737aa0?}})
	/src/loki/pkg/loki/loki.go:458 +0x74
main.main()
	/src/loki/cmd/loki/main.go:129 +0x10ac

[chaudum]

Hi @diranged Did you run Loki using the vanilla Helm chart?

I got a different panic, see #14110 but could not reproduce the duplicate metrics registration.

[chaudum]

I am able to reproduce this state now. Loki built from main does not have this issue, so needs to be fixed on the release-3.1.x branch only.

[diranged]

Thank you for working to reproduce the issue!

[fculpo]

Hi, same issue on Helm loki@6.16.0