feat: add boostd-data metrics

[LexLuthr]

Closes #1781

Pending tasks

• Fix tests
• Remove older dashboards
• Fix boost-data dashboard
• Explore if any other panels are required
• Update grafana config to use a default dashboard
• document all metrics and how can users scrape them
• update docker monitoring docs
• setup alertmanager X Deferred for a later PR
• Add permanent storage for tempo and prometheus

The updated documents will be merged only after this PR is merged as some of the referenced files do no exist in the current main branch.

[LexLuthr]

Booster-Bitswap Service metrics

 curl http://localhost:9696/metrics
# HELP booster_bitswap_info Boost service information
# TYPE booster_bitswap_info gauge
booster_bitswap_info{commit="+git.0741e86.dirty",node_type="booster-bitswap",version="2.1.0-rc2"} 1

Booster-HTTP service metrics

root@15ec3d838894:/app# curl http://localhost:7777/metrics
# HELP booster_http_info Boost service information
# TYPE booster_http_info gauge
booster_http_info{commit="+git.0741e86.dirty",node_type="booster-http",version="2.1.0-rc2"} 1

Boostd Service metrics

root@3926a0e5b9f6:/app# curl http://boost:1288/metrics
# HELP boost_info Boost service information
# TYPE boost_info gauge
boost_info{commit="+git.0741e86.dirty",node_type="boostd",version="2.1.0-rc2"} 1

Boostd-data service metrics

root@3926a0e5b9f6:/app# curl http://boost:8044/metrics
# HELP boostd_data_info Boost service information
# TYPE boostd_data_info gauge
boostd_data_info{commit="",node_type="boostd-data",version="2.1.0-rc2"} 1

Boostd-data method metrics

root@3926a0e5b9f6:/app# curl http://boost:8044/metrics
# HELP boostd_data_boostddata_success_next_pieces_to_check_count Counter of next pieces to check success
# TYPE boostd_data_boostddata_success_next_pieces_to_check_count counter
boostd_data_boostddata_success_next_pieces_to_check_count 13

Boostd API request duration metrics

root@3926a0e5b9f6:/app# curl http://boost:1288/metrics
# HELP boost_api_request_duration_ms Duration of API requests
# TYPE boost_api_request_duration_ms histogram
boost_api_request_duration_ms_bucket{api="boost",endpoint="OnlineBackup",le="0.01"} 0
.........
boost_api_request_duration_ms_sum{api="boost",endpoint="OnlineBackup"} 7.061916
boost_api_request_duration_ms_count{api="boost",endpoint="OnlineBackup"} 1

[nonsense]

Nit, but you don't need ctxx and you don't need else statement

Suggested change

	ctxx := lcli.ReqContext(cctx)
	var ctx context.Context
	ctx := lcli.ReqContext(cctx)

[nonsense]

Why do we increment twice that metric -- once here and once in the defer statement?

[nonsense]

Suggested change

stats.Record(s.ctx, metrics.FailureGetOffsetSizeCount.M(1))