Backport of feature - [NET-4005] - [Supportability] Reloadable Configuration - enable_debug into release/1.15.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #17565 to be assessed for backporting due to the inclusion of the label backport/1.15.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@absolutelightning
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Description

Making enable_debug a Reloadable Configuration

Summary of customer problem and expectations:
In order to capture pprof to analyze Golang heap, CPU, goroutine, and trace profiling "enable_debug" must be set in the configuration and for it to take effect, Consul process must be restarted. Thereafter the "consul debug" command with the appropriate flags can be run to capture the pprof. Having the ability to apply this update to the configuration through "consul reload" would ensure cluster stability.

What currently happens or is possible:
Currently the Consul process must be restarted for this configuration to take effect.

In this PR - I am making enable_debug a reloadable configuration and adding http handlers before the enableDebug flag check. Earlier we use to check for enable debug and then add http handlers. Now I am adding http handlers and checking for enableDebug and checkACLDisabled() inside the handler when the handler is already attached.

Testing & Reproduction steps

Replicate the issue by -

• Git clone this repo
• Run make dev
• create a config file server.json

{
  "node_name": "consul-server",
  "server": true,
  "bootstrap": true,
  "ui_config": {
    "enabled": true
  },
  "datacenter": "dc1",
  "data_dir": "consul/data",
  "log_level": "INFO",
  "addresses": {
    "http": "0.0.0.0"
  },
  "connect": {
    "enabled": true
  },
  "enable_debug": false
}

• start consul agent ./bin/consul agent --config-file=server.json
• open http://localhost:8500/v1/agent/self -> observer EnableDebug is false
• change enable_debug: true in the server.json file
• ./bin/consul reload
• Open http://localhost:8500/v1/agent/self see - EnableDebug flag does not change to true

Testing

• Git clone this repo
• Run make dev
• create a config file server.json

{
  "node_name": "consul-server",
  "server": true,
  "bootstrap": true,
  "ui_config": {
    "enabled": true
  },
  "datacenter": "dc1",
  "data_dir": "consul/data",
  "log_level": "INFO",
  "addresses": {
    "http": "0.0.0.0"
  },
  "connect": {
    "enabled": false
  },
  "enable_debug": false
}

• start consul agent ./bin/consul agent --config-file=server.json
• open http://localhost:8500/v1/agent/self -> observer EnableDebug is false
• change enable_debug: true in the server.json file
• ./bin/consul reload
• Open http://localhost:8500/v1/agent/self see - EnableDebug flag changes to true
• Next test for PProf Run - ./bin/consul debug -capture pprof
• Notice command runs successfully.

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

---

Overview of commits

• 10f500e - e08c309 - 58638de - eaf8c51 - 0e2c9b2

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR