[Feature] Datadog profiler

[ZeljkoBenovic]

This PR adds Datadog profiler into the codebase, so that it can be easily enabled on demand.
Profiler enables users to see profiles and tracing stats on their Datadog portal, but it requires some configuration of Datadog agent on the node.

APM ( tracing ) should be enabled in Datadog agent configuration yaml file:

apm_config:

  enabled: true
  env: polygon-edge # this sets the environment name that will be shown on DD portal

To enable the profiler, a user must simply set DD_PROFILING_ENABLED="True" environment variable.
To set the IP and port that the DataDog agent, use DD_AGENT_HOST and DD_TRACE_AGENT_PORT environment variables.
By default, it will try communicate with the agent on localhost:8126.

These env vars are now consistent with the DD doc https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/go/?tab=containers#configure-the-datadog-agent-for-apm

Additional parameters can be set by using env vars stated in the official Datadog doc https://docs.datadoghq.com/profiler/enabling/go/

Changes include

• Bugfix (non-breaking change that solves an issue)
• Hotfix (change that solves an urgent issue, and requires immediate attention)
• New feature (non-breaking change that adds functionality)
• Breaking change (change that is not backwards-compatible and/or changes current functionality)

Checklist

• I have assigned this PR to myself
• I have added at least 1 reviewer
• I have added the relevant labels
• I have updated the official documentation
• I have added sufficient documentation in code

Testing

• I have tested this code with the official test suite
• I have tested this code manually

Manual tests

Manual tests would require that the tester has access to Datadog portal.
Just set the DD_ENABLE env var, configure Datadog agent on the same node, and check Datadog portal if you can see your profile.

Documentation update

Public doc will soon with the explanation on how to set this up will follow

Additional comments

Fixes EDGE-764

[zivkovicmilos]

Looks great 💯

[codecov]

Codecov Report

Merging #707 (66832ec) into develop (690b4b4) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #707      +/-   ##
===========================================
- Coverage    52.70%   52.68%   -0.02%     
===========================================
  Files          130      130              
  Lines        17146    17146              
===========================================
- Hits          9037     9034       -3     
- Misses        7461     7463       +2     
- Partials       648      649       +1     

Impacted Files	Coverage Δ
jsonrpc/jsonrpc.go	18.23% <100.00%> (ø)
syncer/client.go	61.79% <0.00%> (-1.42%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[0xAleksaOpacic]

Code looks good to me, WHy are we having issues with CLAssistant first time seeing this

[0xAleksaOpacic]

Can you add a comment: I have read the CLA Document and I hereby sign the CLA

[vcastellm]

The approach looks good in general, but I don't feel confortable with adding features using only env vars, I mean, ideally for me, this can be configurable in CLI params, config file and env vars.

[epikichi]

The approach looks good in general, but I don't feel confortable with adding features using only env vars, I mean, ideally for me, this can be configurable in CLI params, config file and env vars.

I agree as well. Let's include the CLI and config equivalent to this PR. @ZeljkoBenovic

Ideally, we can offload the implementation to https://github.com/spf13/viper @zivkovicmilos 👀

[ZeljkoBenovic]

@vcastellm The primary idea was not to clutter the UX with additional flags as DataDog is not a feature that is so commonly used as opposed to Prometheus for example.
Secondly, we wanted to be aligned with their docs on how to setup things, for example https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/go/?tab=containers#configure-the-datadog-agent-for-apm where they are using env vars.
Additionally, this is not a standalone feature, it still requires DataDog Agent installed and configured on the host ( or as a remote instance )

If you guys think that we need flags/config file parameters for this, it is very easy to add them.

[vcastellm]

LGTM