split logs + metrics Fluentd sts

[samjsong]

Description

Splitting Logs and Metrics Fluentd statefulsets for scalability

Also noticed that we had not updated the service monitors for Prometheus in values.yaml, updated those to include the fluentd name

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[sumo-drosiek]

Are we planning to do some breaking changes in the future (especially configuration separation)?

I feel that we should refactor some configuration logic, but on the other hand that change is crucial as part of 1.0.0 which is already in pre-release phase

[vsinghal13]

LGTM except few changes

[perk-sumo]

@samjsong this is great, thank you! :)
One thing is we need to work out the details before it can be merged.

[frankreno]

Agree with other feedback and left a few minor comments. Also, we will need to make sure doc changes for any debugging depending on name are updated.

[samjsong]

Hi team! I've made the requested changes:

1. move the relevant new configs for metrics under the fluentd.metrics key, as well as moving the parallel configs for logs under the fluentd.logs key
2. add -logs suffix to the logs fluentd name. (However I've left the tracing config in the logs fluentd so the name is a little misleading)

[samjsong]

@sumo-drosiek the CI seems to have failed due to something with the kubernetes-tools change, would you mind taking a look if you get the chance? thank you! :)

[perk-sumo]

@samjsong I've rebased your branch on master so that building tools won't break the build.
Also I like the changes you've made, thank you! 👏

[perk-sumo]

LGTM 👍

[sumo-drosiek]

LGTM, seems to be working :)