feat(inputs.cisco_telemetry_mdt): Add GRPC Keepalive/timeout config options

[danialre]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

This PR adds configuration options for changing server-side GRPC keepalive/ping settings. We've found when some Cisco IOS-XE devices send telemetry data on GRPC-Dialout over a firewall, some timings on ping packets are affected so that pings arrive without data faster than the minimum ping interval time (5 minute default). This can also be exacerbated by using telemetry intervals longer than 5 minutes.
Each time this happens is a "strike", and if it happens more than 2 times the GRPC server in cisco_telemetry_mdt.go closes out the connection with a GRPC GOAWAY (ENHANCE_YOUR_CALM) error. Sometimes the telemetry device sets up a new connection to continue sending data; other times it fails and we lose data that should otherwise make it to Telegraf.

More information on the keepalive/strike behavior is documented here: https://github.com/grpc/grpc/blob/master/doc/keepalive.md

The two config options added are:

• permit_keepalive_without_calls (bool) - enable/disable enforcement of requiring in-flight calls for pings
• keepalive_minimum_time (duration) - set a custom minimum time in between successive pings.

These settings can allow Cisco MDT users to adjust the sensitivity of these keepalive timeouts, so GRPC sessions won't be closed as often for misbehaving devices.
Default/unfilled options won't change the current plugin behavior.

[powersj]

Thank you for digging into this! We get a number of cisco connection issues and it is always helpful when someone puts up a PR to make this plugin easier to use.

Couple of comments are below.

[powersj]

How does a user know if their clients are sending pings without calls in flight? Basically, if I am using Telegraf, what would I need to know to change this from the default? Similar to what you added for the keepalive_minimum_time below description, you specified an error message that helps guide the user.

@reimda thoughts on a different variable to avoid a boolean?

[danialre]

Added some more documentation to that setting - it's probably not going to be used as much as keepalive_minimum_time, but may be helpful for stablizing telemetry on older/unpatched Cisco code

[powersj]

Thanks - I'd still like @reimda's thoughts on a way to avoid the boolean or to go with it given it is directly used to configure the connection.

[reimda]

a couple thoughts:

I'd usually ask if there are other ways telegraf users might want to handle things related to this setting. For example, besides the default of not allowing keepalive pings without active streams, and the new option of allowing them, maybe someone would want to log when this happens. Then the setting should be a string instead of boolean and it should expect values "permit", "send_goaway", and "log".

In this case since the grpc module only has a boolean setting, it doesn't look like there are going to be other options besides permit and don't permit. Maybe a boolean setting is the right way to go.

The opcua and modbus plugins have "workarounds" sections to handle things like this. If this setting is only useful to interoperate with code that is known to be quirky (or clearly buggy), putting it in a workarounds section away from the ordinary plugin settings is a way to tell telegraf users it's only useful in rare occasions, but at the same time make it visible enough that they'll see it if they need it.

The grpc module does the same thing as those "workarounds" sections but calls it "enforcement policy". Maybe we should mirror the module settings exactly and have a section "enforcement_policy" or "grpc_enforcement_policy" and settings "min_time" and "permit_without_stream".

What do you think?

[danialre]

That sounds good to me - after all, these settings are only useful to get around implementation issues, and don't have any effect on the TCP transport anyway. I'm looking at the opcua plugin to see how the workaround section is implemented, I could modify this to have a "grpc_enforcement_policy" section with these settings.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me. Thanks for your contribution @danialre!