added an option to skip failed outputs in agent startup

[eran-gil2]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #3723, resolves #9778

[telegraf-tiger]

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

[eran-gil2]

!signed-cla

[telegraf-tiger]

👍 This pull request doesn't change the Telegraf binary size

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here ! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_armv7.tar.gz
armhf.deb	i386.rpm	freebsd_i386.tar.gz
i386.deb	ppc64le.rpm	linux_amd64.tar.gz
mips.deb	s390x.rpm	linux_arm64.tar.gz
mipsel.deb	x86_64.rpm	linux_armel.tar.gz
ppc64el.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[eran-gil2]

hey could you please approve other workflows so this feature could be merged? :)

[eran-gil2]

@srebhan who is allowed to review my feature? anyone? or are there certain specifications for this?

[Hipska]

General idea is good, but I would add a command line option for that as well.

It seems to me that it might be better if outputs that are not able to connect at startup are retried after some time instead of just being completely ignored for the duration of telegraf run.

[Hipska]

Please also update related agentConfig with example configuration.

[Hipska]

As well as the etc/telegraf.conf and etc/telegraf_windows.conf

[Hipska]

This PR also doesn't solve #3723 since it is requested to have x times or retry. This PR doesn't add this behaviour.

Comments in a related PR (#9051) mentioned that plugins should be implementing the Start or Connect method in order to let telegraf startup without halting if one of the plugins cannot be connected to. It seems that in order to have it like this, a change in agent.go needs to be done.

[srebhan]

@eran-gil2 I'd like to discuss this with the core-devs as this is a very mighty feature IMO. To me it seems too easy to loose data when using this feature without caution as you will get an "everything alright" impression but you easily have no working output. I think this might be some fundamental thing to retry to connect to the outputs until we reach a buffer overflow or anything. We even might want to persist the metrics locally in this case...

[srebhan]

@eran-gil2 after discussing this matter with the team, we came to the conclusion that a global option to ignore all errors is not the way we want to go. There have been similar proposals (#11313) and requests (#10694, #11289) for the input side.

However, we think it's better to fix the individual plugins to avoid connecting to services in Init but use Connect instead. Furthermore, we do see the demand for a more dynamic configuration of telegraf and other features such as life-cycle and connection management. We will start a discussion on how to integrate such features in future versions of telegraf.

For now, I'm closing this PR as a correct implementation requires a rework of Telegraf internals which is out-of-scope for now.