Generic SQL input

[srebhan]

Required for all PRs:

• Associated README.md updated.
• Has appropriate unit tests.

This PR adds a generic SQL input plugin that can be used to query SQL servers and convert the returned data into metrics. In a first attempt only a handful of drivers are supported (see SQL_DRIVERS_INPUT.md) but there are more on the list.

Currently a test-suite is missing, therefore I'm marking this as DRAFT. Reviews, comments, suggestions and tests are welcome!

Takeover of PR #2785
Fixes #352

[telegraf-tiger]

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

[srebhan]

!signed-cla

[ssoroka]

A work of art. well done

[srebhan]

!signed-cla

[telegraf-tiger]

🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

[sspaink]

!signed-cla

[telegraf-tiger]

🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

[srebhan]

For running the tests, please create a testdata directory in the sql-plugin dir. Then download and unzip the MariaDB sample data and put the nation.sql file in the testdata directory. I skipped this as I cannot find out what the License of this dataset is.

If those preparations are successful, run go test -args -spinup in the sql-plugin dir. Keep fingers crossed! ;-P

[srebhan]

!signed-cla

[telegraf-tiger]

🤝 ✅ CLA has been signed. Thank you!

[sspaink]

!retry-checks

[aslgithub]

@srebhan Hi Sven, any chance you could get this pull completed so SQL queries can be run as a Telegraf input please ?
Got excited when you took this PR over in January that someone would be able to complete this functionality..
Thanks !

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• darwin_amd64.tar.gz

• windows_amd64.zip

• windows_i386.zip

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• linux_mips.tar.gz

• mips.deb

• static_linux_amd64.tar.gz

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• linux_mipsel.tar.gz

• mipsel.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

[srebhan]

@aslgithub your wish is my command. ;-P
Testing this is very much appreciated! You can use the artifacts built by CI by expanding telegraf-tiger's comment. :-)

[aslgithub]

@srebhan nice one, thank you, much appreciated :-)
Will test this against Microsoft SQL Server.

[aslgithub]

@aslgithub your wish is my command. ;-P
Testing this is very much appreciated! You can use the artifacts built by CI by expanding telegraf-tiger's comment. :-)

@srebhan
Hi Sven,
I've started testing against Microsoft SQL.

It appears to work as expected in that I am able to connect to the SQL server and complete a basic query that does a select count from a table to return an integer.

Specifying a database in the dsn string within inputs.sql doesn't seem to have any impact?
However I believe that might be as intended given you can have multiple inputs.sql.query each targeting their own databases and tables on the same server connection ?

Would it be possible to support the tags feature at the query level please?
It works at the top level :
[inputs.sql.tags]
environment="dev"

however it doesn't work at the query level :
[inputs.sql.query.tags]
environment="dev"

This would be really helpful as I can have each query tagging its data as appropriate under the same connection.

Thanks,
Paul

[srebhan]

@aslgithub nice to hear it's at least not failing completely! :-)
TBH we do not filter the DSN in this PR. Can you please double check your DSN with the driver documentation.
Regarding the tags, I would like to add this in another PR (you are welcome to beat me at it :-)) once this is in. Complicating the code will make it only harder for the reviewers. Hope that's ok for you!?

[aslgithub]

@srebhan
Hi Sven,
Certainly not failing completely, this build has now been running for 12 days and has collected 227,249 data points from a Microsoft SQL Database using 2 inputs.sql.query sections.

Sorry, I'm going to admit doing a PR to add the tag would be beyond me appreciate your support in doing this.

Thanks,
Paul

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• windows_amd64.zip

• windows_i386.zip

• darwin_amd64.tar.gz

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• linux_mips.tar.gz

• mips.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• static_linux_amd64.tar.gz

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• linux_mipsel.tar.gz

• mipsel.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• windows_amd64.zip

• windows_i386.zip

• darwin_amd64.tar.gz

• linux_mipsel.tar.gz

• mipsel.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• static_linux_amd64.tar.gz

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• linux_mips.tar.gz

• mips.deb

[srebhan]

@aslgithub give master a test if you can... Thanks to @reimda this plugin will (hopefully) make it into 1.19

[aslgithub]

@srebhan I've downloaded the nightly build and it seems to be okay giving the same result as the previous artifact that I have been running on test. Thanks for your efforts on this.
@reimda Thank you so much for merging this :-)