Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(output plugin): Add new output plugin to support Apache IoTDB #11557

Merged
merged 23 commits into from
Aug 10, 2022
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
560c91d
init iotdb plugin
citrusreticulata Jul 7, 2022
6e20acd
complete iotdb plugins
citrusreticulata Jul 14, 2022
e0e454c
to fit linux style.
citrusreticulata Jul 27, 2022
6af6993
make tidy
citrusreticulata Jul 27, 2022
d63c1f3
optimize readme and testing
citrusreticulata Jul 27, 2022
5e39fda
update license of dependencies; fix readme
citrusreticulata Jul 28, 2022
f4cc0ee
optimize code style
citrusreticulata Jul 28, 2022
a6d1789
update readme, added a longer first section
citrusreticulata Jul 28, 2022
021900a
Code optimized; waiting to support container
citrusreticulata Jul 29, 2022
181d437
fix some typo
citrusreticulata Jul 29, 2022
1e5d002
Code optimized; waiting to test
citrusreticulata Jul 29, 2022
0832eaa
Code optimized; fix README.md typo
citrusreticulata Aug 2, 2022
83f6711
fix docker image version
citrusreticulata Aug 2, 2022
27806bc
reuse old style integration test
citrusreticulata Aug 2, 2022
3da2bb8
name changed: generateTestMetric -> newMetricWithOrderedFields
citrusreticulata Aug 2, 2022
fcf0e37
update iotdb_test.go
citrusreticulata Aug 2, 2022
17f0e4f
divided Conversion and Tags-Handling into unit tests
citrusreticulata Aug 2, 2022
2658193
Added entire metrics conversion tests
citrusreticulata Aug 3, 2022
15cef01
Update iotdb_test.go; fix some typo
citrusreticulata Aug 5, 2022
bd08e84
Merge remote-tracking branch 'upstream/master' into apache-iotdb
citrusreticulata Aug 8, 2022
28210db
make tidy
citrusreticulata Aug 8, 2022
cddafdb
update README.md and sample.conf;
citrusreticulata Aug 9, 2022
8c9432e
fix long line bug...
citrusreticulata Aug 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ following works:
- github.com/antchfx/xmlquery [MIT License](https://github.com/antchfx/xmlquery/blob/master/LICENSE)
- github.com/antchfx/xpath [MIT License](https://github.com/antchfx/xpath/blob/master/LICENSE)
- github.com/apache/arrow/go/arrow [Apache License 2.0](https://github.com/apache/arrow/blob/master/LICENSE.txt)
- github.com/apache/iotdb-client-go [Apache License 2.0](https://github.com/apache/iotdb-client-go/blob/main/LICENSE)
- github.com/apache/thrift [Apache License 2.0](https://github.com/apache/thrift/blob/master/LICENSE)
- github.com/aristanetworks/glog [Apache License 2.0](https://github.com/aristanetworks/glog/blob/master/LICENSE)
- github.com/aristanetworks/goarista [Apache License 2.0](https://github.com/aristanetworks/goarista/blob/master/COPYING)
Expand Down
5 changes: 3 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ require (
github.com/Shopify/sarama v1.34.1
github.com/aerospike/aerospike-client-go/v5 v5.7.0
github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1529
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1695
github.com/amir/raidman v0.0.0-20170415203553-1ccc43bfb9c9
github.com/antchfx/jsonquery v1.1.5
github.com/antchfx/xmlquery v1.3.9
Expand Down Expand Up @@ -134,7 +134,7 @@ require (
github.com/sirupsen/logrus v1.8.1
github.com/sleepinggenius2/gosmi v0.4.4
github.com/snowflakedb/gosnowflake v1.6.2
github.com/stretchr/testify v1.7.4
github.com/stretchr/testify v1.8.0
github.com/tbrandon/mbserver v0.0.0-20170611213546-993e1772cc62
github.com/testcontainers/testcontainers-go v0.12.0
github.com/tidwall/gjson v1.14.1
Expand Down Expand Up @@ -202,6 +202,7 @@ require (
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/alecthomas/participle v0.4.1 // indirect
github.com/apache/arrow/go/arrow v0.0.0-20211006091945-a69884db78f4 // indirect
github.com/apache/iotdb-client-go v0.12.2-0.20220722111104-cd17da295b46
github.com/aristanetworks/glog v0.0.0-20191112221043-67e8567f59f3 // indirect
github.com/armon/go-metrics v0.3.3 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.2 // indirect
Expand Down
10 changes: 6 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -275,8 +275,8 @@ github.com/alexflint/go-filemutex v0.0.0-20171022225611-72bdc8eae2ae/go.mod h1:C
github.com/alexkohler/prealloc v1.0.0/go.mod h1:VetnK3dIgFBBKmg0YnD9F9x6Icjd+9cvfHR56wJVlKE=
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
github.com/alicebob/miniredis v2.5.0+incompatible/go.mod h1:8HZjEj4yU0dwhYHky+DxYx+6BMjkBbe5ONFIF1MXffk=
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1529 h1:qAt5MZ3Ukwx/JMAiaagQhNQMBZLcmJbnweBoK3EeHxI=
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1529/go.mod h1:RcDobYh8k5VP6TNybz9m++gL3ijVI5wueVr0EM10VsU=
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1695 h1:ISpCPksXFyDOLO/8OQAdO6pKkC31NKldG6q+lPxM/ao=
github.com/aliyun/alibaba-cloud-sdk-go v1.61.1695/go.mod h1:RcDobYh8k5VP6TNybz9m++gL3ijVI5wueVr0EM10VsU=
github.com/amir/raidman v0.0.0-20170415203553-1ccc43bfb9c9 h1:FXrPTd8Rdlc94dKccl7KPmdmIbVh/OjelJ8/vgMRzcQ=
github.com/amir/raidman v0.0.0-20170415203553-1ccc43bfb9c9/go.mod h1:eliMa/PW+RDr2QLWRmLH1R1ZA4RInpmvOzDDXtaIZkc=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
Expand All @@ -297,6 +297,8 @@ github.com/apache/arrow/go/arrow v0.0.0-20191024131854-af6fa24be0db/go.mod h1:VT
github.com/apache/arrow/go/arrow v0.0.0-20210818145353-234c94e4ce64/go.mod h1:2qMFB56yOP3KzkB3PbYZ4AlUFg3a88F67TIx5lB/WwY=
github.com/apache/arrow/go/arrow v0.0.0-20211006091945-a69884db78f4 h1:nPUln5QTzhftSpmld3xcXw/GOJ3z1E8fR8tUrrc0YWk=
github.com/apache/arrow/go/arrow v0.0.0-20211006091945-a69884db78f4/go.mod h1:Q7yQnSMnLvcXlZ8RV+jwz/6y1rQTqbX6C82SndT52Zs=
github.com/apache/iotdb-client-go v0.12.2-0.20220722111104-cd17da295b46 h1:28HyUQcr8ZCyCAatR0gkf9PuLr52U2T+66tx5Th0nxI=
github.com/apache/iotdb-client-go v0.12.2-0.20220722111104-cd17da295b46/go.mod h1:1z89VPGCUGHGqxkPW8p2Haq6WJwrRBKZN+WOjDBiQQM=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/apache/thrift v0.13.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/apache/thrift v0.14.1/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
Expand Down Expand Up @@ -2234,8 +2236,8 @@ github.com/stretchr/testify v1.6.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.4 h1:wZRexSlwd7ZXfKINDLsO4r7WBt3gTKONc6K/VesHvHM=
github.com/stretchr/testify v1.7.4/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/syndtr/gocapability v0.0.0-20170704070218-db04d3cc01c8/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
Expand Down
1 change: 1 addition & 0 deletions plugins/outputs/all/all.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
_ "github.com/influxdata/telegraf/plugins/outputs/influxdb"
_ "github.com/influxdata/telegraf/plugins/outputs/influxdb_v2"
_ "github.com/influxdata/telegraf/plugins/outputs/instrumental"
_ "github.com/influxdata/telegraf/plugins/outputs/iotdb"
_ "github.com/influxdata/telegraf/plugins/outputs/kafka"
_ "github.com/influxdata/telegraf/plugins/outputs/kinesis"
_ "github.com/influxdata/telegraf/plugins/outputs/librato"
Expand Down
123 changes: 123 additions & 0 deletions plugins/outputs/iotdb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# IoTDB Output Plugin

This output plugin saves Telegraf metrics to an Apache IoTDB backend,
supporting session connection and data insertion.

## Apache IoTDB

Apache IoTDB (Database for Internet of Things) is an IoT native database with
high performance for data management and analysis, deployable on the edge and
citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved
the cloud. Due to its light-weight architecture, high performance and rich
feature set together with its deep integration with Apache Hadoop, Spark and
Flink, Apache IoTDB can meet the requirements of massive data storage,
high-speed data ingestion and complex data analysis in the IoT industrial
fields.

For more details consult the [Apache IoTDB website](https://iotdb.apache.org)
or the [Apache IoTDB GitHub page](https://github.com/apache/iotdb).

## Getting started

Before using this plugin, please configure the IP address, port number,
user name, password and other information of the database server,
as well as some data type conversion, time unit and other configurations.

Please see the [configuration section](#Configuration) for an example
configuration.

## Please pay attention to these points

1. IoTDB (version 0.13.x or older) **DO NOT support unsigned integer**.
There are three available options of converting uint64, which are specified by
citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved
parameter `uint64_conversion`.

- `int64_clip`, default option. If an unsigned integer is greater than
`math.MaxInt64`, save it as `int64`; else save `math.MaxInt64`
(9223372036854775807).
- `int64`, force converting an unsigned integer to a`int64`,no mater
what the value it is. This option may lead to exception if the value is
greater than `int64`.
- `text`force converting an unsigned integer to a string, no mater what the
value it is.
citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved

2. IoTDB supports a variety of time precision, but no matter what precision,
timestamp is stored in `Int64`, so users need to specify the unit of timestamp.
Default unit is `nanosecond`.

citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved
3. Till now, IoTDB can not support Tag indexing well. To see current process
method, please refer to [InfluxDB-Protocol Adapter](
https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html).
There are two available options of converting tags, which are specified by
parameter `convert_tags_to`:

- `fields`. Treat Tags as measurements. For each Key:Value in Tag,
convert them into Measurement, Value, DataType, which are supported in IoTDB.
- `device_id`, default option. Treat Tags as part of device id. Tags
is subtree of 'Name'.

For example, there is a metric:

`Name="root.sg.device", Tags={tag1="private", tag2="working"}, Fields={s1=100, s2="hello"}`

- `fields`, result: `root.sg.device, s1=100, s2="hello", tag1="private", tag2="working"`
- `device_id`, result: `root.sg.device.private.working, s1=100, s2="hello"`

## Testing

**Please prepare running database before testing**.
Target address is`localhost:6667` by default, which can be edit in
`iotdb_test.go`. `test_host` is the target ip address of database server.

Testing contains: network connection, error correction, datatype conversion,
data writing.
citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved

## Configuration

```toml @sample.conf
# Save metrics to an IoTDB Database
[[outputs.iotdb]]
## Configuration of IoTDB server connection
host = "127.0.0.1"
# port = "6667"
citrusreticulata marked this conversation as resolved.
Show resolved Hide resolved

## Configuration of authentication
user = "root"
password = "root"

## Timeout to open a new session.
## A value of zero means no timeout.
# timeout = "5s"

## Configuration of type conversion for 64-bit unsigned int
## IoTDB currently DOES NOT support unsigned integers (version 13.x).
## 32-bit unsigned integers are safely converted into 64-bit signed integers by the plugin,
## however, this is not true for 64-bit values in general as overflows may occur.
## The following option allows to specify the handling of 64-bit unsigned integers.
## Available options are:
## - "int64" -- convert to 64-bit signed integers and accept overflows
## - "int64_clip" -- convert to 64-bit signed integers and clip the values on overflow to 9,223,372,036,854,775,807
## - "text" -- convert to the string representation of the value
# uint64_conversion = "int64_clip"

## Configuration of TimeStamp
## TimeStamp is always saved in 64bits int. timestamp_precision specifies the unit of timestamp.
## Available value:
## "second", "millisecond", "microsecond", "nanosecond"(default)
# timestamp_precision = "nanosecond"

## Handling of tags
## Tags are not fully supported by IoTDB.
## A guide with suggestions on how to handle tags can be found here:
## https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html
##
## Available methods are:
## - "fields" -- convert tags to fields in the measurement
## - "device_id" -- attach tags to the device ID
##
## For Example, a metric named "root.sg.device" with the tags `tag1: "private"` and `tag2: "working"` and
## fields `s1: 100` and `s2: "hello"` will result in the following representations in IoTDB
## - "fields" -- root.sg.device, s1=100, s2="hello", tag1="private", tag2="working"
## - "device_id" -- root.sg.device.private.working, s1=100, s2="hello"
# convert_tags_to = "device_id"

```
Loading