The Swinging Door Trending (SDT) Filter Processor

[SteveYurongSu]

Motivation

Recently, I've been trying to build actual IoT data processing pipelines using StreamPipes.

I encountered some difficulties in building a scene for UHF time series data processing. For such UHF time series data processing scenarios, we generally do not want to process the original data directly, but only the characteristic data of the original data. Therefore, I hope StreamPipes can provide a way to extract the characteristic data of UHF time series data.

After some of my research, I found that SDT algorithm is a kind of characteristic time series data extration algorithm (or a down-frequency / lossy compression algorithm). The SDT algorithm is typically built into real-time/time database (OSI Pi / IoTDB), but it would also make sense to build it into a filter processor of streampipes.

I would like to contribute to the SDT filter processor implementation. 😆

The Swinging Door Trending Algorithm

Here are some descriptions of the SDT algorithm that I found:

Link: https://www.alibabacloud.com/blog/using-the-swinging-door-trending-algorithm-in-postgresql_595129

The Swinging Door Trending (SDT) algorithm is a linear trend compression algorithm. In essence, it replaces a series of continuous data points with a straight line determined by the start and end points.

The algorithm needs to record the length of each time interval, the data of the start point and the data of the end point. The data of the end point for the previous segment is the data of the start point for the next segment.

The basic principle is relatively simple, please see the figure.

The first data point "a" has one point above and one point below. The distance between them and "a" is E (that is, the width of the door). These two points serve as the two pivot points of the "door". When only the first data point is available, both doors are closed. As the number of points increases, the doors will open gradually. Note that the width of each door can be scaled, and within a certain period of time, once a door is opened, it cannot be closed.

As long as the two doors are not parallel, or the sum of the two inner angles is less than 180° (the algorithm in this article uses this point to judge), this "swinging door" operation can continue. In the figure, the first time interval is from "a" to "e". The result is that the straight line from "a" to "e" are used to replace the data points (a, b, c, d, e), and it plays a role in controlling the compression distortion (E).

The second interval starts from the point "e". At the beginning, the two doors are closed and gradually opened. The subsequent operation is similar to that of the previous interval.

[bossenti]

@SteveYurongSu thanks for this great feature request 🙏🏼
This is really a great idea and pays right into where we want StreamPipes to go
If you would like to contribute here, we would really appreciate it 🤩
Are you looking for someone to join you to contribute here, or do you have the confidence to contribute on your own?
If not, that's perfectly fine, we'd love to work together without on this idea.

[SteveYurongSu]

@bossenti Of course, I want to contribute to this feature here 😆
I already have some ideas about the implementation, and I am confident to implement it.
I will create a issue from this discussion to track the implementation.
If I have any further questions, I will ask here :D

[SteveYurongSu]

It seems that there is no issue template to record feature implementation?

Do we need to create an issue to track the feature implementation?

[bossenti]

@SteveYurongSu Awesome!
Well, that's true.. we might need to come up with one :)
You can go with the fast track, that's fine for now :)

[SteveYurongSu]

@SteveYurongSu Awesome! Well, that's true.. we might need to come up with one :) You can go with the fast track, that's fine for now :)

@bossenti Oh thanks :)

I have created the issue using the Fast Track (Committers only) template.