Add moving average function

[mbondyra]

Built on top of #81178

Related to #61777

This adds a moving average expression function similar to the existing cumulative sum and (in process of reviewing) derivative function.

Similar to #80129 and #81178

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[flash1293]

I compared this to the Elasticsearch behavior and I think we should change it slightly to behave the same and also simplify it when doing so:

• If there is a value (0 or other truthy value) in the current row, calculate moving average based on the current window, then add the to the window and cut off tail if necessary
• If there is no value (null or undefined) in the current row, set the output column to undefined as well, then proceed with the next row

[flash1293]

I tried to replicate this using Elasticsearch and it seems like it's doing something different regarding null values. This is what I tried (maybe I have made a mistake in here):

Click to expand!

DELETE timeindex

POST timeindex/_doc
{
  "ts": "2020-09-30"
}

POST timeindex/_doc
{
  "ts": "2020-10-01"
}

POST timeindex/_doc
{
  "ts": "2020-10-02"
}

POST timeindex/_doc
{
  "ts": "2020-10-03",
  "val": 1
}

POST timeindex/_doc
{
  "ts": "2020-10-04",
  "val": 2
}

POST timeindex/_doc
{
  "ts": "2020-10-05",
  "val": 0
}


POST timeindex/_doc
{
  "ts": "2020-10-06"
}


POST timeindex/_doc
{
  "ts": "2020-10-07",
  "val": 0
}


POST timeindex/_doc
{
  "ts": "2020-10-08"
}


POST timeindex/_doc
{
  "ts": "2020-10-09",
  "val": 0
}


POST timeindex/_doc
{
  "ts": "2020-10-10",
  "val": 8
}


POST timeindex/_doc
{
  "ts": "2020-10-11",
  "val": 0
}


POST timeindex/_search
{
  "size": 0,
  "aggs": {
    "time": {
      "date_histogram": {
        "field": "ts",
        "interval": "day"
      },
      "aggs": {
        "sum": {
          "avg": {
              "field": "val"
          }
        },
        "deri": {
          "moving_fn": {
            "buckets_path": "sum",
            "window": 3,
            "script": "MovingFunctions.unweightedAvg(values)"
          }
        }
      }
    }
  }
}

Running this results in the following series:

Values	Moving avg	Expression function
undefined	undefined	undefined
undefined	undefined	undefined
undefined	undefined	undefined
1	undefined	undefined
2	1.0	1
0	1.5	1.5
undefined	undefined	1
0	1.0	1
undefined	undefined	0
0	0.66666	0
8	0	0
0	2.6666	4

It seems like the major difference is:

• Elasticsearch: If an undefined value is encountered, skip the current row and do not change the window
• Expression function: If an undefined value is encountered, calculate the moving average based on the existing window and add "undefined" to window (which might push real values out)

[flash1293]

Behavior looks good to me, but it seems like the algorithm can be simplified a bit based on the recent changes

[flash1293]

LGTM, code review only (but the unit test suite is convincing 🙂 ) Left three nits about further simplifications of the logic

[lukeelmers]

Code LGTM, thanks for the detailed docs & tests!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 25b9073

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
expressions	105	106	+1

distributable file count

id	before	after	diff
default	48214	48215	+1
oss	28676	28677	+1

page load bundle size

id	before	after	diff
expressions	186.7KB	190.2KB	+3.5KB

History

• 💔 Build #85741 failed 292ab645ea90725911a63db21b87e35ca216bc40
• 💚 Build #85640 succeeded c08bd88
• 💚 Build #85479 succeeded 910b2be
• 💚 Build #85276 succeeded 9f5d531
• 💚 Build #85142 succeeded 1c909be

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream