[followup] Refactor JSON function and add TO_JSON_STRING, ARRAY_LENGHT functions

[LantaoJin]

Description

This is a followup of PR #780

1. Refactor JSON: Currently the JSON can evaluate STRING/JSON_ARRAY/JSON_OBJECT. It brings many confusions and bugs. It because in Splunk or OpenSearch, the data could be stored as JSON object, JSON function evaluates JSON object and return its value or null if invalid. But in Spark, there is no JSON object, but it contains STRING, StructType and ArrayType. So to evaluate a json string, JSON function should accept STRING type only.

2. Add ARRAY_LENGHT: for the same reason, the original JSON_ARRAY_LENGHT accepts both STRINGand JSON_ARRAY(ArrayType). We separate JSON_ARRAY_LENGHT to JSON_ARRAY_LENGHT and ARRAY_LENGHT. The JSON_ARRAY_LENGHT only accepts STRING type, ARRAY_LENGHT only accepts ArrayType.

3. Add TO_JSON_STRING: After the refactor of (1), we still need a method to convert JSON_ARRAY/JSON_OBJECT to valid JSON STRING. TO_JSON_STRING accepts both StructType and ArrayType as input and returns JSON formatted string.

Related Issues

Resolves #869

Check List

• Updated documentation (docs/ppl-lang/README.md)
• Implemented unit tests
• Implemented tests for combination with other commands
• New added source code should include a copyright header
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[LantaoJin]

About the naming: to_json is good name for Spark since there is no json object data type. So to_json in Spark is converting StructType to json String. But to_json might not a good name for PPL since it's easy to cause misunderstanding as converting something to json object. to_json_string could be more straightforward.

[YANG-DB]

yes sounds like a better distinction ...

[LantaoJin]

We choose array_length instead of array_size since we have had a function json_array_length. So it maps to the Spark builtin function array_size.

[LantaoJin]

The current logic in this if-else could cause bugs.
Both
| eval b = json_array_length(json_array(1,2,3)) and
| eval b = json_array_length("[1,2,3]") work fine.

But imaging following two cases:
| eval a = json_array(1,2,3), b = json_array_length(a)
| eval a = "[1,2,3]", b = json_array_length(a)
The args of json_array_length are both UnresolvedAttribute("a") which go into the else branch. But a in the first query will be resolved to UnresolveFunction("array"..) and a in the second case will be resolved to StringLiteral.

[YANG-DB]

@LantaoJin I approved but plz make sure we cover the next issue

[YANG-DB]

@LantaoJin did we cover the next issue?

[LantaoJin]

With #780 , we can access the first element in a array by

| eval first_elem = json_extract('{"items": ["a", "b", "c"]}', '$.items[0]') // json object string

or

| eval first_elem = json_extract('["a", "b", "c"]', '$.[0]') // json array string

But it requires the input is a json string. If the input is json array (ArrayType), I think we need a new function such as array_get:

| eval first_elem = array_get(json_array("a", "b", "c"), 0)

But as a workaround, we could rewrite by

| eval first_elem = json_extract(to_json_string(json_array("a", "b", "c")), '$.[0]')

So keep #675 open until we have array_get.

[LantaoJin]

Em, I think I can add array_get ASAP in this PR too.

Let merge this first. From the description of #675 ,
json_extract('{"items": ["a", "b", "c"]}', '$.items[0]') seems the right solution.