Support user-defined and incomplete date formats

[GumpacG]

Description

• Added support user defined (custom) date formats (backed by java syntax)
• Updated support for predefined formats (doc, list, another doc)
• Combinations of custom and named formats are also supported
• Values produced by incomplete date formatters (e.g. year, week_year) returned as TIMESTAMP

TODOs

• Add user docs - Update docs to reflect recent changes. #285

Sample for test

Mapping:

{
  "mappings" : {
    "properties" : {
      "custom_date" : {
        "type" : "date",
        "format" : "yyyy-MM-dd"
      }
   }
}

Data:

{"index": {}}
{"epoch_millis": "1984-04-12"}

Sample 2

Mapping

{
    "mappings":
    {
        "properties":
        {
            "custom_time" :
            {
                "type" : "date",
                "format" : "::: k-A || A    "
            },
            "incomplete_1" :
            {
                "type" : "date",
                "format" : "year"
            },
            "incomplete_2" :
            {
                "type" : "date",
                "format" : "E-w"
            },
            "incomplete_custom_date" :
            {
                "type" : "date",
                "format" : "uuuu"
            },
            "incomplete_custom_time" :
            {
                "type" : "date",
                "format" : "HH"
            },
            "incorrect" :
            {
                "type" : "date",
                "format" : "'___'"
            },
            "epoch_sec" :
            {
                "type" : "date",
                "format" : "epoch_second"
            },
            "epoch_milli" :
            {
                "type" : "date",
                "format" : "epoch_millis"
            },
            "custom_no_delimiter_date" :
            {
                "type" : "date",
                "format" : "uuuuMMdd"
            },
            "custom_no_delimiter_time" :
            {
                "type" : "date",
                "format" : "HHmmss"
            },
            "custom_no_delimiter_ts" :
            {
                "type" : "date",
                "format" : "uuuuMMddHHmmss"
            }
        }
    }
}

Data

{"index": {}}
{ "custom_time":  "85476321", "incomplete_1" : 1984, "incomplete_2": null, "incomplete_custom_date": 1999, "incomplete_custom_time" : 10, "incorrect" : null, "epoch_sec" : 42, "epoch_milli" : 42, "custom_no_delimiter_date" : "19841020", "custom_no_delimiter_time" : "102030", "custom_no_delimiter_ts" : "19841020153548" }
{"index": {}}
{ "custom_time":  "::: 9-32476542", "incomplete_1" : 2022, "incomplete_2": null, "incomplete_custom_date": 3021, "incomplete_custom_time" : 20, "incorrect" : null, "epoch_sec" : 100500, "epoch_milli" : 100500, "custom_no_delimiter_date" : "19610412", "custom_no_delimiter_time" : "090700", "custom_no_delimiter_ts" : "19610412090700" }

Result set

SELECT * FROM test

+--------------------------+-----------+-------------------------+---------------------+------------------------+--------------+------------------------+--------------------------+--------------+------------------------+---------------------+ 
| custom_no_delimiter_date | incorrect | epoch_milli             | epoch_sec           | incomplete_custom_time | custom_time  | custom_no_delimiter_ts | custom_no_delimiter_time | incomplete_2 | incomplete_custom_date | incomplete_1        |
+--------------------------+-----------+-------------------------+---------------------+------------------------+--------------+------------------------+--------------------------+--------------+------------------------+---------------------+
| date                     | timestamp | timestamp               | timestamp           | time                   | time         | timestamp              | time                     | timestamp    | date                   | timestamp           |
+--------------------------+-----------+-------------------------+---------------------+------------------------+--------------+------------------------+--------------------------+--------------+------------------------+---------------------+
| 1984-10-20               | null      | 1970-01-01 00:00:00.042 | 1970-01-01 00:00:42 | 10:00:00               | 23:44:36.321 | 1984-10-20 15:35:48    | 10:20:30                 | null         | 1999-01-01             | 1984-01-01 00:00:00 |
| 1961-04-12               | null      | 1970-01-01 00:01:40.5   | 1970-01-02 03:55:00 | 20:00:00               | 09:01:16.542 | 1961-04-12 09:07:00    | 09:07:00                 | null         | 3021-01-01             | 2022-01-01 00:00:00 |
+--------------------------+-----------+-------------------------+---------------------+------------------------+--------------+------------------------+--------------------------+--------------+------------------------+---------------------+

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[acarbonetto]

This is getting really big.
We should split this out into a separate file when we do the real fix.

[acarbonetto]

Found one issue. Note that the year of 1999 isn't being received in the response.

Mapping:

{
  "mappings" : {
    "properties" : {
      "custom_time" : {
        "type" : "date",
        "format" : "yyyy HH:mm"
      }
    }
  }
}

data:

{"index": {}}
{"custom_time": "1999 01:01"}

returns:

{
    "schema": [
        {
            "name": "custom_time",
            "type": "timestamp"
        }
    ],
    "datarows": [
        [
            "1970-01-01 01:01:00"
        ]
    ],
    "total": 2,
    "size": 2,
    "status": 200
}

[GumpacG]

When a mapping with incomplete time/date format is loaded and queried, the return type should remain the same.
For example:

"incomplete_custom_time" :
            {
                "type" : "date",
                "format" : "HH"
            },

Should remain a time. It currently is returned as a timestamp. Incomplete custom date also has the same behaviour.

[GumpacG]

When a mapping with incomplete time/date format is loaded and queried, the return type should remain the same. For example:

"incomplete_custom_time" :
            {
                "type" : "date",
                "format" : "HH"
            },

Should remain a time. It currently is returned as a timestamp. Incomplete custom date also has the same behaviour.

Also test that CAST to timestamp from incomplete_custom_time returns todays date with the custom time.

[GumpacG]

Checkstyle is failing and a few IT needs to get updated

[Yury-Fridlyand]

Found one issue. Note that the year of 1999 isn't being received in the response.

This comes from org.opensearch.common.time.DateFormatters. Maybe not a bug though. To avoid that we probably need to split custom format into date and time parts and then split values.

---

Should remain a time. It currently is returned as a timestamp. Incomplete custom date also has the same behaviour

Fixed in 6d214a5

---

Also test that CAST to timestamp from incomplete_custom_time returns todays date with the custom time.

Works as expected: converting time to date/dt/ts adds today's date (not epoch).

opensearchsql> select CAST(incomplete_custom_time as TIMESTAMP) from dt_formats;
fetched rows / total rows = 2/2
+---------------------------------------------+
| CAST(incomplete_custom_time as TIMESTAMP)   |
|---------------------------------------------|
| 2023-07-07 10:00:00                         |
| 2023-07-07 20:00:00                         |
+---------------------------------------------+

---

Checkstyle is failing and a few IT needs to get updated

Fixed in c086dad thanks for noticing