Revert EMR info in Flint metadata

[dai-chen]

Description

Revert previous changes https://github.com/opensearch-project/opensearch-spark/pull/125/files#diff-3ba13ed226c6bfc748e11ae7c79ad2b47783ab2a16805dfdd142818fa9796c70 because PPL plugin still assumes these 2 fields present when deserialization.

Without it, drop statement will throw syntax exception:

$ curl localhost:9200/_plugins/_async_query -X POST -H "Content-Type: application/json" -d'
{
    "datasource" : "glue_1",
    "lang" : "sql",
    "query" : "DROP INDEX chen_all_columns ON mys3.default.http_logs"
}'
{
  "status": 400,
  "error": {
    "type": "IllegalArgumentException",
    "reason": "Invalid Request",
    "details": "Provided Index doesn\u0027t exist"
  }
}

Test

Mock EMR env variable and create skipping index:

$ export SERVERLESS_EMR_VIRTUAL_CLUSTER_ID=test_app
$ export SERVERLESS_EMR_JOB_ID=test_job
$ spark-shell  ...

scala> System.getenv("SERVERLESS_EMR_VIRTUAL_CLUSTER_ID")
res0: String = test_app
scala> System.getenv("SERVERLESS_EMR_JOB_ID")
res1: String = test_job

scala> spark.sql("""
     | CREATE SKIPPING INDEX ON stream.lineitem_tiny
     | (l_shipdate VALUE_SET)
     | WITH (
     |   auto_refresh = true,
     |   checkpoint_location = "s3://checkpoint")
     | """)

{
  "flint_myglue_stream_lineitem_tiny_skipping_index": {
    "mappings": {
      "_meta": {
        "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "kind": "skipping",
        "indexedColumns": [
          {
            "columnType": "date",
            "kind": "VALUE_SET",
            "columnName": "l_shipdate"
          }
        ],
        "name": "flint_myglue_stream_lineitem_tiny_skipping_index",
        "options": {
          "auto_refresh": "true",
          "checkpoint_location": "s3://chen-emr-test/checkpoints/job-14"
        },
        "source": "myglue.stream.lineitem_tiny",
        "version": "0.1.0",
        "properties": {
          "env": {
            "SERVERLESS_EMR_VIRTUAL_CLUSTER_ID": "test_app",
            "SERVERLESS_EMR_JOB_ID": "test_job"
          }
        }
      },
      ......
    }
  }
}

Issues Resolved

opensearch-project/sql#2424

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.