Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup search ignores the "format" option in date_histogram #34391

Closed
kiju98 opened this issue Oct 11, 2018 · 1 comment
Closed

Rollup search ignores the "format" option in date_histogram #34391

kiju98 opened this issue Oct 11, 2018 · 1 comment
Labels
:StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v6.4.0

Comments

@kiju98
Copy link

kiju98 commented Oct 11, 2018

Elasticsearch version (bin/elasticsearch --version): Version: 6.4.0, Build: default/tar/595516e/2018-08-17T23:18:47.308994Z, JVM: 1.8.0_162

Plugins installed: []

JVM version (java -version): 1.8.0_162, Java HotSpot(TM) 64-Bit Server VM, 25.162-b12, Oracle Corporation

OS version (uname -a if on a Unix-like system): Linux oss-elastic02 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

I tried a rollup search but "format" option in date_histogram was ignored.

Steps to reproduce:

  1. Prepare an index.
PUT stat
{
  "mappings": {
    "doc": {
      "properties": {
        "agent": {
          "properties": {
            "@timestamp": {
              "type": "date"
            }
          }
        },
        "cpu": {
          "properties": {
            "idle": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "iowait": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "sys": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            },
            "total": {
              "properties": {
                "pct": {
                  "type": "scaled_float",
                  "store": true,
                  "scaling_factor": 100
                }
              }
            },
            "user": {
              "properties": {
                "pct": {
                  "type": "float"
                }
              }
            }
          }
        }
      }
    }
  }
}

PUT stat/doc/1
{
  "agent": {
    "@timestamp": "2018-09-01T09:19:00.000Z"
  },
  "cpu": {
    "total": {
      "pct": 11
    },
    "iowait": {
      "pct": 0
    },
    "sys": {
      "pct": 1
    },
    "idle": {
      "pct": 88.6
    },
    "user": {
      "pct": 10.5
    }
  }
}
  1. Create and start a rollup job.
PUT _xpack/rollup/job/rollup-stat
{
  "index_pattern": "stat",
  "rollup_index": "rollup-stat",
  "cron": "* * * * * ?",
  "page_size": 2000,
  "groups": {
    "date_histogram": {
      "interval": "1h",
      "field": "agent.@timestamp"
    }
  },
  "metrics": [
    {
      "field": "cpu.total.pct",
      "metrics": [
        "min",
        "max",
        "avg"
      ]
    }
  ]
}

POST _xpack/rollup/job/rollup-stat/_start
  1. Run a rollup query
GET rollup-stat/_rollup_search
{
  "size": 0,
  "aggs": {
    "1day_val": {
      "date_histogram": {
        "field": "agent.@timestamp",
        "interval": "1d",
        "format": "yyyy-MM-dd",
        "time_zone": "UTC"
      },
      "aggs": {
        "cpu_day_max": {
          "max": {
            "field": "cpu.total.pct"
          }
        }
      }
    }
  }
}

and see the format option in the date_histogram aggregation was ignored as below:

{
  "took": 2,
  "timed_out": false,
  "terminated_early": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "1day_val": {
      "meta": {},
      "buckets": [
        {
          "key_as_string": "2018-09-01T00:00:00.000Z",
          "key": 1535760000000,
          "doc_count": 1,
          "cpu_day_max": {
            "value": 11
          }
        }
      ]
    }
  }
}

"key_as_string" should be "2018-09-01".

Provide logs (if relevant):

@kiju98 kiju98 added :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v6.4.0 labels Oct 11, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search-aggs

polyfractal added a commit to polyfractal/elasticsearch that referenced this issue Oct 16, 2018
Adds support for query-time formatting of the date histo keys
when executing a rollup search.

Closes elastic#34391
polyfractal added a commit that referenced this issue Oct 18, 2018
Adds support for query-time formatting of the date histo keys
when executing a rollup search.

Closes #34391
polyfractal added a commit that referenced this issue Oct 18, 2018
Adds support for query-time formatting of the date histo keys
when executing a rollup search.

Closes #34391
matriv pushed a commit to matriv/elasticsearch that referenced this issue Oct 19, 2018
Adds support for query-time formatting of the date histo keys
when executing a rollup search.

Closes elastic#34391
kcm pushed a commit that referenced this issue Oct 30, 2018
Adds support for query-time formatting of the date histo keys
when executing a rollup search.

Closes #34391
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v6.4.0
Projects
None yet
Development

No branches or pull requests

2 participants