Ingest Node's grok can't set the same field from two patterns

[tsg]

Elasticsearch version: 5.0.1

Plugins installed: ingest-node-geoip, ingest-node-ua

JVM version: 1.8

OS version: macOS sierra

Description of the problem including expected versus actual behavior:

See the following Ingest node simulate API call:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "Pipeline for parsing MySQL slow logs.",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{DATA:mysql.error.timestamp} %{NUMBER:mysql.error.id} \\[%{DATA:mysql.error.level}\\] %{GREEDYDATA:mysql.error.message}",
            "%{LOCALDATETIME:mysql.error.timestamp} %{DATA:mysql.error.name} %{GREEDYDATA:mysql.error.message}"
          ],
          "ignore_missing": true,
          "pattern_definitions": {
            "LOCALDATETIME": "[0-9]+ %{TIME}"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "161209 13:08:33 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql"
      }
    }
  ]
}

There are two Grok patterns, and the provided doc should match the second one. This works fine, but the mysql.error.message is not created. If I rename it to mysql.error.message1 in either of the two grok patterns, it works.

A workaround I found is to define another grok pattern definition for GREEDYDATA, like this:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "Pipeline for parsing MySQL slow logs.",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{DATA:mysql.error.timestamp} %{NUMBER:mysql.error.id} \\[%{DATA:mysql.error.level}\\] %{GREEDYDATA:mysql.error.message}",
            "%{LOCALDATETIME:mysql.error.timestamp} %{DATA:mysql.error.name} %{GREEDYDATA1:mysql.error.message}"
          ],
          "ignore_missing": true,
          "pattern_definitions": {
            "LOCALDATETIME": "[0-9]+ %{TIME}",
            "GREEDYDATA1": ".*"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "161209 13:08:33 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql"
      }
    }
  ]
}

[clintongormley]

@talevy any thoughts?

[talevy]

I'll take a look at this

[talevy]

did some digging, this was definitely a bug on my end. Opened the PR above and tested it out with your example in Console, picks it up.

[y0299]

i want use grok processor in elasticsearch to parse some message, which have double quotes in text, like this "message":"192.168.1.2 "GET" 168".
and i use

curl -XPOST 'localhost:9200/_ingest/pipeline/_simulate?pretty' -H 'Content-Type: application/json' -d'
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:ip}\\s"%{REQUEST:request}"\\s%{NUM:num}"],
		"pattern_definitions" : {
          "IP" : "\\S+",
		  "REQUEST" : "\\S+",
		  "NUM" : "\\d+"
        }
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "192.168.1.2 "GET" 168"
    }
  }
  ]
}
'

but has error belows:

"error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Unexpected character ('%' (code 37)): was expecting comma to separate Array entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@290fc36d; line: 9, column: 36]"
    }
  },
  "status" : 400
}

I think the reason is the " (double quotation) cannot support in grok processor patterns，can someone help me resolve this issues?

[talevy]

Hi @y0299!

Indeed you need to escape the " in the javascript.

the offending lines are

        "patterns": ["%{IP:ip}\\s"%{REQUEST:request}"\\s%{NUM:num}"],

and

      "message": "192.168.1.2 "GET" 168"

both should escape the " with a \ that preceeds it.

like so:

        "patterns": ["%{IP:ip}\\s\"%{REQUEST:request}\"\\s%{NUM:num}"],

and

      "message": "192.168.1.2 \"GET\" 168"

Hope that helps!

For the future, it is best to ask for this type of help on The Elastic Discuss Forum since you may even find your questions were already answered there!
Github is still great for posting bugs you may have found with our features in your use-cases.

[y0299]

hi, @talevy ,thanks for giving me advice!
but i use

curl -XPOST 'localhost:9200/_ingest/pipeline/_simulate?pretty' -H 'Content-Type: application/json' -d'
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:ip}\\s\"%{REQUEST:request}\"\\s%{NUM:num}"],
		"pattern_definitions" : {
          "IP" : "\\S+",
		  "REQUEST" : "\\S+",
		  "NUM" : "\\d+"
        }
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "192.168.1.2 "GET" 168"
    }
  }
  ]
}
'

there still has error occurs, like below:

"error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Unexpected character ('G' (code 71)): was expecting comma to separate Object entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@6bcebab4; line: 22, column: 33]"
    }
  },
  "status" : 400
}

do i must change the log to 192.168.1.2 \"GET\" 168 ??

[talevy]

you forgot the offending " in the message.

this should work:

curl -XPOST "http://localhost:9200/_ingest/pipeline/_simulate?pretty" -H 'Content-Type: application/json' -d'
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:ip}\\s\"%{REQUEST:request}\"\\s%{NUM:num}"],
		"pattern_definitions" : {
          "IP" : "\\S+",
		  "REQUEST" : "\\S+",
		  "NUM" : "\\d+"
        }
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "192.168.1.2 \"GET\" 168"
    }
  }
  ]
}'

[y0299]

en, this way can work, but i must change the original log 192.168.1.2 "GET" 168 to 192.168.1.2 \"GET\" 168. but actually some log, which only has " not with a blakslash\"，how can i process this?

[talevy]

@y0299 I'm not sure I completely understand