Grok Processor does not support non-(a-zA-Z_) field characters for field names

[talevy]

example Ingest pipeline that fails: @ field name.

POST _ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "grok" : {
          "field" : "foo",
          "patterns" : [
            "%{WORD:@}"
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_type": "type",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}

exception:

{
  "docs": [
    {
      "error": {
        "root_cause": [
          {
            "type": "exception",
            "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [bar]",
            "header": {
              "processor_type": "grok"
            }
          }
        ],
        "type": "exception",
        "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [bar]",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [bar]",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Provided Grok expressions do not match field value: [bar]"
          }
        },
        "header": {
          "processor_type": "grok"
        }
      }
    }
  ]
}

The Grok Parser in Ingest requires that field names match a-zA-Z_, this should be expanded to support all unicode characters.

Must update the regex here to do so: https://github.com/talevy/elasticsearch/blob/82f7bfad98253e94305136df481cd1c7dc4e8ca8/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/Grok.java#L47-L47

might be a relevant Issue in Joni: jruby/joni#13

[talevy]

Maybe something like this would do the job

"(?::(?<subname>[^%:{}]+))?" +

accept all non %, :, }, { characters

cc/ @ruflin

[clintongormley]

@talevy ALL unicode characters? Including vertical tab, non-breaking space, newline, combining characters? Maybe you want to use unicode properties instead, and include allowed punctuation?

[talevy]

@clintongormley maybe not all :) does Elasticsearch have a specific allowed set of field name character? ideally it should follow that as much as it can, right?

mainly used as an example to remind me what to change. I am still in the process of investigating how to properly declare these specific unicode character types within Joni

[clintongormley]

No we don't :( You can quite happily create a field named {. We really need to get back to defining allowed characters in named entities (#9059)

[talevy]

OK, then I guess we should allow that here as well.

looking at Joni, I think I found a solution. called the [:graph:] unicode character match. They have quite extensive tests here: https://github.com/jruby/jruby/blob/f7746d0be94f5d9f7cd27af610f1bc5dd2622794/spec/ruby/language/regexp/character_classes_spec.rb

and

  it "matches Unicode letter characters with [[:graph:]]" do
  it "matches Unicode digits with [[:graph:]]" do
  it "matches Unicode marks with [[:graph:]]" do
  it "matches Unicode punctuation characters with [[:graph:]]" do
  it "match Unicode format characters with [[:graph:]]" do
  it "match Unicode private-use characters with [[:graph:]]" do
  it "doesn't match Unicode control characters with [[:graph:]]" do

another that is similar is called [:word:] which matches the below:

  it "matches Unicode lowercase characters with [[:word:]]" do
  it "matches Unicode uppercase characters with [[:word:]]" do
  it "matches Unicode title-case characters with [[:word:]]" do
  it "matches Unicode decimal digits with [[:word:]]" do
  it "matches Unicode marks with [[:word:]]" do
  it "match Unicode Nl characters with [[:word:]]" do
  it "doesn't match Unicode No characters with [[:word:]]" do
  it "doesn't match Unicode punctuation characters with [[:word:]]" do
  it "doesn't match Unicode control characters with [[:word:]]" do
  it "doesn't match Unicode format characters with [[:word:]]" do
  it "doesn't match Unicode private-use characters with [[:word:]]" do

so the change could be:

"(?::(?<subname>[[[:graph:]]]+))?" +

running locally, and it looks pretty good, not sure we want to match all of the types of unicode characters as is described by the test handles above ^^

I'll start a PR and incorporate these different character tests on our end as well

[elasticmachine]

Pinging @elastic/es-core-infra