Skip to content

V11 filter syntax

Jump to bottom Edit New page
FURUHASHI Sadayuki edited this page Nov 13, 2013 · 32 revisions

Filter/Copy syntax and nested match

We have several ideas to go. We need feedback!

## Ideas
  • idea 1: Add <filter> syntax. Add filter_ plugins.
  • idea 2: No filter_ plugins (keep using out_* plugins). Users use nested <match> to filter/convert records. <source> can't have nested <match> sections.
  • Idea 3: Similar to 2 but <source> outputs records ONLY to the nested <match> (<source> must have nested <match> sections).
  • Idea 4: Similar to 3 but <source> can't be in <label> (must be toplevel). <source> and <match> can have to_label attribute.
  • Idea 5:

Welcome to add more examples to this page!

## Example A

Scenario:

  • Read data from TCP
  • Streaming aggregation to MongoDB
  • Archive to S3

Idea 1

<source>
  type forward
</source>

<filter **>
  type copy

  # real-time aggregation
  <filter>
    type groupcounter
  </filter>
 
  <match>
    type mongodb
  </match>
</filter>

# archive
<match **>
  type s3
</match>

Idea 2

<source>
  type forward
</source>

# real-time aggregation
<copy **>
  type groupcounter
  <match>
    type mongodb
  </match>
</copy>

# archive
<match **>
  type s3
</match>

Idea 3

<source>
  type forward

  # real-time aggregation
  <copy **>
    type groupcounter
    <match>
      type mongodb
    </match>
  </copy>

  # archive
  <match **>
    type s3
  </match>
</source>

Idea 4

(Same with idea 2)

Idea 5

<source => @aggregate_and_archive>
  type forward
</source>

<label @aggregate_and_archive>
  # real-time aggregation
  <match **>
    type groupcounter
    <match>
      type mongodb
    </match>
  </match>

  # archive
  <match **>
    type s3
  </match>
</label>
## Example B

Scenario:

  • Read from TCP and do streaming aggregation
  • Read from HTTP
  • Archive all logs to S3

Idea 1

<source>
  type forward
  # only forward has copy to groupcounter + forward

  <filter>
    type copy

    <filter>
      type groupcounter
    </filter>
 
    <match>
      type forward
    </match>
  </filter>
</source>

<source>
  type http
  # http goes to s3 directly
</source>

# archive
<match>
  type s3
</match>

Idea 2

<label aggregate_and_archive>
  # forwarded data go to aggregate, then to archive
  <source>
    type forward
    # source can't have nested match
  </source>

  <copy>
    type groupcounter
    <match>
      type forward
    </match>
  </copy>

  <match>
    type redirect
    label archive
  </match>
</label>

<label archive>
  # http go to archive directly
  <source>
    type http
  </source>

  <match>
    type s3
  </match>
</label>

Idea 3

<source>
  type forward
  # source can't have nested match

  <copy>
    type groupcounter
    <match>
      type forward
    </match>
  </copy>

  <match>
    type redirect
    label archive
  </match>
</source>

<source>
  type http

  <match>
    type redirect
    label archive
  </match>
</source>

<label archive>
  <match>
    type s3
  </match>
</label>

Idea 4

# forwarded data go to aggregate, then to archive
<source>
  type forward
  to_label @aggregate_and_archive
</source>

# http go to archive directly
<source>
  type http
  to_label @archive
</source>

<label @aggregate_and_archive>
  <copy>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR to_label
  </copy>

  <match>
    type redirect
    to_label @archive
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>

Idea 5

# forwarded data go to aggregate, then to archive
<source => @aggregate>
  type forward
</source>

# http go to archive directly
<source => @archive>
  type http
</source>

<label @aggregate>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR " => @label"
  </match>

  # same level match means always copy
  <match>
    type redirect
    to_label @archive
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>
## Example C

Scenario:

  • Read from system, app, or other logs from TCP or UNIX socket
  • Do streaming aggregation only system and app logs
  • Archive all logs to S3

Idea 1

<source>
  type forward
</source>

<source>
  type unix
</source>

<filter {syslog,applog}.**>
  type copy

  <filter>
    type groupcounter
  <filter>
  <match>
    type forward
  </match>
</filter>

<match>
  type s3
</match>

Idea 2

<source>
  type forward
</source>

<source>
  type unix
</source>

<copy {syslog,applog}.**>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
  </match>
</copy>

<match>
  type s3
</match>

Idea 3

<source>
  type forward
  
  <match>
    type redirect
    label @aggregate
  </match>
</source>

<source>
  type unix

  <match>
    type redirect
    label aggregate
  </match>
</source>

<label aggregate>
  <copy {syslog,applog}.**>
    <match>
      type groupcounter
      <match>
        type forward
      </match>
    </match>
  </filter>

  <match>
    type s3
  </match>
</label>

Idea 4

<source>
  type forward
</source>

<source>
  type unix
</source>

<copy {syslog,applog}.**>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
  </match>
</copy>

<match>
  type s3
</match>

Idea 4 with :route built-in plugin

<source>
  type forward
</source>

<source>
  type unix
</source>

<match **>
  type route
  <rule {syslog,applog}.**>
    to_label @aggregate
    copy
  </rule>
  <rule>
    to_label @archive
  </rule>
</copy>

<label @aggregate>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>

Idea 5

# forwarded data go to aggregate, then to archive
<source => @split>
  type forward
</source>

# passed data from unix socket are routed by its tags
<source => @split>
  type unix
</source>

<label @split>
  <match>
    type redirect
    <case {syslog,applog}.** => @aggregate />
    <else => @archive />
  </match>
</label>

<label @aggregate>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR " => @label"
  </match>

  # same level match means always copy
  <match>
    type redirect
    to_label @archive
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>

Idea 5 (shorter configuration: same rule)

# forwarded data go to aggregate, then to archive
<source => @out>
  type forward
</source>

# passed data from unix socket are routed by its tags
<source => @out>
  type unix
</source>

<label @out>
  <match {syslog,applog}.**>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR " => @label"
  </match>
  # same level match means always copy
  <match **>
    type s3
  </match>
</label>
Add a custom footer
Add a custom sidebar
Clone this wiki locally