V11 filter syntax

Filter/Copy syntax and nested match

We have several ideas to go. We need feedback!

Ideas

• idea 1: Add <filter> syntax. Add filter_ plugins.
• idea 2: No filter_ plugins (keep using out_* plugins). Users use nested <match> to filter/convert records. <source> can't have nested <match> sections.
• Idea 3: Similar to 2 but <source> outputs records ONLY to the nested <match> (<source> must have nested <match> sections).
• Idea 4: Similar to 3 but <source> can't be in <label> (must be toplevel). <source> and <match> can have to_label attribute.

Welcome to add more examples to this page!

Example 1

Idea 1

<source>
  type forward
</source>

<filter **>
  type copy

  # real-time aggregation
  <filter>
    type groupcounter
  </filter>
 
  <match>
    type forward
  </match>
</filter>

# archive
<match **>
  type s3
</match>

Idea 2

<source>
  type forward
</source>

# real-time aggregation
<copy **>
  type groupcounter
  <match>
    type forward
  </match>
</copy>

# archive
<match **>
  type s3
</match>

Idea 3

<source>
  type forward

  # real-time aggregation
  <copy **>
    type groupcounter
    <match>
      type forward
    </match>
  </copy>

  # archive
  <match **>
    type s3
  </match>
</source>

Idea 4

(Same with idea 2)

Example 2

Idea 1

<source>
  type forward
  # only forward has copy to groupcounter + forward

  <filter>
    type copy

    <filter>
      type groupcounter
    </filter>
 
    <match>
      type forward
    </match>
  </filter>
</source>

<source>
  type http
  # http goes to s3 directly
</source>

# archive
<match>
  type s3
</match>

Idea 2

<label aggregate_and_archive>
  # forwarded data go to aggregate, then to archive
  <source>
    type forward
    # source can't have nested match
  </source>

  <copy>
    type groupcounter
    <match>
      type forward
    </match>
  </copy>

  <match>
    type redirect
    label archive
  </match>
</label>

<label archive>
  # http go to archive directly
  <source>
    type http
  </source>

  <match>
    type s3
  </match>
</label>

Idea 3

<source>
  type forward
  # source can't have nested match

  <copy>
    type groupcounter
    <match>
      type forward
    </match>
  </copy>

  <match>
    type redirect
    label archive
  </match>
</source>

<source>
  type http

  <match>
    type redirect
    label archive
  </match>
</source>

<label archive>
  <match>
    type s3
  </match>
</label>

Idea 4

# forwarded data go to aggregate, then to archive
<source>
  type forward
  to_label @aggregate_and_archive
</source>

# http go to archive directly
<source>
  type http
  to_label @archive
</source>

<label @aggregate_and_archive>
  <copy>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR to_label
  </copy>

  <match>
    type redirect
    to_label @archive
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>

Idea 5

# forwarded data go to aggregate, then to archive
<source => @aggregate>
  type forward
</source>

# http go to archive directly
<source => @archive>
  type http
</source>

<label @aggregate>
  <match>
    type groupcounter
    <match>
      type forward
    </match>
    # groupcounter needs to have nested <match> OR " => @label"
  </match>
  <match>  # same level match means always copy
    type redirect
    to_label @archive
  </match>
</label>

<label @archive>
  <match>
    type s3
  </match>
</label>