Enhance Suricata pipeline to handle destination.domain being set

[andrewkroh]

This replaces the usage of a rename processor with an append + remove processor.
Then a script processor is used to deduplicate the domains.

This makes the pipeline compatible with the reverse dns processor being used on the Beat side.

Fixes #10510

[adriansr]

LGTM

[webmat]

I wonder if domain containing an array will be a problem.

Suggestion: dedupe before if (domain.length == 1).

[andrewkroh]

I've been sending in arrays for a few weeks and have made the team aware that this can be an array.

[andrewkroh]

And I implemented your suggestion. Good idea.

[webmat]

Yeah we'll need to start looking into defining this more clearly in ECS.

But I agree that we have to support arrays here, because of the reverse DNS possibility.

[webmat]

Ah wait. You're only keeping the first of the array? 🤔How do you know it's the interesting entry?

[andrewkroh]

No. This is the code. It doesn't drop any data.

    def domain = ctx.destination?.domain; 
    if (domain instanceof Collection) {
      // Deduplicate
      ctx.destination.domain = ctx.destination.domain.stream().distinct().collect(Collectors.toList());
      
      // Make it a plain old string if there's only one item.
      if (domain.length == 1) { 
        ctx.destination.domain = domain[0]; 
      } 
    }

[webmat]

Can the reverse DNS processor return more than one DNS entry for a given IP?

If so, this only keeps the first one, right?

Or does the processor always return an array of one?

[andrewkroh]

Oh, I see the problem. When I refactored it from the initial version I broke it. The fixed version is:

    def domain = ctx.destination?.domain; 
    if (domain instanceof Collection) {
      // Deduplicate
      domain = domain.stream().distinct().collect(Collectors.toList());
      
      // Make it a plain old string if there's only one item.
      if (domain.length == 1) { 
        domain = domain[0]; 
      }
      
      // Set the value
      ctx.destination.domain = domain;
    }

Does this address your concern?

[webmat]

Ah, happy to see we weren't just going around in circles :-)

Yes, this makes sense now :-)

[webmat]

LGTM