[Filebeat] Replace copy_from with templated value

[andrewkroh]

What does this PR do?

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of copy_from in set processors.

Why is it important?

This will make some modules work with earlier Elasticsearch versions.

Checklist

• My code follows the style guidelines of this project
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates [Filebeat] Module incompatibility with older ES/Kibana versions #26629

[andrewkroh]

@adriansr I opened this to manually remove copy_from. Would you want to help me with this by pushing a change for the two you mention in #26593 (comment).

[andrewkroh]

We should probably put a check in place to catch the introduction of copy_from in the future. I'm just not sure where that should go.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: andrewkroh commented: run tests

• Start Time: 2021-07-06T03:13:20.310+0000

• Duration: 92 min 7 sec

• Commit: b1929b7

Test stats 🧪

Test	Results
Failed	0
Passed	14209
Skipped	2312
Total	16521

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14209
Skipped	2312
Total	16521

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Thanks @adriansr. LGTM

[legoguy1000]

Instead of actually updating the pipelines themselves, why not add an additional function to automatically change the processor if connecting to an older version of ES? What was just merged in #26676

filebeat/fileset/compatibility.go

	{
		procType: "set",
		checkVersion: func(esVersion *common.Version) bool {
			return esVersion.LessThan(common.MustNewVersion("7.11.0"))
		},
		adaptConfig: replaceSetCopyFrom,
	},

// replaceSetIgnoreEmptyValue replaces ignore_empty_value option with an if
// statement so ES less than 7.11 will work.
func replaceSetCopyFrom(config map[string]interface{}, log *logp.Logger) compatAction {
	value_field, ok := config["copy_from"].(string)
	if !ok {
		return keepProcessor
	}

	log.Debug("Removing unsupported 'copy_from' from set processor.")
	delete(config, "copy_from")

	val, ok := config["value"].(string)
	if ok {
		delete(config, "value")
	}
        new_value := "{{{" + value_field + "}}}"
	log.Debug("Adding value %s to replace 'copy_from' in set processor.", new_value)
	config["value"] = new_value
	return keepProcessor
}

[andrewkroh]

@legoguy1000 We considered that approach in #26593, but during review we identified some edge cases that we couldn't handle by automatic rewriting.

[andrewkroh]

run tests

[legoguy1000]

@legoguy1000 We considered that approach in #26593, but during review we identified some edge cases that we couldn't handle by automatic rewriting.

Does the templated value not work with objects?

[andrewkroh]

It will run, but the behavior is not the same as with copy_from. You end up with a string instead of an object.

[andrewkroh]

@Mergifyio backport 7.x

[mergify]

Command backport 7.x: success

Backports have been created

• #26964 [Filebeat] Replace copy_from with templated value (backport #26631) has been created for branch 7.x