[ci] Trigger stages on changes to vendor/

[andrewkroh]

The conditional logic for triggering build stages in the Jenkins pipeline should also account for changes to the vendor/ directory. Changes to the dependencies stored in the vendor/ directory can impact any project so they should all be built and tested.

Related code: https://github.com/elastic/beats/blob/master/Jenkinsfile#L721-L787

[andrewkroh]

@urso also suggested to exclude changes from libbeat's asciidoc files otherwise a documentation change would trigger all of CI.

[urso]

These checks are still quite coarse. We could refine them even further. Some ideas:

• do not run filebeat/metricbeat/X tests if libbeat/tests/system/test_*.py has been touched only
• Use tools to check actual code dependencies if vendor/libbeat is updated. One can create a list of all packages filebeat depends on like this: $go list -mod=vendor -f '{{ .ImportPath }}{{ "\n" }}{{ join .Deps "\n" }}' .\filebeat (using -tags or GOARCH, GOOS one can influence the outcome). A vendor update for metricbeat will not trigger filebeat tests then. Due to the change to go mod the package names require some post-processing (e.g. remove v7, remove stdlib packages, exclude packages not in vendor or beginning with github.com/elastic/beats).

[kuisathaverat]

@urso your recent PR #17358 I think that covers the vendor request here.

Use tools to check actual code dependencies if vendor/libbeat is updated. One can create a list of all packages filebeat depends on like this: $go list -mod=vendor -f '{{ .ImportPath }}{{ "\n" }}{{ join .Deps "\n" }}' .\filebeat (using -tags or GOARCH, GOOS one can influence the outcome). A vendor update for metricbeat will not trigger filebeat tests then. Due to the change to go mod the package names require some post-processing (e.g. remove v7, remove stdlib packages, exclude packages not in vendor or beginning with github.com/elastic/beats).

about this, it returns a list of packages I guess those packages match with a path on vendors/ so we can see if there are changes on those path to trigger a beat, isn't it?

>go list -mod=vendor -f '{{ .ImportPath }}{{ "\n" }}{{ join .Deps "\n" }}' ./filebeat|grep '^vendor'|awk '{print "^"$1"/.*"}'

^vendor/golang.org/x/crypto/chacha20poly1305/.*
^vendor/golang.org/x/crypto/cryptobyte/.*
^vendor/golang.org/x/crypto/cryptobyte/asn1/.*
^vendor/golang.org/x/crypto/curve25519/.*
^vendor/golang.org/x/crypto/hkdf/.*
^vendor/golang.org/x/crypto/internal/chacha20/.*
^vendor/golang.org/x/crypto/internal/subtle/.*
^vendor/golang.org/x/crypto/poly1305/.*
^vendor/golang.org/x/net/dns/dnsmessage/.*
^vendor/golang.org/x/net/http/httpguts/.*
^vendor/golang.org/x/net/http/httpproxy/.*
^vendor/golang.org/x/net/http2/hpack/.*
^vendor/golang.org/x/net/idna/.*
^vendor/golang.org/x/net/route/.*
^vendor/golang.org/x/sys/cpu/.*
^vendor/golang.org/x/text/secure/bidirule/.*
^vendor/golang.org/x/text/transform/.*
^vendor/golang.org/x/text/unicode/bidi/.*
^vendor/golang.org/x/text/unicode/norm/.*

[urso]

The list command reports all dependencies, including vendor, Beats internal packages, and stdlib packages. The vendor output is even incomplete. For example the k8s client is missing from your output as it is only reported as k8s.io/api/.

The idea is to check for Beats internal changes as well. For example if we do a change to go files in libbeat, than filebeat might be affected or not.
In metricbeat we have some of the python testing files already next to the module implementation. By computing transitive dependencies we can check if a module needs to be tested or not.

This would affect auditbeat more, though. E.g. right now if a metricbeat module is touched we run all CI for auditbeat (which depends on the metricbeat code base) and metricbeat. But in fact we would just need to test a single module.
Fix for auditbeat dependencies: #17393

[kuisathaverat]

This will return all the dependencies of x-pack/heartbeat by replacing github.com/elastic/beats/v7 with nothing it also returns the heartbeat and libbeat dependency paths, even itself 🤔

go list -mod=vendor -f '{{ .ImportPath }}{{ "\n" }}{{ join .Deps "\n" }}' ./x-pack/heartbeat|awk '{print $1"/.*"}'|sed -e "s#github.com/elastic/beats/v7/##g"

[urso]

Correct. The command reports all packages and their transitive dependencies for main.go. Including itself and libbeat. But it only includes packages from libbeat it actually depends on, not all libbeat. On linux/windows it might even report a different set of packages due to build/linking being different for different OSes. E.g. in metricbeat we have modules that are only compiled into metricbeat for the windows build only.