[Filebeat] fix organization and customer prefix for aws/cloudtrail (#…

…23126) (#23138)

* fix organization and customer prefix for aws/cloudtrail

- use ^.*AWSLogs as beginning of prefix, matches:
  + custom-prefix/AWSLogs/1234567890/CloudTrail/
  + o-xxxxxxx/AWSLogs/1234567890/CloudTrail/
  + AWSLogs/1234567890/CloudTrail/

Closes #23109

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit b4aba8e)

CHANGELOG.next.asciidoc

@@ -243,6 +243,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix aws s3 overview dashboard. {pull}23045[23045]
 - Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072]
 - Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966]
+- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126]
 
 *Heartbeat*
 
@@ -674,4 +675,3 @@ port. {pull}19209[19209]
 
 
 
-

x-pack/filebeat/module/aws/cloudtrail/config/s3.yml

@@ -2,16 +2,16 @@ type: s3
 queue_url: {{ .queue_url }}
 file_selectors:
 {{ if .process_cloudtrail_logs }}
-  - regex: '^AWSLogs/\d+/CloudTrail/'
+  - regex: 'AWSLogs/\d+/CloudTrail/'
     expand_event_list_from_field: 'Records'
 {{ end }}
 
 {{ if .process_digest_logs }}
-  - regex: '^AWSLogs/\d+/CloudTrail-Digest/'
+  - regex: 'AWSLogs/\d+/CloudTrail-Digest/'
 {{ end }}
 
 {{ if .process_insight_logs }}
-  - regex: '^AWSLogs/\d+/CloudTrail-Insight/'
+  - regex: 'AWSLogs/\d+/CloudTrail-Insight/'
     expand_event_list_from_field: 'Records'
 {{ end }}