Add automated tests for security packages #377

andrewkroh · 2020-11-02T15:58:04Z

Automated tests need to be added to the security packages that we migrated over from Filebeat. This will prevent future regressions and save us from having to manually test. The tests should verify:

Data can be collected through each supported input type.
Fields contained in the final documents are defined in package.
There are no data type conflicts with latest ECS release.

I think it would be ideal if all of this testing could be accomplished through system tests with the elastic-package tool. I'll work through setting up a test and see what additional features will be needed and work that via issues in the elastic-package repo.

Input Types

These are the different input types used in packages.

logfile
udp
tcp
tcp with tls
aws-s3
gcp-pubsub
netflow
httpjson
o365audit
windows event log (via .evtx file)

Data Streams to Test

This is every data stream and the inputs that they support.

AWS
- CloudTrail (pipeline test Add pipeline tests for AWS CloudTrail #408)
  - s3
- VPC Flow
  - s3
Auditd
- Log
  - logfile (system & pipeline)
CEF
- Log (has pipeline test)
  - logfile (system test Add system tests for CEF #575)
  - syslog (system test Add system tests for CEF #575)
CheckPoint
- Firewall (pipeline test Add pipeline tests for CheckPoint Firewall #409)
  - logfile (system test Add logfile system test to CheckPoint Firewall #574)
  - udp (system test Add UDP and TCP Syslog tests for CheckPoint Firewall #553)
  - tcp (system test Add UDP and TCP Syslog tests for CheckPoint Firewall #553)
Cisco
- ASA
  - logfile Add pipeline tests to Cisco ASA #414
  - udp (system Add tests to Cisco package #578)
- FTD
  - logfile (system Add tests to Cisco package #578)
  - udp (system Add tests to Cisco package #578)
- IOS
  - logfile (system test Add system test for Cisco IOS #416)
  - syslog (system Add tests to Cisco package #578)
- Meraki
  - logfile (system test Add system test for Cisco Meraki #428)
  - udp (system Add tests to Cisco package #578)
  - tcp (system Add tests to Cisco package #578)
- Nexus
  - logfile (system Add tests to Cisco package #578)
  - udp (system Add tests to Cisco package #578)
  - tcp (system Add tests to Cisco package #578)
CrowdStrike
- Falcon
  - logfile (system test Add system test for CrowdStrike Falcon #429) (add it again Add system test to CrowdStrike Falcon #583)
Fortinet
- FortiClient Endpoint security
  - udp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - tcp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - logfile (system test Add system test for Fortinet client endpoint #432)
- Firewall
  - udp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - tcp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - logfile (pipeline test Add pipeline test for Fortinet Firewall #437) (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
- FortiMail
  - udp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - tcp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - logfile (system test Add system test for Fortinet Fortimail #438)
- Manager/Analyzer
  - udp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - tcp (system Add system tests for Fortinet Firewall udp/tcp inputs #585)
  - logfile (system test Add system test for Fortinet Fortimanager #439)
Google Workspace
- admin
  - httpjson
  - logfile (system test ✅)
- drive
  - httpjson
  - logfile (system test ✅)
- groups
  - httpjson
  - logfile (system test ✅)
- login
  - httpjson
  - logfile (system test ✅)
- saml
  - httpjson
  - logfile (system test ✅)
- user_accounts
  - httpjson
  - logfile (system test ✅)
iptables
- log (pipeline test ✅)
  - logfile (system test ✅)
  - syslog (udp) (system test ✅)
Juniper
- JUNOS (rsa2elk, no pipeline test))
  - udp (system test ✅ Add system tests to Juniper #588)
  - tcp (system test ✅ Add system tests to Juniper #588)
  - logfile (system test ✅ Add system tests to Juniper #588)
- Netscreen (rsa2elk, no pipeline test)
  - udp (system test ✅ Add system tests to Juniper #588)
  - tcp (system test ✅ Add system tests to Juniper #588)
  - logfile (system test Add system test for Juniper Netscreen #444)
- SRX (pipeline test Add pipeline test for Juniper SRX #443)
  - udp (system test ✅ Add system tests to Juniper #588)
  - tcp (system test ✅ Add system tests to Juniper #588)
  - logfile (system test ✅ Add system tests to Juniper #588)
Microsoft
- DHCP
  - logfile (system test Add system test for Microsoft DHCP #447)
Netflow
- Log
  - netflow (system tests ✅ Add system test for Netflow Log (udp) #590)
O365
- Audit
  - o365audit
  - logfile (system test Add system test for Office 365 audit #449)
Okta
- System
  - httpjson
  - logfile (system test Add system test for Okta System #452)
PANW
- PAN-OS
  - syslog (system test Add system test for panw pan-os (syslog input) #594)
  - logfile (system+pipeline Add system and pipeline test for Palo Alto PAN-OS #454)
Suricata
- EVE
  - logfile (system+pipeline Add system and pipeline tests for Suricata EVE #457)
System
- Auth
  - logfile (pipeliine Add pipeline test for System Auth #463)
Windows
- Forwarded
  - winlog
- PowerShell
  - winlog
- PowerShell Operational
  - winlog
- Security
  - winlog
- Sysmon
  - winlog
Zeek (@leehinman)
- logfile (systems test add zeek system tests #448)
Zoom
- Webhook (pipeline Add system and pipeline tests for Zoom #598)
  - http_endpoint (system Add system and pipeline tests for Zoom #598)

elasticmachine · 2020-11-02T15:58:05Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2020-12-17T13:53:01Z

Issues encountered while implementing testing (I will open issues and link here):

Need to be able to execute multiple system tests for a data stream. Some data streams have several input options (syslog, tcp, udp) that need exercised. [System test runner] Support for multiple tests per data stream elastic-package#208
Need to be able to select one of several docker-compose services to use during the tests. The service requirements are different when testing logs vs. syslog+tcp vs. syslog+udp.
Some input types are not easily testable like s3+sqs. In order to add more test coverage of these data streams it would be nice if we could add an input for testing that's not exposed in the UI (like show_user: false). We could expose it in the UI and have it disabled, but I think that would likely lead to confusion. Allow defining inputs only for testing package-spec#97
Golden file testing for system tests. This would be useful for detecting breaking changes in agent/collector/inputs.

* Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

#24077) * Sync fixes from Integration Package Testing (#23424) * Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

andrewkroh added enhancement New feature or request Team:Security-External Integrations labels Nov 2, 2020

epixa added the v7.11.0 label Nov 19, 2020

This was referenced Nov 19, 2020

Add pipeline tests for AWS CloudTrail #408

Merged

Add pipeline tests for CheckPoint Firewall #409

Merged

Add pipeline tests to Cisco ASA #414

Merged

Add system test for Cisco IOS #416

Merged

leehinman mentioned this issue Dec 4, 2020

add zeek system tests #448

Merged

2 tasks

This was referenced Dec 5, 2020

Add system test for Office 365 audit #449

Merged

Add system test for Okta System #452

Merged

Add system and pipeline test for Palo Alto PAN-OS #454

Merged

Add system and pipeline tests for Suricata EVE #457

Merged

andrewkroh mentioned this issue Dec 14, 2020

Add pipeline test for System Auth #463

Merged

2 tasks

andrewkroh mentioned this issue Dec 18, 2020

[System test runner] Support for multiple tests per data stream elastic/elastic-package#208

Closed

jamiehynds added 7.12 candidate Theme: just_ingest_it and removed v7.11.0 labels Jan 6, 2021

andrewkroh mentioned this issue Jan 11, 2021

Sync fixes from Integration Package Testing elastic/beats#23424

Merged

3 tasks

andrewkroh mentioned this issue Feb 9, 2021

Add system test for Zeek FTP, OCSP #676

Merged

2 tasks

jamiehynds removed the 7.12 candidate label Feb 15, 2021

andrewkroh closed this as completed Feb 16, 2021

andrewkroh mentioned this issue Feb 16, 2021

Cherry-pick #23424 to 7.x: Sync fixes from Integration Package Testing elastic/beats#24077

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automated tests for security packages #377

Add automated tests for security packages #377

andrewkroh commented Nov 2, 2020 •

edited

Loading

elasticmachine commented Nov 2, 2020

andrewkroh commented Dec 17, 2020 •

edited

Loading

Add automated tests for security packages #377

Add automated tests for security packages #377

Comments

andrewkroh commented Nov 2, 2020 • edited Loading

Input Types

Data Streams to Test

elasticmachine commented Nov 2, 2020

andrewkroh commented Dec 17, 2020 • edited Loading

andrewkroh commented Nov 2, 2020 •

edited

Loading

andrewkroh commented Dec 17, 2020 •

edited

Loading