Add system test for Netflow Log (udp)

[andrewkroh]

What does this PR do?

Add a system test for the netflow input in the Netflow Log data stream.

Initially the errors were:

netflow/log netflow:
[0] field "netflow.art_client_network_time_sum" is undefined
[1] field "netflow.art_count_late_responses" is undefined
[2] field "netflow.art_count_responses" is undefined
[3] field "netflow.art_count_retransmissions" is undefined
[4] field "netflow.art_count_transactions" is undefined
[5] field "netflow.art_network_time_sum" is undefined
[6] field "netflow.art_response_time_sum" is undefined
[7] field "netflow.art_server_network_time_sum" is undefined
[8] field "netflow.art_server_response_time_maximum" is undefined
[9] field "netflow.art_server_response_time_sum" is undefined
[10] field "netflow.art_total_response_time_sum" is undefined
[11] field "netflow.art_total_transaction_time_sum" is undefined
[12] field "netflow.waasoptimization_segment" is undefined
[13] parsing field value failed: field "network.iana_number"'s Go type, float64, does not match the expected field type: keyword (field value: 6)

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.

Related issues

• Relates Add automated tests for security packages #377
• Requires [Filebeat] Missing netflow field definitions for vendor fields beats#23771

Screenshots

Run system tests for the package
--- Test results for package: netflow - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ netflow │ log         │ system    │ netflow   │ PASS   │ 49.80578431s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: netflow - END   ---
Done
Run asset tests for the package
--- Test results for package: netflow - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                            │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ netflow │             │ asset     │ dashboard netflow-14387a13-53bc-43a4-b9cd-63977aa8d87c is loaded     │ PASS   │ 8.567861395s │
│ netflow │             │ asset     │ dashboard netflow-34e26884-161a-4448-9556-43b5bf2f62a2 is loaded     │ PASS   │       2.45µs │
│ netflow │             │ asset     │ dashboard netflow-38012abe-c611-4124-8497-381fcd85acc8 is loaded     │ PASS   │        421ns │
│ netflow │             │ asset     │ dashboard netflow-77326664-23be-4bf1-a126-6d7e60cfc024 is loaded     │ PASS   │        412ns │
│ netflow │             │ asset     │ dashboard netflow-94972700-de4a-4272-9143-2fa8d4981365 is loaded     │ PASS   │        392ns │
│ netflow │             │ asset     │ dashboard netflow-acd7a630-0c71-4840-bc9e-4a3801374a32 is loaded     │ PASS   │        417ns │
│ netflow │             │ asset     │ dashboard netflow-c64665f9-d222-421e-90b0-c7310d944b8a is loaded     │ PASS   │        421ns │
│ netflow │             │ asset     │ dashboard netflow-feebb4e6-b13e-4e4e-b9fc-d3a178276425 is loaded     │ PASS   │        374ns │
│ netflow │             │ asset     │ visualization netflow-0177bf1a-cba8-4ba6-a1d7-73caed86ffc2 is loaded │ PASS   │        560ns │
│ netflow │             │ asset     │ visualization netflow-036aef95-ec90-468d-ad7c-3cc4405e9e81 is loaded │ PASS   │        385ns │
│ netflow │             │ asset     │ visualization netflow-0528bc66-6981-400a-a02d-c1d221b38890 is loaded │ PASS   │        409ns │
│ netflow │             │ asset     │ visualization netflow-0b2818fd-aecc-4bef-b566-9466eb702ae4 is loaded │ PASS   │        534ns │
│ netflow │             │ asset     │ visualization netflow-12aad647-c45d-4667-a029-152c1a97cbbc is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-14c7136d-b4aa-4367-9461-52bf8b5c4796 is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-15295ea6-ba84-47db-8ced-9312abbf495c is loaded │ PASS   │        412ns │
│ netflow │             │ asset     │ visualization netflow-1558508d-591c-49be-bef4-85fdac18a960 is loaded │ PASS   │        375ns │
│ netflow │             │ asset     │ visualization netflow-15e2a267-2495-4df2-a121-abe410d2f18c is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-16262df9-a979-4136-935e-d883c7d373d7 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-1cd36f5d-d9c7-4098-acdb-14d312ecfb72 is loaded │ PASS   │        373ns │
│ netflow │             │ asset     │ visualization netflow-1cf30eac-aae8-47fa-a156-37f6346d2d5a is loaded │ PASS   │      2.484µs │
│ netflow │             │ asset     │ visualization netflow-1e74d5cb-556d-42ee-8042-88f6c1af47f0 is loaded │ PASS   │        464ns │
│ netflow │             │ asset     │ visualization netflow-201d7dd1-a880-4a64-b631-db5629340db9 is loaded │ PASS   │        398ns │
│ netflow │             │ asset     │ visualization netflow-2316bb53-d98a-4f0f-8cd8-51e9fb317823 is loaded │ PASS   │        425ns │
│ netflow │             │ asset     │ visualization netflow-248e00b4-8fc2-406f-8907-729d5380aaa7 is loaded │ PASS   │        364ns │
│ netflow │             │ asset     │ visualization netflow-2b3d4e86-2254-4033-8fe3-ce4753fafd03 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-2dca3025-692c-4876-8bcc-e0b248dc9819 is loaded │ PASS   │        391ns │
│ netflow │             │ asset     │ visualization netflow-30cd1009-2925-4c9b-820d-d689f5d1efda is loaded │ PASS   │        377ns │
│ netflow │             │ asset     │ visualization netflow-31708a70-4957-4a8a-8065-5c88a344ad02 is loaded │ PASS   │        399ns │
│ netflow │             │ asset     │ visualization netflow-31b5f6fd-eb9d-4e97-90fd-367062ef217f is loaded │ PASS   │        368ns │
│ netflow │             │ asset     │ visualization netflow-32e712ed-fa15-4db7-8575-8476e8d65b03 is loaded │ PASS   │        391ns │
│ netflow │             │ asset     │ visualization netflow-3a4209e2-281c-467e-b5cb-315bf4a2661f is loaded │ PASS   │        388ns │
│ netflow │             │ asset     │ visualization netflow-3dec20c0-0d4f-43ef-8864-3779e1a1b33f is loaded │ PASS   │        498ns │
│ netflow │             │ asset     │ visualization netflow-3e27fb83-b3e3-4c15-b999-ed6da49b7a86 is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-441c6c50-fa1a-489c-96c6-76f7925dea24 is loaded │ PASS   │        373ns │
│ netflow │             │ asset     │ visualization netflow-4ac97841-c89f-4d50-b3c6-6253f7e1dd1a is loaded │ PASS   │        435ns │
│ netflow │             │ asset     │ visualization netflow-4bb0255e-18ed-45e4-bfb9-de8e35b12094 is loaded │ PASS   │        367ns │
│ netflow │             │ asset     │ visualization netflow-5292a65b-c532-422a-9008-1251a8073a3a is loaded │ PASS   │        363ns │
│ netflow │             │ asset     │ visualization netflow-5303e99b-389c-47b7-ae7a-945c5a92ba49 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-57e13a20-e94f-4465-a942-42148634a1d2 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-5ccac452-e90a-4dde-ae9b-1be36ce3f761 is loaded │ PASS   │        378ns │
│ netflow │             │ asset     │ visualization netflow-5cfb2c9a-4815-4a25-9d7e-ab0ef55ffe63 is loaded │ PASS   │        380ns │
│ netflow │             │ asset     │ visualization netflow-5d868836-c7b2-4812-bf47-4838aac281d9 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-63ef5338-fdf2-488e-b78a-f0e98daccc95 is loaded │ PASS   │        363ns │
│ netflow │             │ asset     │ visualization netflow-67fdca65-a9df-47f0-a8a4-1e8b056325de is loaded │ PASS   │        466ns │
│ netflow │             │ asset     │ visualization netflow-681f0ce4-d828-4a99-b643-0c0715530050 is loaded │ PASS   │        372ns │
│ netflow │             │ asset     │ visualization netflow-6bbd6712-494a-4fd9-b3d3-757304681f0f is loaded │ PASS   │        424ns │
│ netflow │             │ asset     │ visualization netflow-717cd7c7-bfca-435d-8ee7-38259927aade is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-751ecb6f-11c3-458d-b039-f6d57a6379fa is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-7d447b22-89dc-4f32-b549-4b8620af4d76 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-7fa6cb0a-518d-46e9-a228-15cd4253a957 is loaded │ PASS   │        385ns │
│ netflow │             │ asset     │ visualization netflow-85ebf558-402b-45d2-a186-e15f8673ec07 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-8f83cf97-4a48-421f-8db5-690297d1f4fb is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-a14c3248-952d-42aa-bd7d-9b39157a776f is loaded │ PASS   │        397ns │
│ netflow │             │ asset     │ visualization netflow-a1704d46-15fc-41c2-851d-796ceb49877f is loaded │ PASS   │        444ns │
│ netflow │             │ asset     │ visualization netflow-a5efa3dd-f53a-4d14-9d3f-ee73345fd93d is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-a685420e-c45f-4b62-932b-5b76ac8b8ca2 is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-abfa0b19-60cd-4984-9c3d-02ebf0aa1dfb is loaded │ PASS   │        384ns │
│ netflow │             │ asset     │ visualization netflow-ae334aec-31fa-4df7-a064-40b18831d819 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-aed09724-0a69-4331-84f5-3d2067c43930 is loaded │ PASS   │        366ns │
│ netflow │             │ asset     │ visualization netflow-af707b01-29f1-462b-b279-6d2e803f3645 is loaded │ PASS   │        360ns │
│ netflow │             │ asset     │ visualization netflow-b02c2713-17f0-41dd-88a3-ce33b446f19d is loaded │ PASS   │        444ns │
│ netflow │             │ asset     │ visualization netflow-b677cd82-b33e-49b3-8b6e-0e110177b163 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-c27c6a3b-93ee-44d5-8d0c-9b097e575f52 is loaded │ PASS   │        380ns │
│ netflow │             │ asset     │ visualization netflow-c54f5529-e6d7-4c26-8e8e-3b35de132035 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-cccff92f-cb71-49a9-9caf-84867751d31e is loaded │ PASS   │        395ns │
│ netflow │             │ asset     │ visualization netflow-cf399a85-e348-4ac1-a399-e8f5a44114c4 is loaded │ PASS   │        366ns │
│ netflow │             │ asset     │ visualization netflow-d27b5d74-b3b4-4311-a0e6-08ff8f4345df is loaded │ PASS   │        376ns │
│ netflow │             │ asset     │ visualization netflow-d3df8d28-65f8-4ea1-8b33-f479380a0600 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-d41a9663-e5ad-47a7-955e-3803ae4e23c0 is loaded │ PASS   │        388ns │
│ netflow │             │ asset     │ visualization netflow-d4e6520a-9ced-47c9-a8f2-7246e8cbd2d3 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-d5568704-e30b-4108-bb49-06a9b8dce6a6 is loaded │ PASS   │      7.403µs │
│ netflow │             │ asset     │ visualization netflow-d59a031c-70d6-47d7-966d-7fcb805be9be is loaded │ PASS   │        492ns │
│ netflow │             │ asset     │ visualization netflow-ddd27657-c3c8-4f82-8059-6d7763dd599b is loaded │ PASS   │        376ns │
│ netflow │             │ asset     │ visualization netflow-e822f94c-5f65-4963-a540-74ca9c25bd2d is loaded │ PASS   │        378ns │
│ netflow │             │ asset     │ visualization netflow-e99dc327-03de-4561-9e0c-f550710125c2 is loaded │ PASS   │        372ns │
│ netflow │             │ asset     │ visualization netflow-e9ad835b-b2f2-42d3-a3e7-555a593deacf is loaded │ PASS   │        419ns │
│ netflow │             │ asset     │ visualization netflow-ebea013f-9b5b-4f61-a9c8-c62bebf62ae9 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-f27c1479-0625-4cdc-92de-672e47db0f87 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-f4c8cb5a-7336-449e-ab99-6e867b435b85 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-f531f957-e8c0-497a-ad41-ef39c2d29671 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-f668ecdb-eec7-44c6-9060-26aaf9fc8404 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-f75063c7-48b7-4de4-b8cb-d07eb2cea0e9 is loaded │ PASS   │        384ns │
│ netflow │             │ asset     │ visualization netflow-f772028b-d5a6-4d55-b441-493871981a60 is loaded │ PASS   │        390ns │
│ netflow │             │ asset     │ visualization netflow-f7808e70-df2a-4532-a350-966704567c24 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-f86a7769-8ef6-408d-bbe3-985d0ea0a3f7 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-fd6c1144-5026-4795-b7af-a9aa3fc28c56 is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ search netflow-a34c6611-79d8-4b50-ae3f-8b328d28e24a is loaded        │ PASS   │        513ns │
│ netflow │ log         │ asset     │ index_template logs-netflow.log is loaded                            │ PASS   │        496ns │
│ netflow │ log         │ asset     │ ingest_pipeline logs-netflow.log-0.3.6 is loaded                     │ PASS   │        419ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: netflow - END   ---
Done
Run pipeline tests for the package
--- Test results for package: netflow - START ---
No test results
--- Test results for package: netflow - END   ---

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #590 updated

• • Start Time: 2021-01-31T16:07:05.510+0000

• Duration: 8 min 47 sec

• Commit: 5f68f46

Test stats 🧪

Test	Results
Failed	0
Passed	90
Skipped	0
Total	90

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)