netflow,juniper_srx - Remove invalid and unused field defintions (#8011)

For netflow and juniper_srx, remove 'external: ecs' field definitions that are
invalid usages of ECS fields. These fields were unused in the integrations.

Relates #7808

packages/juniper_srx/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.16.2"
+  changes:
+    - description: Removing additional unused ECS field declarations.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/8011
 - version: "1.16.1"
   changes:
     - description: Removing unused ECS field declarations.

packages/juniper_srx/data_stream/log/fields/ecs.yml

@@ -12,8 +12,6 @@
   name: agent.type
 - external: ecs
   name: agent.version
-- external: ecs
-  name: as.organization.name
 - external: ecs
   name: client.address
 - external: ecs
@@ -78,16 +76,6 @@
   name: cloud.provider
 - external: ecs
   name: cloud.region
-- external: ecs
-  name: code_signature.exists
-- external: ecs
-  name: code_signature.status
-- external: ecs
-  name: code_signature.subject_name
-- external: ecs
-  name: code_signature.trusted
-- external: ecs
-  name: code_signature.valid
 - external: ecs
   name: container.image.tag
 - external: ecs
@@ -364,14 +352,6 @@
   name: group.id
 - external: ecs
   name: group.name
-- external: ecs
-  name: hash.md5
-- external: ecs
-  name: hash.sha1
-- external: ecs
-  name: hash.sha256
-- external: ecs
-  name: hash.sha512
 - external: ecs
   name: host.architecture
 - external: ecs
@@ -552,20 +532,6 @@
   name: package.type
 - external: ecs
   name: package.version
-- external: ecs
-  name: pe.architecture
-- external: ecs
-  name: pe.company
-- external: ecs
-  name: pe.description
-- external: ecs
-  name: pe.file_version
-- external: ecs
-  name: pe.imphash
-- external: ecs
-  name: pe.original_file_name
-- external: ecs
-  name: pe.product
 - external: ecs
   name: process.args
 - external: ecs

packages/juniper_srx/docs/README.md

@@ -50,8 +50,6 @@ The following processes and tags are supported:
 | agent.name | Custom name of the agent. This is a name that can be given to an agent. This can be helpful if for example two Filebeat instances are running on the same host but a human readable separation is needed on which Filebeat instance data is coming from. | keyword |
 | agent.type | Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine. | keyword |
 | agent.version | Version of the agent. | keyword |
-| as.organization.name | Organization name. | keyword |
-| as.organization.name.text | Multi-field of `as.organization.name`. | match_only_text |
 | client.address | Some event client addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket.  You should always store the raw address in the `.address` field. Then it should be duplicated to `.ip` or `.domain`, depending on which one it is. | keyword |
 | client.as.organization.name | Organization name. | keyword |
 | client.as.organization.name.text | Multi-field of `client.as.organization.name`. | match_only_text |
@@ -88,11 +86,6 @@ The following processes and tags are supported:
 | cloud.project.name | The cloud project name. Examples: Google Cloud Project name, Azure Project name. | keyword |
 | cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
 | cloud.region | Region in which this host, resource, or service is located. | keyword |
-| code_signature.exists | Boolean to capture if a signature is present. | boolean |
-| code_signature.status | Additional information about the certificate status. This is useful for logging cryptographic errors with the certificate validity or trust status. Leave unpopulated if the validity or trust of the certificate was unchecked. | keyword |
-| code_signature.subject_name | Subject name of the code signer | keyword |
-| code_signature.trusted | Stores the trust status of the certificate chain. Validating the trust of the certificate chain may be complicated, and this field should only be populated by tools that actively check the status. | boolean |
-| code_signature.valid | Boolean to capture if the digital signature is verified against the binary content. Leave unpopulated if a certificate was unchecked. | boolean |
 | container.id | Unique container id. | keyword |
 | container.image.name | Name of the image the container was built on. | keyword |
 | container.image.tag | Container image tags. | keyword |
@@ -246,10 +239,6 @@ The following processes and tags are supported:
 | group.domain | Name of the directory the group is a member of. For example, an LDAP or Active Directory domain name. | keyword |
 | group.id | Unique identifier for the group on the system/platform. | keyword |
 | group.name | Name of the group. | keyword |
-| hash.md5 | MD5 hash. | keyword |
-| hash.sha1 | SHA1 hash. | keyword |
-| hash.sha256 | SHA256 hash. | keyword |
-| hash.sha512 | SHA512 hash. | keyword |
 | host.architecture | Operating system architecture. | keyword |
 | host.containerized | If the host is a container. | boolean |
 | host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
@@ -502,13 +491,6 @@ The following processes and tags are supported:
 | package.size | Package size in bytes. | long |
 | package.type | Type of package. This should contain the package file type, rather than the package manager name. Examples: rpm, dpkg, brew, npm, gem, nupkg, jar. | keyword |
 | package.version | Package version | keyword |
-| pe.architecture | CPU architecture target for the file. | keyword |
-| pe.company | Internal company name of the file, provided at compile-time. | keyword |
-| pe.description | Internal description of the file, provided at compile-time. | keyword |
-| pe.file_version | Internal version of the file, provided at compile-time. | keyword |
-| pe.imphash | A hash of the imports in a PE file. An imphash -- or import hash -- can be used to fingerprint binaries even after recompilation or other code-level transformations have occurred, which would change more traditional hash values. Learn more at https://www.fireeye.com/blog/threat-research/2014/01/tracking-malware-import-hashing.html. | keyword |
-| pe.original_file_name | Internal name of the file, provided at compile-time. | keyword |
-| pe.product | Internal product name of the file, provided at compile-time. | keyword |
 | process.args | Array of process arguments, starting with the absolute path to the executable. May be filtered to protect sensitive information. | keyword |
 | process.args_count | Length of the process.args array. This field can be useful for querying or performing bucket analysis on how many arguments were provided to start a process. More arguments may be an indication of suspicious activity. | long |
 | process.code_signature.exists | Boolean to capture if a signature is present. | boolean |

packages/juniper_srx/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 2.11.0
 name: juniper_srx
 title: Juniper SRX
-version: "1.16.1"
+version: "1.16.2"
 description: Collect logs from Juniper SRX devices with Elastic Agent.
 categories: ["network", "security", "firewall_security"]
 type: integration

packages/netflow/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.15.2"
+  changes:
+    - description: Removing additional unused ECS field declarations.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/8011
 - version: "2.15.1"
   changes:
     - description: Removing unused ECS field declarations.

packages/netflow/data_stream/log/fields/ecs.yml

@@ -10,8 +10,6 @@
   name: agent.type
 - external: ecs
   name: agent.version
-- external: ecs
-  name: as.organization.name
 - external: ecs
   name: client.address
 - external: ecs
@@ -280,36 +278,12 @@
   name: file.type
 - external: ecs
   name: file.uid
-- external: ecs
-  name: geo.city_name
-- external: ecs
-  name: geo.continent_name
-- external: ecs
-  name: geo.country_iso_code
-- external: ecs
-  name: geo.country_name
-- external: ecs
-  name: geo.location
-- external: ecs
-  name: geo.name
-- external: ecs
-  name: geo.region_iso_code
-- external: ecs
-  name: geo.region_name
 - external: ecs
   name: group.domain
 - external: ecs
   name: group.id
 - external: ecs
   name: group.name
-- external: ecs
-  name: hash.md5
-- external: ecs
-  name: hash.sha1
-- external: ecs
-  name: hash.sha256
-- external: ecs
-  name: hash.sha512
 - external: ecs
   name: host.architecture
 - external: ecs
@@ -464,18 +438,6 @@
   name: organization.id
 - external: ecs
   name: organization.name
-- external: ecs
-  name: os.family
-- external: ecs
-  name: os.full
-- external: ecs
-  name: os.kernel
-- external: ecs
-  name: os.name
-- external: ecs
-  name: os.platform
-- external: ecs
-  name: os.version
 - external: ecs
   name: package.architecture
 - external: ecs

packages/netflow/docs/README.md

@@ -30,8 +30,6 @@ The `log` dataset collects netflow logs.
 | agent.name | Custom name of the agent. This is a name that can be given to an agent. This can be helpful if for example two Filebeat instances are running on the same host but a human readable separation is needed on which Filebeat instance data is coming from. | keyword |
 | agent.type | Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine. | keyword |
 | agent.version | Version of the agent. | keyword |
-| as.organization.name | Organization name. | keyword |
-| as.organization.name.text | Multi-field of `as.organization.name`. | match_only_text |
 | client.address | Some event client addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket.  You should always store the raw address in the `.address` field. Then it should be duplicated to `.ip` or `.domain`, depending on which one it is. | keyword |
 | client.as.organization.name | Organization name. | keyword |
 | client.as.organization.name.text | Multi-field of `client.as.organization.name`. | match_only_text |
@@ -189,21 +187,9 @@ The `log` dataset collects netflow logs.
 | file.uid | The user ID (UID) or security identifier (SID) of the file owner. | keyword |
 | flow.id | Hash of source and destination IPs. | keyword |
 | flow.locality | Identifies whether the flow involved public IP addresses or only private address. | keyword |
-| geo.city_name | City name. | keyword |
-| geo.continent_name | Name of the continent. | keyword |
-| geo.country_iso_code | Country ISO code. | keyword |
-| geo.country_name | Country name. | keyword |
-| geo.location | Longitude and latitude. | geo_point |
-| geo.name | User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation. | keyword |
-| geo.region_iso_code | Region ISO code. | keyword |
-| geo.region_name | Region name. | keyword |
 | group.domain | Name of the directory the group is a member of. For example, an LDAP or Active Directory domain name. | keyword |
 | group.id | Unique identifier for the group on the system/platform. | keyword |
 | group.name | Name of the group. | keyword |
-| hash.md5 | MD5 hash. | keyword |
-| hash.sha1 | SHA1 hash. | keyword |
-| hash.sha256 | SHA256 hash. | keyword |
-| hash.sha512 | SHA512 hash. | keyword |
 | host.architecture | Operating system architecture. | keyword |
 | host.containerized | If the host is a container. | boolean |
 | host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
@@ -1619,14 +1605,6 @@ The `log` dataset collects netflow logs.
 | organization.id | Unique identifier for the organization. | keyword |
 | organization.name | Organization name. | keyword |
 | organization.name.text | Multi-field of `organization.name`. | match_only_text |
-| os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
-| os.full | Operating system name, including the version or code name. | keyword |
-| os.full.text | Multi-field of `os.full`. | match_only_text |
-| os.kernel | Operating system kernel version as a raw string. | keyword |
-| os.name | Operating system name, without the version. | keyword |
-| os.name.text | Multi-field of `os.name`. | match_only_text |
-| os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |
-| os.version | Operating system version as a raw string. | keyword |
 | package.architecture | Package architecture. | keyword |
 | package.checksum | Checksum of the installed package for verification. | keyword |
 | package.description | Description of the package. | keyword |

packages/netflow/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 2.11.0
 name: netflow
 title: NetFlow Records
-version: "2.15.1"
+version: "2.15.2"
 description: Collect flow records from NetFlow and IPFIX exporters with Elastic Agent.
 type: integration
 categories: