Added dns.question.top_level_domain and dns.question.subdomain ecs field

[mbudge]

Added fields recently added to the elastic common schema to packeteat dns output.

Added the following fields to packetbeat

• dns.question.subdomain
• dns.question.top_level_domain

I tried adding unit tests for these fields but you might want to check that.

I also can't find the fields.yml file to define the exported packetbeat fields in the documentation.

Also not sure how to update the change log to document updates to packetbeat.

Here is an example of the code to get the top_level_domain and the subdomain.

`

                if eTLDPlusOne != "" && strings.Contains(eTLDPlusOne, ".") {
			topLevelDomain := strings.Join(strings.Split(eTLDPlusOne, ".")[1:], ".")
			qMapStr["top_level_domain"] = topLevelDomain

			subdomain := strings.Trim(strings.TrimRight(q.Name, eTLDPlusOne), ".")
			if subdomain != "" {
				qMapStr["subdomain"] = subdomain
			}
		}

`

The top_level_domain field removes the PlusOne to get the TLD (org, net, com etc)

The subdomain field is the full question name, minus the eTLDPlusOne/Registered_Domain with dots trimmed. If the value isn't an empty string then a subdomain is present.

Elastic common schema fields

Add dns.question.subdomain field #574
elastic/ecs#574

Added top_level_domain #572
elastic/ecs#572

Both the top_level_domain and subdomain are useful for security analytics (discussed in the linked ECS pull requests in more detail).

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[mbudge]

How do I fix this?

`
/go/src/beats/packetbeat$ mage fmt update
panic: failed determine libbeat dir location: failed to find github.com/elastic/beats/dev-tools/mage in the project's vendor

goroutine 1 [running]:
github.com/elastic/beats/dev-tools/mage.LibbeatDir(0xc0000b5e30, 0x1, 0x1, 0xc0000cad80, 0x34)
...../go/src/github.com/elastic/beats/dev-tools/mage/common.go:729 +0x1d7
github.com/elastic/beats/dev-tools/mage.init.ializers()
...../go/src/github.com/elastic/beats/dev-tools/mage/config.go:45 +0x443

`

[webmat]

jenkins, test this

[webmat]

Thanks @mbudge!

@andrewkroh or @adriansr can you guys take a look?

[mbudge]

Please can someone help with this?

Thanks

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[adriansr]

Thanks for your contribution @mbudge !

I've pushed a commit with some test cases. Now it just needs a changelog entry, I'll leave that to you.

The file to edit is CHANGELOG.next.asciidoc, you need to a add a new line under Added -> Packetbeat. See 831c544 for an example.

[mbudge]

Updated the change log.

[mbudge]

Found a bug in the subdomain extract

https://play.golang.org/p/xydzQ0thVBg

fmt.Println(strings.TrimRight("news.joules.com", "joules.com"))

[mbudge]

I found the code didn't handle domains like gov.ky because it threw an error.

The code only runs if the error contains "cannot derive eTLD+1 for domain", 1 dot and the top_level_domain string to the right of the dot isn't empty.

               else if strings.Contains(err.Error(), "cannot derive eTLD+1 for domain") && strings.Count(q.Name, ".") == 1 {
		// Handle publicsuffix.EffectiveTLDPlusOne eTLD+1 error with 1 dot in the domain.
		s := strings.Split(eTLDPlusOne, ".")
		if len(s) == 2 && s[1] != "" {
			qMapStr["top_level_domain"] = s[1]
		}
		qMapStr["registered_domain"] = q.Name
	}

Here's an example in Go playground https://play.golang.org/p/Kn3PQ70OGp0

[andrewkroh]

Here's my take on some utility methods for getting at the tld, etld, registered domain, and subdomain. https://play.golang.org/p/x0sUqRePN7-

One difference from what this does today is that no longer considers private suffixes in the PSL when getting the ETLD.

[andrewkroh]

Splitting and then re-joining seems inefficient. I think it could slice up the eTLDPlusOne string by finding the index of the first dot.

[andrewkroh]

Matching error strings is really error prone because the error can change upstream without notice. Is there a way you can do this without matching the error string?

[mbudge]

I've removed the string.contains eTLD+1 error as it doesn't matter what the error is. Also replaced strings.split with strings.indexbyte.

The helper functions would be good in a shared library so they can be accessed by all the beats agents. I've added the top_level_domain and registered_domain to all the ECS domain fields. It's only sub-domain which is dns only in ECS.

[andrewkroh]

jenkins, test this