Allow probes to attach metadata directly to entities

[peterbourgon]

Currently the probes produce reports that look like

type Report struct {
    Process Topology
    Network Topology
    HostMetadatas
}

We will change them to look like

type Report struct {
    Endpoint  Topology // former Process
    Process   Topology // only nodes, no edges
    Container Topology // only nodes, no edges
    Address   Topology // former Network
    Host      Topology // former HostMetadatas; only nodes, no edges
}

Probes will attach metadata to the actual entities they scrape, rather than having to explode the metadata into the topologies that happen to be present. This will also make merging a little easier, as we're only dealing with a single type. (via #115)

As part of this work, we'll also add more process and container metadata. (via #114)

[tomwilkie]

Could you elaborate on the key formats inside these topologies please?

And I guess endpoints will need enough metadata on the nodes to link to processes and containers.

Whats in Address thats not in endpoint? Ie could you just put host names as metadata on nodes in endpoint, and then reduce that to get Address?

[peterbourgon]

Could you elaborate on the key formats inside these topologies please?

Sure, I'll do this properly in the PR. I need to revisit it to get it straight in my head, to be honest :)

And I guess endpoints will need enough metadata on the nodes to link to processes and containers.

Yes, we'll have foreign key relationships across topologies where we can.

Whats in Address thats not in endpoint?

Address is an IP address i.e. L3. Endpoint is the complete TCP n-tuple i.e. L4. Crucially, there is information we can get and associate with lower levels that won't translate to higher levels. Think ICMP: present in Address, invisible in Endpoint.

Ie could you just put host names as metadata on nodes in endpoint, and then reduce that to get Address?

Yes, but the point of each topology is that it represents a different source of information, which doesn't map perfectly across OSI layer boundaries.

[tomwilkie]

Yeah that makes sense. Am I right in thinking that address current has the
same info as endpoint (minus the port) ie they use the same source?

On Wednesday, 27 May 2015, Peter Bourgon notifications@github.com wrote:

Could you elaborate on the key formats inside these topologies please?

Sure, I'll do this properly in the PR. I need to revisit it to get it
straight in my head, to be honest :)

And I guess endpoints will need enough metadata on the nodes to link to
processes and containers.

Yes, we'll have foreign key relationships across topologies where we can.

Whats in Address thats not in endpoint?

Address is an IP address i.e. L3. Endpoint is the complete TCP n-tuple
i.e. L4. Crucially, there is information we can get and associate with
lower levels that won't translate to higher levels. Think ICMP: present in
Address, invisible in Endpoint.

Ie could you just put host names as metadata on nodes in endpoint, and
then reduce that to get Address?

Yes, but the point of each topology is that it represents a different
source of information, which doesn't map perfectly across OSI layer
boundaries.

—
Reply to this email directly or view it on GitHub
#120 (comment).

[peterbourgon]

Am I right in thinking that address current has the same info as endpoint (minus the port) ie they use the same source?

Yes, as an implementation detail, both Network and Process topologies are currently populated by procspy.Connections. I want to change that ASAP :)

[tomwilkie]

Sounds to me like this can be closed? We've got the extra topologies and the beginning of merging them...

[peterbourgon]

Agreed.