Skip to content

Commit

Permalink
graph statistics format (#36)
Browse files Browse the repository at this point in the history
  • Loading branch information
s0rg authored Sep 6, 2023
1 parent 3c8cb76 commit c811939
Show file tree
Hide file tree
Showing 4 changed files with 173 additions and 67 deletions.
134 changes: 68 additions & 66 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,74 @@ Takes all network connections from your docker containers and exports them as:
- [structurizr dsl](https://github.com/structurizr/dsl)
- [compose yaml](https://github.com/compose-spec/compose-spec/blob/master/spec.md)
- pseudographical tree
- json stream of items:
- json stream

# features

- os-independent, it uses different strategies to get container connections:
- running on **linux as root** is the fastest way and it will work with all types of containers (even
`scratch`-based)
- running as non-root or on non-linux OS will attempt to run `netsat` inside container, if this fails
(i.e. for missing `netstat` binary), no connections for such container will be gathered
- produces detailed connections graph **with ports**
- save `json` stream once and process it later in any way you want
- fast, scans ~400 containers in around 5 sec
- 100% test-coverage

# known limitations

- only established and listen connections are listed (but script like [snapshots.sh](examples/snapshots.sh) can beat this)
- `composer-yaml` is not intended to be working out from the box, it can lack some of crucial information (even in `-full` mode),
or may contains cycles between nodes (removing `links` section in services may help), its main purpose is for system overview

# installation

- [binaries / deb / rpm](https://github.com/s0rg/decompose/releases) for Linux, FreeBSD, macOS and Windows.

# usage

```
decompose [flags]
possible flags with default values:
-cluster string
json file with clusterization rules
-follow string
follow only this container by name
-format string
output format: dot, json, yaml, tree or sdsl for structurizr dsl (default "dot")
-full
extract full process info: (cmd, args, env) and volumes info
-help
show this help
-load value
load json stream, can be used multiple times
-local
skip external hosts
-meta string
json file with metadata for enrichment
-no-loops
remove connection loops (node to itself) from output
-out string
output: filename or "-" for stdout (default "-")
-proto string
protocol to scan: tcp, udp or all (default "all")
-silent
suppress progress messages in stderr
-skip-env string
environment variables name(s) to skip from output, case-independent, comma-separated
-version
show version
```

## environment variables:

- `DOCKER_HOST` - connection uri
- `DOCKER_CERT_PATH` - directory path containing key.pem, cert.pm and ca.pem
- `DOCKER_TLS_VERIFY` - enable client TLS verification

# json stream format

```go
type Item struct {
Expand Down Expand Up @@ -145,71 +212,6 @@ type Node struct {

See: [cluster.json](examples/cluster.json) for detailed example.

# features

- os-independent, it uses different strategies to get container connections:
- running on **linux as root** is the fastest way and it will work with all types of containers (even
`scratch`-based)
- running as non-root or on non-linux OS will attempt to run `netsat` inside container, if this fails
(i.e. for missing `netstat` binary), no connections for such container will be gathered
- produces detailed connections graph **with ports**
- save `json` stream once and process it later in any way you want
- fast, scans ~400 containers in around 5 sec
- 100% test-coverage

# known limitations

- only established and listen connections are listed (but script like [snapshots.sh](examples/snapshots.sh) can beat this)
- `composer-yaml` is not intended to be working out from the box, it can lack some of crucial information (even in `-full` mode),
or may contains cycles between nodes (removing `links` section in services may help), its main purpose is for system overview

# installation

- [binaries / deb / rpm](https://github.com/s0rg/decompose/releases) for Linux, FreeBSD, macOS and Windows.

# usage

```
decompose [flags]
possible flags with default values:
-cluster string
json file with clusterization rules
-follow string
follow only this container by name
-format string
output format: dot, json, yaml, tree or sdsl for structurizr dsl (default "dot")
-full
extract full process info: (cmd, args, env) and volumes info
-help
show this help
-load value
load json stream, can be used multiple times
-local
skip external hosts
-meta string
json file with metadata for enrichment
-no-loops
remove connection loops (node to itself) from output
-out string
output: filename or "-" for stdout (default "-")
-proto string
protocol to scan: tcp, udp or all (default "all")
-silent
suppress progress messages in stderr
-skip-env string
environment variables name(s) to skip from output, case-independent, comma-separated
-version
show version
```

## environment variables:

- `DOCKER_HOST` - connection uri
- `DOCKER_CERT_PATH` - directory path containing key.pem, cert.pm and ca.pem
- `DOCKER_TLS_VERIFY` - enable client TLS verification

# examples

Get `dot` file:
Expand Down
2 changes: 1 addition & 1 deletion cmd/decompose/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ func setupFlags() {
&fFormat,
"format",
defaultFormat,
"output format: dot, json, yaml, tree or sdsl for structurizr dsl",
"output format: dot, json, yaml, stat, tree or sdsl for structurizr dsl",
)
flag.StringVar(
&fSkipEnv,
Expand Down
4 changes: 4 additions & 0 deletions internal/builder/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ const (
kindJSON = "json"
kindTREE = "tree"
kindYAML = "yaml"
kindSTAT = "stat"
kindStructurizr = "sdsl"
)

Expand All @@ -26,6 +27,8 @@ func Create(kind string) (b graph.NamedBuilderWriter, ok bool) {
return NewTree(), true
case kindYAML:
return NewYAML(), true
case kindSTAT:
return NewStat(), true
}

return
Expand All @@ -41,6 +44,7 @@ func Names() (rv []string) {
kindJSON,
kindTREE,
kindYAML,
kindSTAT,
kindStructurizr,
}
}
100 changes: 100 additions & 0 deletions internal/builder/stat.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
//go:build !test

package builder

import (
"cmp"
"fmt"
"io"
"slices"

"github.com/s0rg/set"

"github.com/s0rg/decompose/internal/node"
)

type portStat struct {
Port string
Count int
}

type Stat struct {
conns map[string]set.Unordered[string]
ports map[string]int
nodes int
edges int
exts int
}

func NewStat() *Stat {
return &Stat{
ports: make(map[string]int),
conns: make(map[string]set.Unordered[string]),
}
}

func (s *Stat) Name() string {
return "graph-stats"
}

func (s *Stat) AddNode(n *node.Node) error {
if n.IsExternal() {
s.exts++

return nil
}

s.nodes++

for _, p := range n.Ports {
s.ports[p.Label()]++
}

s.conns[n.ID] = make(set.Unordered[string])

return nil
}

func (s *Stat) AddEdge(src, dst string, _ *node.Port) {
if _, ok := s.conns[dst]; !ok {
return
}

conn, ok := s.conns[src]
if !ok {
return
}

if conn.Has(dst) {
return
}

s.edges++
}

func (s *Stat) Write(w io.Writer) {
ports := make([]*portStat, 0, len(s.ports))

for k, c := range s.ports {
ports = append(ports, &portStat{Port: k, Count: c})
}

slices.SortStableFunc(ports, func(a, b *portStat) int {
return cmp.Compare(a.Count, b.Count)
})

slices.Reverse(ports)

fmt.Fprintln(w, "Total:")
fmt.Fprintf(w, "\tNodes:\t%d\n", s.nodes)
fmt.Fprintf(w, "\tEdges:\t%d\n", s.edges)
fmt.Fprintf(w, "\tExternals:\t%d\n", s.exts)

fmt.Fprintln(w, "")

fmt.Fprintln(w, "Ports:")

for _, p := range ports {
fmt.Fprintf(w, "\t%s:\t%d\n", p.Port, p.Count)
}
}

0 comments on commit c811939

Please sign in to comment.