Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/dockerstatsreceiver] Initial PR to onboard dockerstats onto mdatagen #12322

Merged
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
2bfdcfc
initial setup for dockerstats mdatagen, with just memory metrics
jamesmoessis Jun 11, 2022
dbea8b9
add mdatagen metrics config option
jamesmoessis Jun 28, 2022
96a0518
further updates to metadata.yaml
jamesmoessis Jun 28, 2022
86a2c6c
initial alternate implementation of scrape()
jamesmoessis Jun 28, 2022
ba3a74a
rearrange scraper so follows a stronger pattern and is more easily te…
jamesmoessis Jun 29, 2022
b474d88
testing
jamesmoessis Jun 30, 2022
ddd4c43
expected default metrics json
jamesmoessis Jun 30, 2022
3ada5d6
add option to add more docker client opts
jamesmoessis Jun 30, 2022
e4406b4
work around nasty go-connections bug
jamesmoessis Jun 30, 2022
c9c00d6
remove unneeded
jamesmoessis Jul 6, 2022
c1cbbc2
create test for exist scraper while mocking docker engine
jamesmoessis Jul 6, 2022
191b48b
partway through onboarding more metrics to mdatagen
jamesmoessis Jul 8, 2022
f81c212
add all remaining metrics to metadata.yaml and generate
jamesmoessis Jul 11, 2022
b4a2538
more detailed memory metrics
jamesmoessis Jul 11, 2022
da4b203
record more detailed mem metrics
jamesmoessis Jul 11, 2022
178b39e
Merge branch 'main' into mdatagen-dockerstatsreceiver
jamesmoessis Jul 11, 2022
3d7eb46
more mock testdata for docker engine
jamesmoessis Jul 12, 2022
c646c0f
final touch ups of metadata.yaml
jamesmoessis Jul 12, 2022
121fa2d
tests for two containers in one engine for both scrape impls
jamesmoessis Jul 12, 2022
a800ec5
fix bug where container metrics were overridden but subsequent resour…
jamesmoessis Jul 12, 2022
e0be23c
fix search and replace error
jamesmoessis Jul 12, 2022
8ff4540
implement moviestoreguy suggestions log + remove copy
jamesmoessis Jul 13, 2022
1955866
implement moviestoreguy suggestion; move new implementation to new file
jamesmoessis Jul 13, 2022
a12f0e7
add const for default resource len
jamesmoessis Jul 13, 2022
8cfeb4f
appease linter
jamesmoessis Jul 13, 2022
a1ec764
Merge branch 'main' into mdatagen-dockerstatsreceiver
jamesmoessis Jul 13, 2022
5a9dd28
changelog entry
jamesmoessis Jul 13, 2022
c36a27f
make goporto
jamesmoessis Jul 13, 2022
612b144
sort json keys
jamesmoessis Jul 14, 2022
7ac8631
override percoreCPU metrics flag with existing flag
jamesmoessis Jul 14, 2022
7d34f6d
only override flag if it's true, otherwise leave as is
jamesmoessis Jul 14, 2022
3ab8923
remove unnecessary
jamesmoessis Jul 14, 2022
771456b
only neutralise scope name, version, and metric description if not te…
jamesmoessis Jul 14, 2022
9aa9a96
enable per-cpu metric validation
jamesmoessis Jul 14, 2022
6547cf0
remove unneeded
jamesmoessis Jul 14, 2022
2709245
pin go-connections to 0.4.0 and remove dialer override in client
jamesmoessis Jul 14, 2022
c51d3be
Merge branch 'main' into mdatagen-dockerstatsreceiver
jamesmoessis Jul 15, 2022
a8f1cb0
go mod tidy
jamesmoessis Jul 15, 2022
d7f4afd
exclude go-connections v0.4.1 in main go.mod
jamesmoessis Jul 18, 2022
3750d20
replace instead of exclude
jamesmoessis Jul 19, 2022
a4aac5b
give more detail in device major/minor number attrs
jamesmoessis Jul 21, 2022
c396e60
rearrange doc.go as per review
jamesmoessis Jul 21, 2022
fb33b3b
Merge branch 'main' into mdatagen-dockerstatsreceiver
jamesmoessis Jul 21, 2022
e6d74f2
go generate
jamesmoessis Jul 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -923,6 +923,9 @@ replace github.com/googleapis/gnostic v0.5.6 => github.com/googleapis/gnostic v0
// see https://github.com/DataDog/dd-trace-go/issues/1220
replace github.com/go-chi/chi/v4 => github.com/go-chi/chi v4.0.0+incompatible

// see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/12322#issuecomment-1185029670
replace github.com/docker/go-connections v0.4.1-0.20210727194412-58542c764a11 => github.com/docker/go-connections v0.4.0

// see https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/4433
exclude github.com/StackExchange/wmi v1.2.0

Expand Down
3 changes: 1 addition & 2 deletions go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions internal/docker/docker.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,13 @@ type Client struct {
logger *zap.Logger
}

func NewDockerClient(config *Config, logger *zap.Logger) (*Client, error) {
func NewDockerClient(config *Config, logger *zap.Logger, opts ...docker.Opt) (*Client, error) {
jamesmoessis marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have reverted the go-connections version, does this still need to happen?
Or is this just a safe nice to have?

Copy link
Contributor Author

@jamesmoessis jamesmoessis Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it doesn't need to be there anymore, but imo it doesn't hurt to leave it in, and it could be nice to have for future purposes or similar issues. I don't feel strongly either way

client, err := docker.NewClientWithOpts(
docker.WithHost(config.Endpoint),
docker.WithVersion(fmt.Sprintf("v%v", config.DockerAPIVersion)),
docker.WithHTTPHeaders(map[string]string{"User-Agent": userAgent}),
append([]docker.Opt{
docker.WithHost(config.Endpoint),
docker.WithVersion(fmt.Sprintf("v%v", config.DockerAPIVersion)),
docker.WithHTTPHeaders(map[string]string{"User-Agent": userAgent}),
}, opts...)...,
)
if err != nil {
return nil, fmt.Errorf("could not create docker client: %w", err)
Expand Down
5 changes: 5 additions & 0 deletions receiver/dockerstatsreceiver/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import (

"go.opentelemetry.io/collector/config"
"go.opentelemetry.io/collector/receiver/scraperhelper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver/internal/metadata"
)

var _ config.Receiver = (*Config)(nil)
Expand Down Expand Up @@ -56,6 +58,9 @@ type Config struct {

// Docker client API version. Default is 1.22
DockerAPIVersion float64 `mapstructure:"api_version"`

// Metrics config. Enable or disable stats by name.
MetricsConfig metadata.MetricsSettings `mapstructure:"stats"`
}

func (config Config) Validate() error {
Expand Down
17 changes: 17 additions & 0 deletions receiver/dockerstatsreceiver/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package dockerstatsreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver"

//go:generate mdatagen --experimental-gen metadata.yaml
jamesmoessis marked this conversation as resolved.
Show resolved Hide resolved
141 changes: 141 additions & 0 deletions receiver/dockerstatsreceiver/documentation.md

Large diffs are not rendered by default.

10 changes: 7 additions & 3 deletions receiver/dockerstatsreceiver/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import (
"go.opentelemetry.io/collector/config"
"go.opentelemetry.io/collector/consumer"
"go.opentelemetry.io/collector/receiver/scraperhelper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver/internal/metadata"
)

const (
Expand All @@ -44,21 +46,23 @@ func createDefaultConfig() config.Receiver {
Endpoint: "unix:///var/run/docker.sock",
Timeout: 5 * time.Second,
DockerAPIVersion: defaultDockerAPIVersion,
MetricsConfig: metadata.DefaultMetricsSettings(),
}
}

func createMetricsReceiver(
ctx context.Context,
_ context.Context,
params component.ReceiverCreateSettings,
config config.Receiver,
consumer consumer.Metrics,
) (component.MetricsReceiver, error) {
dockerConfig := config.(*Config)
dsr := newReceiver(params, dockerConfig)

dsr, err := NewReceiver(ctx, params, dockerConfig, consumer)
scrp, err := scraperhelper.NewScraper(typeStr, dsr.scrape, scraperhelper.WithStart(dsr.start))
if err != nil {
return nil, err
}

return dsr, nil
return scraperhelper.NewScraperControllerReceiver(&dsr.config.ScraperControllerSettings, params, consumer, scraperhelper.AddScraper(scrp))
}
8 changes: 6 additions & 2 deletions receiver/dockerstatsreceiver/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
github.com/docker/docker v20.10.17+incompatible
github.com/open-telemetry/opentelemetry-collector-contrib/internal/containertest v0.55.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker v0.55.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/scrapertest v0.55.0
github.com/stretchr/testify v1.8.0
go.opentelemetry.io/collector v0.55.1-0.20220711160057-6133c820fd50
go.opentelemetry.io/collector/pdata v0.55.1-0.20220711160057-6133c820fd50
Expand All @@ -21,15 +22,15 @@ require (
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/docker/distribution v2.8.0-beta.1+incompatible // indirect
github.com/docker/go-connections v0.4.1-0.20210727194412-58542c764a11 // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/knadh/koanf v1.4.2 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
Expand All @@ -40,6 +41,7 @@ require (
github.com/pelletier/go-toml v1.9.4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rogpeppe/go-internal v1.6.1 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
go.opencensus.io v0.23.0 // indirect
go.opentelemetry.io/otel v1.8.0 // indirect
Expand All @@ -60,5 +62,7 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/internal/conta

replace github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker => ../../internal/docker

replace github.com/open-telemetry/opentelemetry-collector-contrib/internal/scrapertest => ../../internal/scrapertest

// see https://github.com/distribution/distribution/issues/3590
exclude github.com/docker/distribution v2.8.0+incompatible
11 changes: 2 additions & 9 deletions receiver/dockerstatsreceiver/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading