Skip to content

Commit

Permalink
Merge branch 'main' into SUMO-202643-import-default-metrics-source
Browse files Browse the repository at this point in the history
  • Loading branch information
Przemek Delewski committed Oct 5, 2022
2 parents 9cd3254 + 968670a commit d01ff12
Show file tree
Hide file tree
Showing 15 changed files with 645 additions and 13 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ Sumo Logic Helm Chart Version

| version | status |
|-----------------------------------------------------------------------------------------------------------|-----------------------------------------|
| [v2.16](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.15/deploy/README.md) | current / supported |
| [v2.17](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.17/deploy/README.md) | current / supported |
| [v2.16](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.16/deploy/README.md) | deprecated / supported until 2023-03-15 |
| [v2.15](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.15/deploy/README.md) | deprecated / supported until 2023-03-13 |
| [v2.14](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.14/deploy/README.md) | deprecated / supported until 2023-03-02 |
| [v2.13](https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/release-v2.13/deploy/README.md) | deprecated / supported until 2023-01-29 |
Expand Down
5 changes: 5 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ for details on our Kubernetes Solution.
- [Minimum Requirements](#minimum-requirements)
- [Support Matrix](#support-matrix)
- [ARM support](#arm-support)
- [Falco support](#falco-support)

Documentation for other versions can be found in the [main README file](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/main/README.md#documentation).

Expand Down Expand Up @@ -133,3 +134,7 @@ The only exception to the above is Falco, which currently lacks official ARM Doc

[falco]: https://github.com/falcosecurity/falco/issues/1589
[issues]: https://github.com/SumoLogic/sumologic-kubernetes-collection/issues

### Falco support

Falco is embedded in this Helm Chart for user convenience only - Sumo Logic does not provide production support for it.
18 changes: 18 additions & 0 deletions deploy/docs/Best_Practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
- [Using NodeSelectors](#using-nodeselectors)
- [Binding pods to linux nodes](#binding-pods-to-linux-nodes)
- [Disable Thanos](#disable-thanos)
- [Parsing log content as json](#parsing-log-content-as-json)

## Overriding chart resource names with `fullnameOverride`

Expand Down Expand Up @@ -1448,3 +1449,20 @@ kube-prometheus-stack:
prometheusSpec:
thanos: null
```

## Parsing log content as json

In order to parse and store log content as json following configuration has to be applied:

```yaml
fluentd:
logs:
containers:
extraOutputPluginConf: |-
<filter **>
@type record_modifier
<record>
_sumo_metadata ${record["_sumo_metadata"][:log_format] = 'json_merge'; record["_sumo_metadata"]}
</record>
</filter>
```
5 changes: 3 additions & 2 deletions deploy/helm/sumologic/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ sumologic:
replicaCount: 3
image:
repository: public.ecr.aws/sumologic/nginx-unprivileged
tag: 1.21-alpine
tag: 1.23-alpine
pullPolicy: IfNotPresent
resources:
limits:
Expand Down Expand Up @@ -5024,7 +5024,8 @@ telegraf-operator:
metric_version = 2
# imagePullSecrets: []

## Configure falco
## Configure Falco
## Please note that Falco is embedded in this Helm Chart for user convenience only - Sumo Logic does not provide production support for it
## This is an experimental configuration and shouldn't be used in production environment
## https://github.com/falcosecurity/charts/tree/master/falco
falco:
Expand Down
2 changes: 1 addition & 1 deletion tests/helm/remote_write_proxy/static/basic.output.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ spec:
{}
containers:
- name: nginx
image: public.ecr.aws/sumologic/nginx-unprivileged:1.21-alpine
image: public.ecr.aws/sumologic/nginx-unprivileged:1.23-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
Expand Down
6 changes: 3 additions & 3 deletions tests/integration/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ go 1.18
require (
github.com/gruntwork-io/terratest v0.40.22
github.com/stretchr/testify v1.8.0
k8s.io/api v0.25.1
k8s.io/apimachinery v0.25.1
k8s.io/api v0.25.2
k8s.io/apimachinery v0.25.2
k8s.io/klog/v2 v2.80.1
sigs.k8s.io/e2e-framework v0.0.7
)
Expand Down Expand Up @@ -83,7 +83,7 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/client-go v0.25.1
k8s.io/client-go v0.25.2
k8s.io/kube-openapi v0.0.0-20220803162953-67bda5d908f1 // indirect
k8s.io/utils v0.0.0-20220728103510-ee6ede2d64ed // indirect
sigs.k8s.io/controller-runtime v0.11.2 // indirect
Expand Down
12 changes: 6 additions & 6 deletions tests/integration/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1023,17 +1023,17 @@ honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
k8s.io/api v0.24.1/go.mod h1:JhoOvNiLXKTPQ60zh2g0ewpA+bnEYf5q44Flhquh4vQ=
k8s.io/api v0.25.1 h1:yL7du50yc93k17nH/Xe9jujAYrcDkI/i5DL1jPz4E3M=
k8s.io/api v0.25.1/go.mod h1:hh4itDvrWSJsmeUc28rIFNri8MatNAAxJjKcQmhX6TU=
k8s.io/api v0.25.2 h1:v6G8RyFcwf0HR5jQGIAYlvtRNrxMJQG1xJzaSeVnIS8=
k8s.io/api v0.25.2/go.mod h1:qP1Rn4sCVFwx/xIhe+we2cwBLTXNcheRyYXwajonhy0=
k8s.io/apiextensions-apiserver v0.24.1 h1:5yBh9+ueTq/kfnHQZa0MAo6uNcPrtxPMpNQgorBaKS0=
k8s.io/apiextensions-apiserver v0.24.1/go.mod h1:A6MHfaLDGfjOc/We2nM7uewD5Oa/FnEbZ6cD7g2ca4Q=
k8s.io/apimachinery v0.24.1/go.mod h1:82Bi4sCzVBdpYjyI4jY6aHX+YCUchUIrZrXKedjd2UM=
k8s.io/apimachinery v0.25.1 h1:t0XrnmCEHVgJlR2arwO8Awp9ylluDic706WePaYCBTI=
k8s.io/apimachinery v0.25.1/go.mod h1:hqqA1X0bsgsxI6dXsJ4HnNTBOmJNxyPp8dw3u2fSHwA=
k8s.io/apimachinery v0.25.2 h1:WbxfAjCx+AeN8Ilp9joWnyJ6xu9OMeS/fsfjK/5zaQs=
k8s.io/apimachinery v0.25.2/go.mod h1:hqqA1X0bsgsxI6dXsJ4HnNTBOmJNxyPp8dw3u2fSHwA=
k8s.io/apiserver v0.24.1/go.mod h1:dQWNMx15S8NqJMp0gpYfssyvhYnkilc1LpExd/dkLh0=
k8s.io/client-go v0.24.1/go.mod h1:f1kIDqcEYmwXS/vTbbhopMUbhKp2JhOeVTfxgaCIlF8=
k8s.io/client-go v0.25.1 h1:uFj4AJKtE1/ckcSKz8IhgAuZTdRXZDKev8g387ndD58=
k8s.io/client-go v0.25.1/go.mod h1:rdFWTLV/uj2C74zGbQzOsmXPUtMAjSf7ajil4iJUNKo=
k8s.io/client-go v0.25.2 h1:SUPp9p5CwM0yXGQrwYurw9LWz+YtMwhWd0GqOsSiefo=
k8s.io/client-go v0.25.2/go.mod h1:i7cNU7N+yGQmJkewcRD2+Vuj4iz7b30kI8OcL3horQ4=
k8s.io/code-generator v0.24.1/go.mod h1:dpVhs00hTuTdTY6jvVxvTFCk6gSMrtfRydbhZwHI15w=
k8s.io/component-base v0.24.1 h1:APv6W/YmfOWZfo+XJ1mZwep/f7g7Tpwvdbo9CQLDuts=
k8s.io/component-base v0.24.1/go.mod h1:DW5vQGYVCog8WYpNob3PMmmsY8A3L9QZNg4j/dV3s38=
Expand Down
220 changes: 220 additions & 0 deletions tests/integration/helm_otelcol_traces_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
package integration

import (
"context"
"fmt"
"testing"
"time"

"github.com/SumoLogic/sumologic-kubernetes-collection/tests/integration/internal"
"github.com/SumoLogic/sumologic-kubernetes-collection/tests/integration/internal/ctxopts"
"github.com/SumoLogic/sumologic-kubernetes-collection/tests/integration/internal/stepfuncs"
terrak8s "github.com/gruntwork-io/terratest/modules/k8s"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"sigs.k8s.io/e2e-framework/klient/k8s"
"sigs.k8s.io/e2e-framework/klient/k8s/resources"
"sigs.k8s.io/e2e-framework/klient/wait"
"sigs.k8s.io/e2e-framework/klient/wait/conditions"
"sigs.k8s.io/e2e-framework/pkg/envconf"
"sigs.k8s.io/e2e-framework/pkg/features"
)

func Test_Helm_Otelcol_Traces(t *testing.T) {
const (
tickDuration = 3 * time.Second
waitDuration = 3 * time.Minute
tracesPerExporter uint = 10 // number of traces generated per exporter
spansPerTrace uint = 5
)
featInstall := features.New("traces").
Assess("sumologic secret is created with endpoints",
func(ctx context.Context, t *testing.T, envConf *envconf.Config) context.Context {
terrak8s.WaitUntilSecretAvailable(t, ctxopts.KubectlOptions(ctx), "sumologic", 60, tickDuration)
secret := terrak8s.GetSecret(t, ctxopts.KubectlOptions(ctx), "sumologic")
require.Len(t, secret.Data, 2, "Secret has incorrect number of endpoints. There should be 2 endpoints.")
return ctx
}).
// TODO: Rewrite into similar step func as WaitUntilStatefulSetIsReady but for deployments
Assess("otelcol deployment is ready", func(ctx context.Context, t *testing.T, envConf *envconf.Config) context.Context {
res := envConf.Client().Resources(ctxopts.Namespace(ctx))
releaseName := ctxopts.HelmRelease(ctx)
labelSelector := fmt.Sprintf("app=%s-sumologic-otelcol", releaseName)
ds := appsv1.DeploymentList{}

require.NoError(t,
wait.For(
conditions.New(res).
ResourceListN(&ds, 1,
resources.WithLabelSelector(labelSelector),
),
wait.WithTimeout(waitDuration),
wait.WithInterval(tickDuration),
),
)
require.NoError(t,
wait.For(
conditions.New(res).
DeploymentConditionMatch(&ds.Items[0], appsv1.DeploymentAvailable, corev1.ConditionTrue),
wait.WithTimeout(waitDuration),
wait.WithInterval(tickDuration),
),
)
return ctx
}).
// TODO: Rewrite into similar step func as WaitUntilStatefulSetIsReady but for daemonsets
Assess("otelagent daemonset is ready", func(ctx context.Context, t *testing.T, envConf *envconf.Config) context.Context {
res := envConf.Client().Resources(ctxopts.Namespace(ctx))
nl := corev1.NodeList{}
if !assert.NoError(t, res.List(ctx, &nl)) {
return ctx
}

releaseName := ctxopts.HelmRelease(ctx)
labelSelector := fmt.Sprintf("app=%s-sumologic-otelagent", releaseName)
ds := appsv1.DaemonSetList{}

require.NoError(t,
wait.For(
conditions.New(res).
ResourceListN(&ds, 1,
resources.WithLabelSelector(labelSelector),
),
wait.WithTimeout(waitDuration),
wait.WithInterval(tickDuration),
),
)
require.NoError(t,
wait.For(
conditions.New(res).
ResourceMatch(&ds.Items[0], func(object k8s.Object) bool {
d := object.(*appsv1.DaemonSet)
return d.Status.NumberUnavailable == 0 &&
d.Status.NumberReady == int32(len(nl.Items))
}),
wait.WithTimeout(waitDuration),
wait.WithInterval(tickDuration),
),
)
return ctx
}).Feature()

featTraces := features.New("traces").
Setup(stepfuncs.GenerateTraces(
tracesPerExporter,
spansPerTrace,
internal.TracesGeneratorName,
internal.TracesGeneratorNamespace,
internal.TracesGeneratorImage,
)).
Assess("wait for otlp http traces", stepfuncs.WaitUntilExpectedTracesPresent(
tracesPerExporter,
spansPerTrace,
map[string]string{
"__name__": "root-span-otlpHttp",
"service.name": "customer-trace-test-service",
"_collector": "kubernetes",
"k8s.cluster.name": "kubernetes",
"k8s.container.name": internal.TracesGeneratorName,
"k8s.deployment.name": internal.TracesGeneratorName,
"k8s.namespace.name": internal.TracesGeneratorNamespace,
"k8s.pod.pod_name": internal.TracesGeneratorName,
"k8s.pod.label.app": internal.TracesGeneratorName,
// "_sourceCategory": "kubernetes/customer/trace/tester/customer/trace/tester",
"_sourceName": fmt.Sprintf("%s.%s.%s", internal.TracesGeneratorNamespace, internal.TracesGeneratorName, internal.TracesGeneratorName),
},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
internal.ReceiverMockServicePort,
waitDuration,
tickDuration,
)).
Assess("wait for otlp grpc traces", stepfuncs.WaitUntilExpectedTracesPresent(
tracesPerExporter,
spansPerTrace,
map[string]string{
"__name__": "root-span-otlpGrpc",
"service.name": "customer-trace-test-service",
"_collector": "kubernetes",
"k8s.cluster.name": "kubernetes",
"k8s.container.name": internal.TracesGeneratorName,
"k8s.deployment.name": internal.TracesGeneratorName,
"k8s.namespace.name": internal.TracesGeneratorNamespace,
"k8s.pod.pod_name": internal.TracesGeneratorName,
"k8s.pod.label.app": internal.TracesGeneratorName,
// "_sourceCategory": "kubernetes/customer/trace/tester/customer/trace/tester",
"_sourceName": fmt.Sprintf("%s.%s.%s", internal.TracesGeneratorNamespace, internal.TracesGeneratorName, internal.TracesGeneratorName),
},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
internal.ReceiverMockServicePort,
waitDuration,
tickDuration,
)).
Assess("wait for zipkin traces", stepfuncs.WaitUntilExpectedTracesPresent(
tracesPerExporter,
spansPerTrace,
map[string]string{
"__name__": "root-span-zipkin",
"service.name": "customer-trace-test-service",
"_collector": "kubernetes",
"k8s.cluster.name": "kubernetes",
"k8s.container.name": internal.TracesGeneratorName,
"k8s.deployment.name": internal.TracesGeneratorName,
"k8s.namespace.name": internal.TracesGeneratorNamespace,
"k8s.pod.pod_name": internal.TracesGeneratorName,
"k8s.pod.label.app": internal.TracesGeneratorName,
// "_sourceCategory": "kubernetes/customer/trace/tester/customer/trace/tester",
"_sourceName": fmt.Sprintf("%s.%s.%s", internal.TracesGeneratorNamespace, internal.TracesGeneratorName, internal.TracesGeneratorName),
},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
internal.ReceiverMockServicePort,
waitDuration,
tickDuration,
)).
Assess("wait for jaeger thrift http traces", stepfuncs.WaitUntilExpectedTracesPresent(
tracesPerExporter,
spansPerTrace,
map[string]string{
"__name__": "root-span-jaegerThriftHttp",
"service.name": "customer-trace-test-service",
"_collector": "kubernetes",
"k8s.cluster.name": "kubernetes",
"k8s.container.name": internal.TracesGeneratorName,
"k8s.deployment.name": internal.TracesGeneratorName,
"k8s.namespace.name": internal.TracesGeneratorNamespace,
"k8s.pod.pod_name": internal.TracesGeneratorName,
"k8s.pod.label.app": internal.TracesGeneratorName,
// "_sourceCategory": "kubernetes/customer/trace/tester/customer/trace/tester",
"_sourceName": fmt.Sprintf("%s.%s.%s", internal.TracesGeneratorNamespace, internal.TracesGeneratorName, internal.TracesGeneratorName),
"otel.library.name": "jaegerThriftHttp",
},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
internal.ReceiverMockServicePort,
waitDuration,
tickDuration,
)).
Assess("wait for all spans", stepfuncs.WaitUntilExpectedSpansPresent(
4*tracesPerExporter*spansPerTrace, // there are 4 exporters
map[string]string{},
internal.ReceiverMockNamespace,
internal.ReceiverMockServiceName,
internal.ReceiverMockServicePort,
waitDuration,
tickDuration,
)).
Teardown(func(ctx context.Context, t *testing.T, envConf *envconf.Config) context.Context {
opts := *ctxopts.KubectlOptions(ctx)
opts.Namespace = internal.TracesGeneratorNamespace
terrak8s.RunKubectl(t, &opts, "delete", "deployment", internal.TracesGeneratorName)
return ctx
}).
Teardown(stepfuncs.KubectlDeleteNamespaceOpt(internal.TracesGeneratorNamespace)).
Feature()

testenv.Test(t, featInstall, featTraces)
}
4 changes: 4 additions & 0 deletions tests/integration/internal/constants.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ const (
LogsGeneratorName = "logs-generator"
LogsGeneratorImage = "sumologic/kubernetes-tools:2.13.0"

TracesGeneratorNamespace = "customer-trace-tester"
TracesGeneratorName = "customer-trace-tester"
TracesGeneratorImage = "sumologic/kubernetes-tools:2.13.0"

MultilineLogsNamespace = "multiline-logs-generator"
MultilineLogsPodName = "multiline-logs-generator"
MultilineLogsGenerator = "yamls/multiline-logs-generator.yaml"
Expand Down
Loading

0 comments on commit d01ff12

Please sign in to comment.