Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: aggregate byte and count metrics #13731

Merged
merged 24 commits into from
Aug 16, 2024
Merged

feat: aggregate byte and count metrics #13731

merged 24 commits into from
Aug 16, 2024

Commits on Aug 12, 2024

  1. Squashed commit of the following: 

    commit b9e647f
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Mon Aug 12 11:31:31 2024 -0600

    Squashed commit of the following:

    commit c6ab6b3
    Author: Trevor Whitney <trevorjwhitney@gmail.com>
    Date:   Mon Aug 12 11:09:43 2024 -0600

        chore: remove initial metric aggregation experiment (#13729)

    commit 3c0e3e2
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 12:00:51 2024 -0400

        fix(deps): update module github.com/baidubce/bce-sdk-go to v0.9.186 (#13864)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 6f79194
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 10:20:34 2024 -0400

        fix(deps): update module github.com/aliyun/aliyun-oss-go-sdk to v2.2.10+incompatible (#13861)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit ad60738
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 09:53:11 2024 -0400

        chore(deps): update grafana/loki-build-image docker tag to v0.33.6 (#13859)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 292f911
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 09:46:36 2024 -0400

        chore(deps): update helm/chart-testing-action action to v2.6.1 (#13855)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit e0bde12
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 09:45:49 2024 -0400

        chore(deps): update grafana/promtail docker tag to v1.6.1 (#13851)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit a9cd5e5
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 09:42:53 2024 -0400

        chore(deps): update grafana/promtail docker tag to v2.9.10 (#13854)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 0c28cc7
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 09:42:21 2024 -0400

        chore(deps): update dependency go to v1.22.6 (#13842)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
        Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>

    commit c3a5141
    Author: Paul Rogers <129207811+paul1r@users.noreply.github.com>
    Date:   Mon Aug 12 09:29:19 2024 -0400

        chore: Update loki build image to go 1.22.6 (#13857)

    commit 717623b
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Mon Aug 12 15:47:29 2024 +0300

        chore:  manual changelog backport (#13852)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit f933a3b
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 08:40:39 2024 -0400

        chore(deps): update grafana/loki docker tag to v2.9.10 (#13848)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
        Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>

    commit ea6395a
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 08:33:10 2024 -0400

        chore(deps): update golang docker tag to v1.22.6 (#13847)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 53c0c48
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Mon Aug 12 08:32:08 2024 -0400

        chore(deps): update grafana/loki-build-image docker tag to v0.33.5 (#13849)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 9315b3d
    Author: Grot (@grafanabot) <43478413+grafanabot@users.noreply.github.com>
    Date:   Fri Aug 9 12:38:47 2024 +0200

        chore(ci): Update yaml file `./production/helm/loki/values.yaml` (+1 other) (#13832)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>
        Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>

    commit 66d7138
    Author: Grot (@grafanabot) <43478413+grafanabot@users.noreply.github.com>
    Date:   Thu Aug 8 17:31:41 2024 +0200

        chore: [main] chore(release-3.1.x): release 3.1.1 (#13817)

        Co-authored-by: loki-gh-app[bot] <160051081+loki-gh-app[bot]@users.noreply.github.com>

    commit d5718eb
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Thu Aug 8 10:49:41 2024 -0400

        fix(deps): update github.com/grafana/jsonparser digest to ea80629 (#13814)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit f253db5
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Thu Aug 8 17:12:11 2024 +0300

        fix(ci): fixed release-please manifest (#13810)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit a93f38c
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Thu Aug 8 09:48:10 2024 -0400

        fix(deps): update github.com/c2h5oh/datasize digest to aa82cc1 (#13807)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
        Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>

    commit e5a3994
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Thu Aug 8 09:39:56 2024 -0400

        fix(deps): update github.com/docker/go-plugins-helpers digest to 45e2431 (#13808)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 67295e0
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Thu Aug 8 09:33:48 2024 -0400

        fix(deps): update github.com/axiomhq/hyperloglog digest to af9851f (#13806)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit c9b0343
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Thu Aug 8 09:32:02 2024 -0400

        chore(deps): update github.com/grafana/regexp digest to a468a5b (#13805)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit 5bb928e
    Author: Paul Rogers <129207811+paul1r@users.noreply.github.com>
    Date:   Thu Aug 8 08:52:50 2024 -0400

        chore: Turn off renovate on non-main branches (#13803)

    commit 217f928
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Thu Aug 8 15:31:43 2024 +0300

        fix(ci): add cleanup step into job `dist` (#13801)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit 00e686d
    Author: benclive <ben.clive@grafana.com>
    Date:   Thu Aug 8 10:40:16 2024 +0100

        chore: Add metastore client as dep for rf1 querier & ignore auth for ListBlocks (#13786)

    commit df61482
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Wed Aug 7 18:12:13 2024 -0400

        fix(deps): update module golang.org/x/text to v0.17.0 (main) (#13794)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit f523530
    Author: Periklis Tsirakidis <periklis@redhat.com>
    Date:   Wed Aug 7 20:05:42 2024 +0200

        fix(operator): Don't overwrite annotations for LokiStack ingress resources (#13708)

    commit 5ef83a7
    Author: Ned Andreev <n@andreev.sh>
    Date:   Wed Aug 7 19:44:43 2024 +0300

        fix: panic when parsing and extracting JSON key values (#13790)

    commit bb257f5
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Wed Aug 7 17:39:04 2024 +0300

        feat(loki):  add ability to disable AWS S3 dualstack endpoints usage  (#13785)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit 1bf9791
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Wed Aug 7 16:10:34 2024 +0300

        fix(helm): fixed memcached and provisioner templates (#13788)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit 638f59f
    Author: benclive <ben.clive@grafana.com>
    Date:   Wed Aug 7 14:06:28 2024 +0100

        chore: Remove unused stream index from RF1 ingester (#13758)

    commit 7683a79
    Author: Ned Andreev <n@andreev.sh>
    Date:   Tue Aug 6 20:26:10 2024 +0300

        fix: Include whitespaces in extracted tokens (#13738)

        Co-authored-by: Travis Patterson <travis.patterson@grafana.com>

    commit da63ca7
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Tue Aug 6 12:41:48 2024 -0400

        chore(deps): update module golang.org/x/net to v0.23.0 [security] (main) (#13763)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
        Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>

    commit f8bf3bb
    Author: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
    Date:   Tue Aug 6 12:29:47 2024 -0400

        fix(deps): update module github.com/docker/docker to v27.1.1+incompatible [security] (main) (#13762)

        Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

    commit b88583d
    Author: Marco Pracucci <marco@pracucci.com>
    Date:   Tue Aug 6 16:04:28 2024 +0200

        feat: upgrade prometheus (#13671)

        Signed-off-by: Marco Pracucci <marco@pracucci.com>
        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>
        Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit 3be5a45
    Author: Dylan Guedes <djmgguedes@gmail.com>
    Date:   Mon Aug 5 17:59:21 2024 -0300

        fix(break): helm: Fix how we set imagePullSecrets for enterprise-gateway and admin-api. (#13761)

    commit bdfc86b
    Author: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>
    Date:   Mon Aug 5 13:46:10 2024 +0300

        chore(helm-chart): added SSE config into AWS storage config (#13746)

        Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>

    commit 7e224d5
    Author: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>
    Date:   Sun Aug 4 19:28:29 2024 +0530

        fix: try reading chunks which have incorrect offset for blocks (#13720)

    commit 629671e
    Author: J Stickler <julie.stickler@grafana.com>
    Date:   Fri Aug 2 16:58:41 2024 -0400

        docs: Update the Visualize topic (#13742)

    commit 917053a
    Author: Cyril Tovena <cyril.tovena@gmail.com>
    Date:   Fri Aug 2 18:07:17 2024 +0200

        feat: Introduce wal segment read path. (#13695)

        Co-authored-by: Ben Clive <ben.clive@grafana.com>

    commit 7c50b43
    Author: Paul Rogers <129207811+paul1r@users.noreply.github.com>
    Date:   Fri Aug 2 08:47:33 2024 -0400

        build: Update loki-build-image to Alpine 3.20.2 (#13744)

    commit 6dd6b65
    Author: jackyin <yincong@baidu.com>
    Date:   Fri Aug 2 01:30:11 2024 +0800

        fix: ast left cycular reference result in oom (#13501)

        Co-authored-by: Travis Patterson <travis.patterson@grafana.com>

    commit e81345e
    Author: J Stickler <julie.stickler@grafana.com>
    Date:   Thu Aug 1 11:29:38 2024 -0400

        docs: fix broken links due to Alloy docs reorg (#13715)

    commit 40e8352
    Author: Jack Baldry <jack.baldry@grafana.com>
    Date:   Thu Aug 1 15:41:48 2024 +0100

        docs: rewrite quickstart with Killercoda metadata (#13234)

        Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
        Co-authored-by: J Stickler <julie.stickler@grafana.com>
        Co-authored-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com>

commit c65ba1f
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Mon Aug 12 11:08:37 2024 -0600

    chore: rework pattern ingester queue metrics

commit b28572a
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Aug 2 13:16:01 2024 -0600

    fix: change metrics around metric appends

commit e430a96
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Aug 2 08:59:05 2024 -0600

    fix: lint and format

commit 2f5485d
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Fri Aug 2 08:45:57 2024 -0600

    feat: improve async tee request processing

commit 3dcc50d
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 15:16:13 2024 -0600

    test: add push and downsample test

commit 019df76
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 14:41:41 2024 -0600

    fix: formatting

commit 7d98cb0
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 13:35:55 2024 -0600

    chore: move some functions around

commit 06269b0
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 13:29:57 2024 -0600

    test: mock out ring in tests

commit 07d57c3
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 11:56:34 2024 -0600

    fix: formatting

commit 93e4add
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 11:53:42 2024 -0600

    fix: remove now unused test

commit c03db1f
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 11:49:59 2024 -0600

    fix: remove duplicate service detection for distributor

commit fe447f5
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Thu Aug 1 11:45:31 2024 -0600

    fix: service name detection on push

commit afb8513
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 21:40:39 2024 -0600

    feat: introduce RingClient abstraction

    * lint and format

commit ce87fd8
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 20:41:38 2024 -0600

    feat: aggregate byte and count metrics

    * aggregate in the pattern ingester and push back into Loki as a stream

commit b49e52b
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 14:36:59 2024 -0600

    chore: fix formatting

commit af22a21
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 14:32:05 2024 -0600

    chore: fix linting

commit d51cecc
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 14:12:31 2024 -0600

    chore: remove the metric aggregation experiment

    * this will make way for a new, simpler approach

commit dad6fb5
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 13:39:07 2024 -0600

    Revert "feat: collect and serve pre-aggregated bytes and counts (#13020)"

    This reverts commit 467eb1b.

commit a6835e1
Author: Trevor Whitney <trevorjwhitney@gmail.com>
Date:   Wed Jul 31 13:18:12 2024 -0600

    Revert "feat: downsample aggregated metrics (#13449)"

    This reverts commit 2c053ee.
    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    7842180 View commit details
    Browse the repository at this point in the history

  2. fix: merge whoops

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    41e36e4 View commit details
    Browse the repository at this point in the history

  3. null check tee in distributor

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    553731b View commit details
    Browse the repository at this point in the history

  4. fix: broken import

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    17bb6de View commit details
    Browse the repository at this point in the history

  5. fix: queue size labels

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    5a3c861 View commit details
    Browse the repository at this point in the history

  6. move context closer to push

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    3ee7945 View commit details
    Browse the repository at this point in the history

  7. slight reduction of allocations

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    8b567c3 View commit details
    Browse the repository at this point in the history

  8. fix: level detection

    @trevorwhitney
    trevorwhitney committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    8de6bd4 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. feat: don't tee aggregated metric lines to pattern ingester

    @trevorwhitney
    trevorwhitney committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    2860dc0 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. feat: convert pattern tee to a service

    @trevorwhitney
    trevorwhitney committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    84d88bf View commit details
    Browse the repository at this point in the history

  2. chore: add error logging to metric appends

    @trevorwhitney
    trevorwhitney committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    78a2066 View commit details
    Browse the repository at this point in the history

  3. feat: remove parsing labels from observe

    @trevorwhitney
    trevorwhitney committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    ca6338f View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. fix: nil check invalid level

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    bfaf8d9 View commit details
    Browse the repository at this point in the history

  2. feat: downcase level string

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    cd20b3b View commit details
    Browse the repository at this point in the history

  3. fix: only write agg metrics we have values for

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    b708cbb View commit details
    Browse the repository at this point in the history

  4. test: add tee service test

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    67164ab View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into store-aggregated-metrics-in-loki-2

    @trevorwhitney
    trevorwhitney authored Aug 15, 2024
    Configuration menu
    Copy the full SHA
    1158d8a View commit details
    Browse the repository at this point in the history

  6. docs: fix configuration reference

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    dd41222 View commit details
    Browse the repository at this point in the history

  7. chore: lint and cleanup

    @trevorwhitney
    trevorwhitney committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    826edc5 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. fix: format and mock

    @trevorwhitney
    trevorwhitney committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    2b89765 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into store-aggregated-metrics-in-loki-2

    @trevorwhitney
    trevorwhitney authored Aug 16, 2024
    Configuration menu
    Copy the full SHA
    562766b View commit details
    Browse the repository at this point in the history

  3. chore: move log levels into constants

    @trevorwhitney
    trevorwhitney committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    3277b34 View commit details
    Browse the repository at this point in the history

  4. chore: cleanup

    @trevorwhitney
    trevorwhitney committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    31b620d View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into store-aggregated-metrics-in-loki-2

    @trevorwhitney
    trevorwhitney authored Aug 16, 2024
    Configuration menu
    Copy the full SHA
    391c683 View commit details
    Browse the repository at this point in the history