tagged-use-daily option to enable/disable filtering tagged-table by date.

[lexx-bright]

About the same idea as with index-use-daily. Table definition could be changed to something like:

CREATE TABLE graphite_tagged (
  Date Date,
  Tag1 String,
  Path String,
  Tags Array(String),
  Version UInt32
) ENGINE = ReplacingMergeTree(Version)
ORDER BY (Tag1, Path); 

[Felixoid]

Very cool, I guess =)

It will reduce tagged table uncontrolled growing.

Can you update documentation as well to reflect a new feature there?

[lexx-bright]

@Felixoid documentation updated.

[Hipska]

Doesn't this also need an update/change on carbon-clickhouse?

[lexx-bright]

@Hipska It's optional. Instead of making changes to carbon-clickhouse, tagged table definition could be tweaked to work it out.

[Hipska]

I don't see how the suggested table definition change would result in less data storage. Could you elaborate that? It seems to me the same data is still stored?

[lexx-bright]

It comes down to the difference between ORDER BY (Tag1, Path, Date) suggested in the carbon-clickhouse docs, where table rows -> #of days * #of metrics and ORDER BY (Tag1, Path) without partitioning, where table rows -> #of_metrics

[Hipska]

Oh yes, I missed it was ReplacingMergeTree. But this change will now always return tags which aren't updated/used anymore? Even when setting tagged-autocomplete-days

[lexx-bright]

The same behaviour as with index table and index-use-daily=false. tagged-autocomplete-days hasn't changed.

[Hipska]

Yeah, maybe that one isn't that clear as well 😛

TLDR; does it make sense to have this new table suggestion while having tagged-use-daily=true?

[lexx-bright]

I'd say it doesn't.

[msaf1980]

@lexx-bright One problem with this, as I see. If some write to the past, tags autocomplete will be broken until write to today/next day with greater version. So, may be this option for carbon-clickhouse is needed also ?

[lexx-bright]

@msaf1980, agree. But could be worked around by using ENGINE = ReplacingMergeTree(Date)

SHOW CREATE TABLE graphite_tagged_new

Query id: 143189e7-9284-4083-914d-fdda528113a0

┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE default.graphite_tagged_new
(
    `Date` Date CODEC(DoubleDelta, LZ4),
    `Tag1` String CODEC(ZSTD(9)),
    `Path` String CODEC(ZSTD(9)),
    `Tags` Array(String) CODEC(ZSTD(9)),
    `Version` UInt32 CODEC(DoubleDelta, LZ4)
)
ENGINE = ReplacingMergeTree(Date)
ORDER BY (Tag1, Path)
SETTINGS index_granularity = 8192 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

echo 'test_metric;minus=-;plus=+;percent=%;underscore=_;colon=:;hash=#;forward=/;host=127.0.0.1' 0 $(date +%s) | nc 127.0.0.1 2003
echo 'test_metric;minus=-;plus=+;percent=%;underscore=_;colon=:;hash=#;forward=/;host=127.0.0.1' 0 $(($(date +%s)-2592000)) | nc 127.0.0.1 2003

SELECT
    Date,
    fromUnixTimestamp(Version),
    Tag1
FROM graphite_tagged_new

Query id: d93756ff-35c7-4def-bc88-602fbc250447

┌───────Date─┬─fromUnixTimestamp(Version)─┬─Tag1─────────────────┐
│ 2022-10-18 │        2022-11-17 07:10:20 │ __name__=test_metric │
│ 2022-10-18 │        2022-11-17 07:10:20 │ colon=:              │
│ 2022-10-18 │        2022-11-17 07:10:20 │ forward=/            │
│ 2022-10-18 │        2022-11-17 07:10:20 │ hash=#               │
│ 2022-10-18 │        2022-11-17 07:10:20 │ host=127.0.0.1       │
│ 2022-10-18 │        2022-11-17 07:10:20 │ minus=-              │
│ 2022-10-18 │        2022-11-17 07:10:20 │ percent=%            │
│ 2022-10-18 │        2022-11-17 07:10:20 │ plus=+               │
│ 2022-10-18 │        2022-11-17 07:10:20 │ underscore=_         │
└────────────┴────────────────────────────┴──────────────────────┘
┌───────Date─┬─fromUnixTimestamp(Version)─┬─Tag1─────────────────┐
│ 2022-11-17 │        2022-11-17 07:10:15 │ __name__=test_metric │
│ 2022-11-17 │        2022-11-17 07:10:15 │ colon=:              │
│ 2022-11-17 │        2022-11-17 07:10:15 │ forward=/            │
│ 2022-11-17 │        2022-11-17 07:10:15 │ hash=#               │
│ 2022-11-17 │        2022-11-17 07:10:15 │ host=127.0.0.1       │
│ 2022-11-17 │        2022-11-17 07:10:15 │ minus=-              │
│ 2022-11-17 │        2022-11-17 07:10:15 │ percent=%            │
│ 2022-11-17 │        2022-11-17 07:10:15 │ plus=+               │
│ 2022-11-17 │        2022-11-17 07:10:15 │ underscore=_         │
└────────────┴────────────────────────────┴──────────────────────┘

18 rows in set. Elapsed: 0.010 sec.

OPTIMIZE TABLE graphite_tagged_new FINAL

SELECT
    Date,
    fromUnixTimestamp(Version),
    Tag1
FROM graphite_tagged_new

Query id: 10353f65-0f1c-4800-addc-23abc0b04b15

┌───────Date─┬─fromUnixTimestamp(Version)─┬─Tag1─────────────────┐
│ 2022-11-17 │        2022-11-17 07:10:15 │ __name__=test_metric │
│ 2022-11-17 │        2022-11-17 07:10:15 │ colon=:              │
│ 2022-11-17 │        2022-11-17 07:10:15 │ forward=/            │
│ 2022-11-17 │        2022-11-17 07:10:15 │ hash=#               │
│ 2022-11-17 │        2022-11-17 07:10:15 │ host=127.0.0.1       │
│ 2022-11-17 │        2022-11-17 07:10:15 │ minus=-              │
│ 2022-11-17 │        2022-11-17 07:10:15 │ percent=%            │
│ 2022-11-17 │        2022-11-17 07:10:15 │ plus=+               │
│ 2022-11-17 │        2022-11-17 07:10:15 │ underscore=_         │
└────────────┴────────────────────────────┴──────────────────────┘

9 rows in set. Elapsed: 0.003 sec.

[msaf1980]

@lexx-bright Yes, this a solution. I update documentation with ENGINE = ReplacingMergeTree(Date)

[msaf1980]

Also may be version based on custom truncate date in the future on carbon-clickhouse ?
For example daily (by default), nodaily, monthly (for prevent nodaily tags bloat and smaller size than daily).