Some zookeeper znodes can remain after database or table are dropped

[hamidtahi]

Version: 23.10.4.25 + zookeeper 3.8.2

It seems with the default replica path, if a table is created and dropped, one or more znodes associated with that table may not get deleted.

This can be problematic if, for example, automated tests are creating and deleting unique databases and tables, causing znodes to grow. It would require manual znode cleanup from time to time, which is not ideal.

Is this intended behaviour, if so could the reasoning be explained? Is there any setting that can help with clean up of these stale znodes?

To reproduce:

Create database:
create database test on cluster default

Create table:

CREATE TABLE IF NOT EXISTS test.mytable ON CLUSTER default
(
   a Int32,
   d DateTime64 DEFAULT now(),
) ENGINE = ReplicatedMergeTree()
PARTITION BY toYYYYMM(d)
ORDER BY (d)

Internally this uses the default replica path: ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')

Drop the database:
drop database test on cluster default sync

The /clickhouse/tables/{uuid} znode will not get deleted:

getAllChildrenNumber /clickhouse/tables
1

SELECT *
FROM system.zookeeper
WHERE path = '/clickhouse/tables'
FORMAT Vertical

Query id: 75e82258-60db-4bd6-bcae-4745023cbcaf

Row 1:
──────
name:  522b38bb-a75a-4fc3-b223-b6f0037ff78d
value:
path:  /clickhouse/tables

[tavplubix]

Will be fixed by #57575

[hamidtahi]

Hi, on version 24.8.5.115 I am still running into this issue and can reproduce as per the steps in the original post. After dropping the database, the node at path '/clickhouse/tables/{uuid}' does not get deleted, but the children nodes do.

Is this expected?

[tavplubix]

Yes, it's expected

[hamidtahi]

Is there any setting/solution to have the node at path '/clickhouse/tables/{uuid}' to get cleaned up as well? This can cause keeper nodes to keep growing in cases where new databases with unique names are created and deleted frequently. For example, if automated testing is creating and dropping databases with timestamp in the name.