Segment did not perform as expected, querynode memory growth was uneven

[yesletgo]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.2.5
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):   pulsar  
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.3
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 84cores/600G
- GPU: 
- Others:

Current Behavior

After manual flush

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

https://video-tag-lib-1251808348.cos.ap-beijing.myqcloud.com/milvus_log/milvus-log.tar.gz

Anything else?

No response

[yanliang567]

/assign @congqixia
it is weird that there are many segments with row_count=0
/unassign

[congqixia]

@yesletgo from the screen shots and the log, it looks like the datanode cannot watch channel in time. It's a known issue and shall be fixed in 2.2.6
You could update this configuration as a quick workaround:

# milvus.yaml
dataCoord:
...
  channel:
    watchTimeoutInterval:  120 # default value is 30

[yesletgo]

@yesletgo from the screen shots and the log, it looks like the datanode cannot watch channel in time. It's a known issue and shall be fixed in 2.2.6 You could update this configuration as a quick workaround:

# milvus.yaml
dataCoord:
...
  channel:
    watchTimeoutInterval:  120 # default value is 30

The configuration has been updated and the datacoord has been restarted，but the phenomenon is still the same.

milvus-log2.tar.gz

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.