Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v4以上版本的writer是否有限流功能? #854

Open
Minnull opened this issue Aug 27, 2024 · 6 comments
Open

v4以上版本的writer是否有限流功能? #854

Minnull opened this issue Aug 27, 2024 · 6 comments
Labels
type: question Further information is requested

Comments

@Minnull
Copy link

Minnull commented Aug 27, 2024

问题描述(Issue Description)

目标redis集群性能太差,迁移速度太快导致目标redis响应时间变长,命令写入超时,redis-shake挂掉。

环境信息(Environment)

  • RedisShake 版本(RedisShake Version):v4.1.1
  • Redis 部署方式(standalone/cluster/sentinel):standalone
  • 是否在云服务商实例上部署(Deployed on Cloud Provider):否

日志信息(Logs)

截屏2024-08-27 15 54 29

其他信息(Additional Information)

`[sync_reader]
cluster = false # set to true if source is a redis cluster
tls = false #
sync_rdb = true # set to false if you don't want to sync rdb
sync_aof = true # set to false if you don't want to sync aof
prefer_replica = true # set to true if you want to sync from replica node
try_diskless = false # set to true if you want to sync by socket and source repl-diskless-sync=yes

[redis_writer]
cluster = false # set to true if target is a redis cluster
sentinel = false # set to true if target is a redis sentinel
tls = false
off_reply = false # ture off the server reply`

@Minnull Minnull added the type: question Further information is requested label Aug 27, 2024
@Minnull
Copy link
Author

Minnull commented Aug 27, 2024

@suxb201 大佬,这里新开了限流相关的问题

@suxb201
Copy link
Member

suxb201 commented Aug 27, 2024

这两个值改小试试:

# redis-shake uses pipeline to improve sending performance.
# This item limits the maximum number of commands in a pipeline.
pipeline_count_limit = 1024

# Client query buffers accumulate new commands. They are limited to a fixed
# amount by default. This amount is normally 1gb.
target_redis_client_max_querybuf_len = 1024_000_000

@Minnull
Copy link
Author

Minnull commented Aug 27, 2024

请问一下,target_redis_client_max_querybuf_len 这个1gb代表什么含义呢?有办法推出来对应的迁移速度是每秒多少吗?

比如限流配置的话:每次要迁移一个集群,可以提前知道目标集群最多可以接受每秒10M的数据。我把迁移工具的限流参数调的小于每秒10M即可。

@suxb201
Copy link
Member

suxb201 commented Aug 27, 2024

你修改 pipeline_count_limit 改小试试先

@Minnull
Copy link
Author

Minnull commented Aug 29, 2024

分别测试了3组case
(1)pipeline_count_limit = 51200,启动后,短时间内很快就报错了
(2)pipeline_count_limit = 1024,运行一段时间还是可以复现上面的报错,比case 1坚持的时间久
(3)pipeline_count_limit = 1,在测试环境可以正常运行,直到完成迁移

以上测试,可以说明调整该参数,可以有效避免上面的报错。
在迁移新集群时,期望保证迁移速度和工具稳定性的话,应该如何合理的设置该值的大小呢?

@suxb201
Copy link
Member

suxb201 commented Aug 29, 2024

@Minnull 我也不知道,没遇到过目的端性能太差的场景。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants