feat:separation of fast and slow commands #2162
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
from
1aeb603
to
daa8a12
Compare
dingxiaoshuai123
changed the title
dingxiaoshuai123
changed the title
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
from
daa8a12
to
f9ed848
Compare
codis/pkg/proxy/backend.go
Outdated
| @@ -491,18 +507,23 @@ func (s *sharedBackendConn) BackendConn(database int32, seed uint, must bool) *B | |||
| type sharedBackendConnPool struct { | |||
| config *Config | |||
| parallel int | |||
| quick int | |||
codis/pkg/proxy/config.go
Outdated
| backend_primary_parallel = 1 | ||
| backend_replica_parallel = 1 | ||
| backend_primary_parallel = 2 | ||
| backend_primary_quick = 1 |
codis/pkg/proxy/mapper.go
Outdated
| opTable[r.Name] = r | ||
| } | ||
|
|
||
| if cmdlist == "" { |
There was a problem hiding this comment.
调用len判断字符串为空
conf/pika.conf
Outdated
| @@ -23,6 +23,13 @@ thread-num : 1 | |||
| # are dedicated to handling user requests. | |||
| thread-pool-size : 12 | |||
|
|
|||
| # Size of the low level thread pool, The threads within this pool | |||
| # are dedicated to handling slow user requests. | |||
| low-level-thread-pool-size : 4 | |||
There was a problem hiding this comment.
改为slowcmdthreadpool
src/pika_admin.cc
Outdated
| } else if (set_item == "slow-cmd-list") { | ||
| g_pika_conf->SetSlowCmd(value); | ||
| ret = "+OK\r\n"; | ||
| }else if (set_item == "max-cache-files") { |
|
todo: |
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
from
1958837
to
859ea43
Compare
Mixficsol
reviewed
Dec 4, 2023
src/pstd/include/pstd_string.h
Outdated
| @@ -50,6 +51,8 @@ int string2int(const char* s, size_t slen, unsigned long* lval); | |||
| int d2string(char* buf, size_t len, double value); | |||
| int string2d(const char* s, size_t slen, double* dval); | |||
| std::vector<std::string>& StringSplit(const std::string& s, char delim, std::vector<std::string>& elems); | |||
| void StringSplit2Set(const std::string&s, char delim, std::unordered_set<std::string>& elems); | |||
There was a problem hiding this comment.
elems这个换成 * 把，一般需要修改的值的用引用+const修饰，需要修改值的用 *
src/pstd/include/pstd_string.h
Outdated
| @@ -50,6 +51,8 @@ int string2int(const char* s, size_t slen, unsigned long* lval); | |||
| int d2string(char* buf, size_t len, double value); | |||
| int string2d(const char* s, size_t slen, double* dval); | |||
| std::vector<std::string>& StringSplit(const std::string& s, char delim, std::vector<std::string>& elems); | |||
| void StringSplit2Set(const std::string&s, char delim, std::unordered_set<std::string>& elems); | |||
There was a problem hiding this comment.
这个delim也尽量不要值传递把，看下用指针还是引用
Mixficsol
reviewed
Dec 4, 2023
Mixficsol
previously approved these changes
Dec 4, 2023
Mixficsol
reviewed
Dec 4, 2023
wangshao1
reviewed
Dec 4, 2023
src/pika_server.cc
Outdated
| // Print the current low level queue size if it exceeds QUEUE_SIZE_THRESHOLD_PERCENTAGE/100 of the maximum queue size. | ||
| size_t low_level_queue_cur_size = LowLevelThreadPoolCurQueueSize(); | ||
| size_t low_level_queue_max_size = LowLevelThreadPoolMaxQueueSize(); | ||
| size_t low_level_queue_thread_hold = (low_level_queue_max_size / 100) * QUEUE_SIZE_THRESHOLD_PERCENTAGE; |
There was a problem hiding this comment.
变量名是想写low_level_queue_thresh_hold？
There was a problem hiding this comment.
这里的名字都要改成slow_cmd相关，下次提交的时候改
| if quick := s.owner.quick; quick > 0 { | ||
| if isQuick { | ||
| i = seed % uint(quick) | ||
| if bc := parallel[i]; bc.IsConnected() { |
There was a problem hiding this comment.
这个地方，原来是在一个循环里，如果一个bc无效会继续找下一个，这里是不是也该加一下。
There was a problem hiding this comment.
这里传入的seed是用key做的hash，为了保证pipeline中相同key的有序，所以没有找下一个连接。之前不知道为什么是进行了遍历，所以新添加的代码没有进行循环找conn
| @@ -55,10 +61,14 @@ const ( | |||
| FlagMasterOnly | |||
| FlagMayWrite | |||
| FlagNotAllow | |||
| FlagQuick | |||
There was a problem hiding this comment.
Quick和Slow不是互斥的吗？需要两个都定义吗？
There was a problem hiding this comment.
理论上只需要FlagSlow即可，但是为了简化一些逻辑就加了个FlagQuick，这样在一些打印快慢列表的函数中逻辑更清晰点，不需要的话删除也行
|
继续改进下，解决下文件冲突 |
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
4 times, most recently
from
62d84fa
to
8df24fc
Compare
github-actions
bot
added
the
📒 Documentation
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
from
8df24fc
to
ceb4cff
Compare
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
4 times, most recently
from
bb78f5f
to
719192d
Compare
dingxiaoshuai123
force-pushed
the
split_fast_slow_cmd_codis
branch
from
719192d
to
c44ba2d
Compare
AlexStocks
approved these changes
Dec 27, 2023
chejinge
reviewed
Jan 18, 2024
| return bc | ||
| } | ||
| } | ||
| } | ||
| } | ||
| return s.backend.bc.BackendConn(database, seed, true) | ||
| // fix:https://github.com/OpenAtomFoundation/pika/issues/2174 |
| flagString := "FlagQuick" | ||
| if flag&FlagSlow != 0 { | ||
| reverseFlag = FlagQuick | ||
| flagString = "FlagSlow" |
There was a problem hiding this comment.
为啥FlagQuick 没有引号 FlagSlow有引号 这是有什么特殊的考虑吗
chejinge
approved these changes
Jan 18, 2024
bigdaronlee163
pushed a commit
to bigdaronlee163/pika
that referenced
this pull request
Jun 8, 2024
* split fast and slow cmd in codis * separation of fast and slow commands * Resolve formatting issues. * use the unordered_set instead of unordered_map * fix:disorderly execution of the same key when using pipeline * update go 1.18 to go 1.21 * change proxy.go & mapper.go
cheniujh
pushed a commit
to cheniujh/pika
that referenced
this pull request
Sep 24, 2024
* split fast and slow cmd in codis * separation of fast and slow commands * Resolve formatting issues. * use the unordered_set instead of unordered_map * fix:disorderly execution of the same key when using pipeline * update go 1.18 to go 1.21 * change proxy.go & mapper.go
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
issue : #2085
issue : #2174
1 separation of fast and slow commands
2 modify the codis proxy & pika
3 add slow command queue and slow command thread pool in pika