Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qcluster经常出现after timeout #1689

Open
ccl123456789012 opened this issue Jul 19, 2022 · 2 comments
Open

qcluster经常出现after timeout #1689

ccl123456789012 opened this issue Jul 19, 2022 · 2 comments

Comments

@ccl123456789012
Copy link

重现步骤

1.docker-compose 启动的archery运行正常
2.k8s运行qcluster.log经常出现reincarnated worker Process-1:14 after timeout导致工单状态需要5分钟才能显示已正常结束
3.请问使用ssh隧道,k8s的archery是不是必须使用宿主机模式

预期外的结果

[2022-07-19 10:33:46,408][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:3 after timeout
[2022-07-19 10:34:12,537][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:4 after timeout
[2022-07-19 10:34:44,709][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:7 after timeout
[2022-07-19 10:34:46,739][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:8 after timeout
[2022-07-19 10:50:40,644][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:9 after timeout
[2022-07-19 10:50:43,168][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:10 after timeout
[2022-07-19 10:50:44,185][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:11 after timeout
[2022-07-19 10:51:05,298][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:12 after timeout
[2022-07-19 10:51:41,987][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:13 after timeout
[2022-07-19 10:51:44,511][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:14 after timeout

日志文本

[2022-07-19 10:33:46,408][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:3 after timeout
[2022-07-19 10:34:12,537][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:4 after timeout
[2022-07-19 10:34:44,709][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:7 after timeout
[2022-07-19 10:34:46,739][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:8 after timeout
[2022-07-19 10:50:40,644][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:9 after timeout
[2022-07-19 10:50:43,168][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:10 after timeout
[2022-07-19 10:50:44,185][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:11 after timeout
[2022-07-19 10:51:05,298][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:12 after timeout
[2022-07-19 10:51:41,987][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:13 after timeout
[2022-07-19 10:51:44,511][MainThread:140276745680704][task_id:django-q][cluster.py:226][WARNING]- reincarnated worker Process-1:14 after timeout

版本

1.8.5

部署方式

K8S

是否还有其他可以辅助定位问题的信息?比如数据库版本等

mysql 5.7
goinc 最新
redis 5.0.7

@ccl123456789012
Copy link
Author

麻烦大家帮忙看看

@hhyo
Copy link
Owner

hhyo commented Jul 20, 2022

这个看起来只是警告,具体表象是什么?
工单执行和回调应该都没有设置超时时间
,只有消息推送有设置60s

关于隧道连接,参考#1362,使用桥接需要做下调整

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants