Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(123pan): optimize rate limiting #6859

Merged
merged 2 commits into from
Jul 25, 2024

Conversation

seiuneko
Copy link
Contributor

perf(123pan): optimize rate limiting

  • eliminating fixed 200 ms delay in getFiles to prevent thread starvation
    Under a concurrency of 5 requests, performance improvement is as follows:
Command Mean [s] Min [s] Max [s] Relative
before 205.099 ± 14.682 188.500 227.832 1.13 ± 0.08
after 182.253 ± 0.188 182.112 182.488 1.00

hyperfine benchmark result

Benchmark Commands
31812 (seiuneko@R9000P) /w
❯❯❯ cat benchmark
#!/usr/bin/zsh

export user=admin pass='D8@Sj&VDXKCM%47s'
# seq 0 255 | xargs printf "%02x\n" > dir_list
xargs -a dir_list -I _ -P 5 curl -u $user:$pass -X PROPFIND -H 'Depth: 1' -o /dev/null -s http://127.0.233.1:5244/dav/123pan/restic-repo/data/_

31804 (seiuneko@R9000P) /w
❯❯❯ hyperfine --shell zsh --runs 5 --prepare './alist-before --data /w/alist-run server 2>/dev/null &; sleep 1' ./benchmark --conclude 'pkill alist; sleep 60' --prepare './alist-after --data /w/alist-run server 2>/dev/null &; sleep 1' ./benchmark --conclude 'pkill alist; sleep 60' --export-markdown result.md --command-name before --command-name after
Benchmark 1: before
  Time (mean ± σ):     205.099 s ± 14.682 s    [User: 0.574 s, System: 0.650 s]
  Range (min … max):   188.500 s … 227.832 s    5 runs

Benchmark 2: after
  Time (mean ± σ):     182.253 s ±  0.188 s    [User: 0.568 s, System: 0.666 s]
  Range (min … max):   182.112 s … 182.488 s    5 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  after ran
    1.13 ± 0.08 times faster than before
  • allowing cancellation via context to mitigate potential DoS attacks by immediately cancelling excessive requests

- eliminating fixed 200 ms delay in getFiles to prevent thread starvation
- allowing cancellation via context to mitigate potential DoS attacks by immediately cancelling excessive requests
Copy link
Contributor

@xrgzs xrgzs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have encountered performance issues these days. After applying these changes, it works well.

@xhofe xhofe merged commit 5fa70e4 into AlistGo:main Jul 25, 2024
3 checks passed
@xrgzs
Copy link
Contributor

xrgzs commented Jul 27, 2024

#6859 导致号被封禁了,估计是修改后调用接口超速了,请回滚到之前的版本

PS:之前的代码及限速部分为123官方人员提交,根据客服回应看,应该不会出现问题

另外之前本人在使用时出现的性能问题为DNS服务器导致,现已查明

@seiuneko @xhofe

xrgzs pushed a commit to xrgzs/alist that referenced this pull request Jul 27, 2024
This reverts commit AlistGo#6859, which introduced performance enhancements in AlistGo#6859. Unfortunately, these changes led to unintended consequences resulting in account bans. The revert is necessary until a more thorough review and testing can be conducted to ensure no negative impacts on account status.
@seiuneko
Copy link
Contributor Author

@xrgzs 经过对代码逻辑的确认,接口限流间隔依然是之前 123 云盘官方提交的 700 ms,下面是实际调试截图,可以看到是按 700 ms 间隔请求的:
alist log

之前在测试时一天内多次触发 123 云盘风控,但账号能正常登录,只是 IP 被封禁,且光猫重新拨号后能正常使用:
123pan web

猜测账号封禁原因可能与你截图里显示的异地登录有关?

@xrgzs
Copy link
Contributor

xrgzs commented Jul 28, 2024

@xrgzs 经过对代码逻辑的确认,接口限流间隔依然是之前 123 云盘官方提交的 700 ms,下面是实际调试截图,可以看到是按 700 ms 间隔请求的:
alist log

之前在测试时一天内多次触发 123 云盘风控,但账号能正常登录,只是 IP 被封禁,且光猫重新拨号后能正常使用:
123pan web

猜测账号封禁原因可能与你截图里显示的异地登录有关?

异地登录倒是没有,我这边的所有登录设备都是处于同一IP的。之前用老版本没有出现问题,更换新版以后跑两天就出事了,现在回到老版本,再测试几天。

Three-taile-dragon pushed a commit to loognsss/blist that referenced this pull request Sep 26, 2024
- eliminating fixed 200 ms delay in getFiles to prevent thread starvation
- allowing cancellation via context to mitigate potential DoS attacks by immediately cancelling excessive requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants