New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

单任务爬虫速度太慢 #1

Open

wxning1107 opened this issue Apr 5, 2020 · 0 comments

Owner

wxning1107 commented Apr 5, 2020

每一个request需要等上一个request执行完并返回后才可以执行，这里可以一口气发几十个上百个，我们没有充分利用网络，每次一个一个fetch，一个一个等，这里应该做成多任务并发爬虫

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment