You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Not always I use the crawler, actually more often a content scraper following a predefined link path, so I can be sure the complete site is scraped.
The content scraper includes a sys.sleep(1), which slows him down. Removed it myself to make it faster. Also, the multithreaded scraping as in rcrawler should be available with the contentscraper, as well, I think.
The text was updated successfully, but these errors were encountered:
Not always I use the crawler, actually more often a content scraper following a predefined link path, so I can be sure the complete site is scraped.
The content scraper includes a sys.sleep(1), which slows him down. Removed it myself to make it faster. Also, the multithreaded scraping as in rcrawler should be available with the contentscraper, as well, I think.
The text was updated successfully, but these errors were encountered: