Scraping all EU leagues and other confederations

[srdjov18]

Got it running and I have some data. Thanks for the help setting this up!

Have a quick question about the scraping process:

data/prep/leagues.csv shows that only a handful of leagues in Europe have been scraped. Where in the code can I make modifications to extend the scraping to all leagues in EU, and perhaps other confederations?

[dcaribou]

data/prep/leagues.csv is an leftover from a previous version. I need to get that cleaned up.

Scraped competitions are controlled by the data/competitions.json file. You can have new competitions added to the dataset by adding entries to that file. For example, by adding a line like this, the scrape should start collecting data for the Swiss league:

{"type": "competition", "parent": {"type": "confederation", "href": "/wettbewerbe/europa", "seasoned_href": "https://www.transfermarkt.co.uk/wettbewerbe/europa"}, "country_id": "148", "country_name": "Switzerland", "country_code": "C1", "competition_type": "first_tier", "href": "/super-league/startseite/wettbewerb/C1"}

competitions.json is actually the output of the competitions crawler from transfermark-scraper

[dcaribou]

Perhaps you missed setting a user agent before your run the scraper?
SInce you probably already run it without the user agent setting, also make sure to clear scrapy http cache by removing the .scrapy folder that got created in the root.

[srdjov18]

Great!

Removing .scrapy and setting USER_AGENT worked to get confederations.json.

But when running:

scrapy crawl competitions -a parents=confederations.json -s USER_AGENT=<my user agent> > competitions.json, I get an Unhandled error in Deferred:

Unhandled error in Deferred:
2022-07-08 11:07:41 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1905, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
--- <exception caught here> ---
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 86, in crawl eturn self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 86, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 98, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 86, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 98, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 98, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\spi  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
ders\__init__.py", line 50, in from_crawler
ders\__init__.py", line 50, in from_crawler
    spider = cls(*args, **kwargs)
  File "C:\Users\Serge Jovanovic\transfermarkt-datasets\transfermarkt-scraper\tfmkt\spiders\common.py", line 22, in __init__
    parents = [ json.loads(line) for line in lines ]
  File "C:\Users\Serge Jovanovic\transfermarkt-datasets\transfermarkt-scraper\tfmkt\spiders\common.py", line 22, in <listcomp>
    parents = [ json.loads(line) for line in lines ]
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

2022-07-08 11:07:41 [twisted] CRITICAL:
Traceback (most recent call last):
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 86, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\crawler.py", line 98, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
    spider = cls(*args, **kwargs)
  File "C:\Users\Serge Jovanovic\transfermarkt-datasets\transfermarkt-scraper\tfmkt\spiders\common.py", line 22, in __init__
    parents = [ json.loads(line) for line in lines ]
  File "C:\Users\Serge Jovanovic\transfermarkt-datasets\transfermarkt-scraper\tfmkt\spiders\common.py", line 22, in <listcomp>
    parents = [ json.loads(line) for line in lines ]
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\serge_jovanovic\anaconda3\envs\transfermarkt-scraper\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[dcaribou]

It's complaining that he's not able to decode lines confederations.json as JSONs.
Have you modified the file? If so, could you double check that there's no malformed JSON in that file?

[srdjov18]

Weird. My confederations.json file is the same but double in size as the JSON file in the samples folder.

I used the one in the samples folder and it worked fine.

I did get this error though:

ERROR: Spider error processing <GET https://www.transfermarkt.co.uk/wettbewerbe/afrika> (referer: None)

but when I checked the competitions.json file I could see African competitions there.

[dcaribou]

Uhm. You can try and wrap that line in a try/catch statement and print the line that is causing the error. Something like this should work

try:
 parents = [ json.loads(line) for line in lines ]
except JSONDecodeError as err:
  print("Failed to parse line: " + err.doc)

I did get this error though:
ERROR: Spider error processing <GET https://www.transfermarkt.co.uk/wettbewerbe/afrika> (referer: None)

That should be fine I believe. Sometimes requests to transfermarkt may fail for whatever reason. If the competitions are in the output file, it means scrapy retried and succeeded.

[Charlotte-br560]

Glad to hear you got it running! 😊 Regarding your question about scraping all EU leagues and potentially other confederations, you can adjust the scraping process in the code itself. Look for the section where the leagues are defined or where the scraping logic is implemented. You may need to modify the code to include additional leagues or confederations. Also, if you're open to using a tool, I recommend checking out Crawlbase for efficient scraping. Hope this helps! Let me know if you need further assistance.