Scraping for season 2022

[rgb18]

Hi guys,

I am using the repo and the data to start my journey in sports analytics. I'm still new to this so I'm hoping for some help and guidance. I followed the README steps to setup the environment and dependencies on my local.
But when I came to the data acquisition part, I ran the given code:

make acquire_local ACQUIRER=transfermarkt-api ARGS="--season 2023"

But I got the error:

(transfermarkt-datasets-py3.10) abd@ABUser-MBP transfermarkt-datasets % make acquire_local ACQUIRER=transfermarkt-api ARGS="--season 2023" scripts/runner.sh scripts/acquiring transfermarkt-api --season 2023 Running process: transfermarkt-api 2024-03-11 16:55:00,069 [INFO]: Starting player market value acquisition for season 2023 Traceback (most recent call last): File "/Users/abd/transfermarkt-datasets/scripts/acquiring/transfermarkt-api.py", line 140, in <module> run_for_season(season) File "/Users/abd/transfermarkt-datasets/scripts/acquiring/transfermarkt-api.py", line 120, in run_for_season market_values = asyncio.run(get_market_values(get_player_ids(season))) File "/Users/abd/transfermarkt-datasets/scripts/acquiring/transfermarkt-api.py", line 49, in get_player_ids with gzip.open(players_asset_path, mode="r") as z: File "/opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gzip.py", line 58, in open binary_file = GzipFile(filename, gz_mode, compresslevel) File "/opt/homebrew/Cellar/python@3.10/3.10.13_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/gzip.py", line 174, in __init__ fileobj = self.myfileobj = builtins.open(filename, mode or 'rb') FileNotFoundError: [Errno 2] No such file or directory: 'data/raw/transfermarkt-scraper/2023/players.json.gz' make: *** [acquire_local] Error 1

So I ran just: make acquire_local and it scraped the 2023 season data for me.

However, when I tried to run the season 2022 again with this: make acquire_local ACQUIRER=transfermarkt-api ARGS="--season 2022"

I got the same error as above which I got for season 2023. I'm wondering what's going on and how to fix it so that I can run for season 2022 as well as 2021 up until 2013. I should also mention that I altered the data/competitions.json file just to include 5 leagues (MLS, Saudi Pro League, MX Liga Clausura abd MX Liga Apertura) instead of the default existing leagues. Thank you.

[dcaribou]

Hi @rgb18,

I'd takle this in two steps

Step 1: Fixing the missing players file error

The transfermarkt-api acquirer depends on some data from the transfermarkt-scraper acquirer. In particular, it depends on the players.json file that is one of the outputs of transfermarkt-scraper. If you haven't either

1. run transfermarkt-scraper for all seasons (make acquire_local ACQUIRER=transfermarkt-scraper ARGS="--seasons 2014-2023 --asset players") or
2. pull raw scraped data from remote storage (dvc pull)

The players.json files will be missing in your local and therefore you will get that error. I suggest that you run either one of the commands above (note that 2 will be much faster) and try again.

Step 2: Updating data for the new leagues

Once you've fixed that error and you are sure that part is working for you can continue with modifying the configuration to include more leagues. For that, after you've added the new leagues to data/competitions.json, you'll need to run the scraper so the new data is fetched (make acquire_local ACQUIRER=transfermarkt-scraper ARGS="--seasons 2014-2023 --asset all").

⚠️ This last command will take quite some time to run, since it's scraping for a lot of entities and dates. You may want to break it up in smaller chunks to get faster feedback. For example, by acquiring first for some seasons and then for the rest, etc.