Custom postgres databse
We provide data scrapers for EDEKA Rezepte and REWE Rezepte. Both of these will output data in the same format.
cd ./apicrawler
rm -r {quantities,ingredients,recipes} # (optinally) reset scraped data
python3 recipeAPIcrawler_edeka.py
python3 recipeAPIcrawler_rewe.py # you may terminate after some minutes
This will create 3 directories with a lot of data
The scraped data is very unhandy right now. This is were our rust-parser
comes to help.
cd ./apicrawler/rust-parser
cargo run
This will produce ./apicrawler/recipes/recipes.json.new
Run this to generate SQL Insert Statements from ./apicrawler/recipes/recipes.json.new
.
python3 testdata_sql_generator.py
docker build --tag ghcr.io/kuehlfrank/database:latest .
cd ./apicrawler
python3 insertRecipesIntoDb.py
This will convert the single SQL Statements from original-testdata.sql
to optimized SQL INSERTS and save them to 2-testdata.sql
.
python3 testdata_converter.py