An app which uses AWS Serverless computing to scrape South African Grain Future Prices. The data is then parsed into an easy-to-view desktop application to monitor the current prices for future contracts.
- AWS Lambda and the basic principles of serverless computing.
- Layers in AWS lambda to use python libraries.
- AWS DynamoDB database for storing futures prices.
- AWS SDK and BOTO3 for Infrastructure as Code.
- Web Scraping using BeautifulSoup4.
- DearPyGui a C++ GUI library wrapped in Python to make my desktop application.
- AWS CLI Tools.
- Automating my Lambda using AWS EventBridge.
- Creating an Architecture of the AWS Services most effective for this project.
- Download the packages imported on all of the script files.
- Download the many Linux x86_64 .whl version of the python packages for the scraping_script.py.
- Zip the packages in one file.
- Create a DynamoDB table corresponding with the data which is being scrapped.
- Create an IAM role in your AWS Management Console to allow a Lambda function to read and write data from your DynamoDB table.
- Create a Lambda function and assign the role.
- Add the zipped file with the create packages in the layers of the function.
- Create an Eventbridge to automate the Lambda function to scrape daily.
- Install AWS CLI Tools on the Desktop Environment where you will run the application and configure it to the same VPC as your DynamoDB and Lambda function.
- Run the app.