Final Pull request for the File upload system

[kghdxx]

Added the functionality of tabs to the streamlit app.

The app now functions with the ability to store data on the VM, preprocess it with a number of choices, view the data in the VM and download either the original or the preprocessed out again and if needed save over the file in the VM after changes. works with .CSV.

Uses a flask API and python script to download and perform ETL bronze bucket (unprocessed data) is private on the minio server needing access keys to access.
These are stored in a .env file which has been omitted, as well as any hardcoded passwords.

[AmirZandiehprojects]

@kghdxx Great work on the File Upload System! Your changes have significantly improved the functionality and overall structure of the application. The implementation of tabs in the Streamlit frontend, the ETL pipeline improvements, and the Flask API integration are all solid enhancements.
I think it's a good recommendation that you implement authentication and authorization in both the Streamlit frontend and Flask API. While the current implementation looks good and functions well, adding these security features would greatly enhance the robustness and security of the system. Here's why it would be beneficial:

It would protect sensitive data in the MinIO buckets from unauthorized access.
It would allow for user-specific permissions on ETL operations and file access.
It would provide an additional layer of security against potential misuse of the system.

Consider implementing:

User authentication for the Streamlit frontend
API key or token-based authentication for the Flask API
Role-based access control (RBAC) to manage user permissions

These additions would take your already good implementation to the next level in terms of security best practices.
Overall, your work looks great, and this security enhancement would be a valuable next step. Let me know if you'd like to discuss how to approach this or if you need any resources on implementing authentication and authorization in Streamlit and Flask.

[kghdxx]

Hey Amir, as always this feedback is very valuable. I won't get the opportunity to implement the security features as leader this trimester but I will note these down as tasks for the next leader next trimester as part of handover. Cheers!

[AmirZandiehprojects]

hey Kghdxx , you absolutelynailed everything and i appreciate your hard work and dedication .
Thank you
amir .

[ben-AI-cybersec]

Love this, great work.