Our team:
Data Science & Machine Learning: Michael Wang, Vera Lin
Full Stack Infrastructure & Mobile: Johnson Han, Sophia Liu
Project Manager: Taylor Zhang
This is our Final Year Design Project in Electrical and Computer Engineering Department at University of Waterloo.
Our team is researching a scenery-music recommender system under the supervision of Dr. Alexander Wong. Designed for the visually impaired, users can start recording a short video of their ambient surroundings. Next, our application extract frames of the video, and then the image captioning model would generate a description of the surroundings. Finally, we recommend suitable songs based on the similarity between the generated captions and the potential lyrics. Recently, we had a successful Presentation Session (Video) (ML demo starts at 10:00 in the video) with the Canadian National Institute for the Blind. Sincerely, we believe that our design philosophy to connect various domains – CV, NLP, and mobile development – with our “AI for Good” initiative blessed us with success.
We also attached a small demo video in the repo (https://www.youtube.com/watch?v=6jiSHDGs1Cs). In this demo, we first select a picture with snow and upload it to our mobile app. Then the algorithm generates both the captions and recommended music. You can see we get pretty accurate prediction and recommendation!
If you want to walk through Machine Learning related code, please check "ml-research" directory.