This is a React Native App that hopes to make learning sign language easy, on the go. The main niche and appeal is the voice to text capabilities.
An educational app and communication tool that converts voice to text to sign language.
-
Main Pitch Video: https://youtu.be/vLvQLgAgt1k
-
IBM API: https://youtu.be/cto9DFt-KPU
-
Prototype Video: https://www.youtube.com/watch?v=sKGbIfZu6UA&feature=youtu.be
-
Figma Deisgn: https://www.figma.com/file/BCgMJeDR6Kzxq7pEuhfE4M/Untitled?node-id=1%3A10
React Styling:
- npm install @react-navigation/bottom-tabs
- npm install react-navigation
- npm start -- --reset-cache
- npm install @react-navigation/native
- npm install --save react-native-gesture-handler react-native-reanimated react-native-screens
Audio Analyzer:
- expo install expo-av
- npm install react-audio-analyser --save
Splash Screen:
- expo install expo-splash-screen
Tensorflow:
- expo install expo-camera
Devpost Submission Link for Hack The North 2021: https://devpost.com/software/sign-5dzuir
Sign is more than an educational app, it’s a communication tool. Our development team was inspired by language apps that incorporate gamification like Duolingo, and educational language apps like Rosetta Stone. With a minimal user-interface, combined with camera and video scan technology, Sign is great for those looking to learn sign language, and those looking to communicate using sign in everyday conversation.
Although there are apps that teach sign language, and even apps that convert voice to sign, we identified the need for a sign language app with a more pared down interface, one that could be used in everyday conversational settings. While the purpose of Sign is to educate, Sign is all about active communication, rather than passive learning.
Incorporating voice-to-text technology, and text-to-image processing, Sign translates spoken word to text and text to sign language, creating a fast and conversational way for beginners to learn sign language and an interactive way for more experienced signers to brush up on their skills. Sign also offers three tiers of language complexity so you can advance in your learning.
Our application is for anyone interested in learning sign language or anyone looking to communicate more effectively with those who are hearing impaired. We have created various levels of complexity so anyone interacting with the app can build on their existing skills. Beginners will learn the sign alphabet to spell out words, intermediate learners will practice words, while advanced learners will learn to structure sentences.
We used Miro and dot voting for ideation and convergent thinking. Low, medium and high fidelity prototypes were built in Figma. The app was built using Javascript, React Native, Android Emulator, and Expo.
We weren’t able to implement all the functionality we envisioned such as text to sign, and camera scanning. Our development team took on the challenge of working with new tools and languages such as javascript, react native, android emulator and expo. Other challenges include, managing version control history, adding audio recorders and listeners, adding camera functionality, incorporating IBM API, and integrating the speech to text translation to our react native app using IBM API, The team had debugging issues that took up a substantial amount of our development time. Another struggle was working remotely from home and the technology limitations this presented.
Our development team learned new technologies by using tools like react native, android emulator, expo and IBM API. In particular, how Expo has made development easier and more intuitive (e.g. publishing app, app technologies such as audio and camera, multi-platform compatibility is automatic, React makes the app building a little more intuitive). During Gear up we learned about different company API’s, and we learned how to incorporate IBM’s API into our project. We also made use of virtual project management tools and ideation tools like Miro and Lettucemeet. Our programmers learned about Figma and Design thinking and our designer learned about coding and programming, we feel this is valuable for better combining these two skill sets in future development work. We also learned a little bit of sign language!
Our development team had limited prior knowledge of mobile development prior to the hackathon. Given this, we’re proud of what we were able to accomplish in a short period of time, as well as the teamwork we exhibited throughout the process. We were able to develop a high fidelity prototype, and create app functionality like, adding audio and listeners, adding camera functionality, translating voice to text using IBM api, creating mobile navigation, a splash screen, building a login system page, learning how to use gradient colour on apps, and sending debugging request to service. We’re also proud of our idea which we believe, if implemented, has the potential to make a positive impact.
What’s next for Sign? We weren’t able to fully develop all the functionality we envisioned. If Sign were to grow, we envision including increased levels of complexity beyond short sentences. We would also incorporate 3D technology and animation to better show hand movement, which is crucial in sign language. We would further develop gamification in the app to make the experience more engaging for learners. Future iterations would include user research upfront as well as usability testing. Our prototype also shows a scanning feature with the purposes of confirming a user’s comprehension. We would continue working on our UI to make the app more user intuitive. We would also implement the functionality we weren’t able to implement.
Arrow by Gajah Mada Studio from the Noun Project https://thenounproject.com/search/?q=arrow&i=3687528
avatar by Creative Stall from the Noun Project https://thenounproject.com/search/?q=avatar&i=1089489
celebrate by sasha willins from the Noun Project https://thenounproject.com/search/?q=celebrating+hands&i=938020
checkmark by Asmuh from the Noun Project https://thenounproject.com/search/?q=checkmark&i=2193585
Dog by Jack Made from the Noun Project https://thenounproject.com/search/?q=dog&i=2274809
Hand by Jamie Yeo from the Noun Project https://thenounproject.com/search/?q=hand&i=1147795
Sign Language by Romain J from the Noun Project https://thenounproject.com/jimenez_romain/collection/sign-language/?i=2029098
legging by Victoruler from the Noun Project https://thenounproject.com/search/?q=legs&i=3339332
locked lock by Alex Bickov from the Noun Project https://thenounproject.com/search/?q=lock&i=1010538
Sign language a by Stephanie Leeson from the Noun Project https://thenounproject.com/search/?q=sign+language&i=250726
Home by Gajah Mada Studio from the Noun Project https://thenounproject.com/search/?q=home&i=3687679
profile by Ananth from the Noun Projecthttps://thenounproject.com/search/?q=profile&i=922821
Sign Language by Romain J from the Noun Project https://thenounproject.com/search/?q=sign+language&i=2029102
word by Richard Schumann from the Noun Project https://thenounproject.com/search/?q=word&i=95000
snap of fingers by Oleksandr Panasovskyi from the Noun Project https://thenounproject.com/search/?q=snap+of+fingers&i=2286121
Sound by Adrien Coquet from the Noun Project https://thenounproject.com/search/?q=sound&i=1939979
Background vector created by pikisuperstar - www.freepik.com