Note We have rebuilt the Entire architecture during a 24 hour Hackathon and brought down the latency to just 3-5 seconds. Checkout the new repo for the latest version of the project.

Live Demo | Pitch Deck | Hackoverflow 1.0

Jab We Meet

Jab We Meet is a web application that allows users who speak different languages to converse with ease by translating the audio on the fly and speaking the translated audio in the native language of the user as well as provide them with translated captions. It also offers HD video and screen share, and can accommodate up to 100 concurrent users. The application also generates automatic meeting summaries and transcripts, making it easy for participants to review important details from the meeting.

This is the Winning solution developed at 36-hour National Level hackathon called 'Hackoverflow 1.0' by PHCET, Navi Mumbai.

Use cases

• Startups and MNCs with international work culture.
• Freelancers with daily foreign communication.
• Content creators & live streamers to reach global range audience.
• Educational organizations to teach courses to multilingual students.

Built with

                             

• Next JS: React-based framework for building server-side rendered and statically exported web apps.
• Typescript: Statically typed superset of JavaScript, adds type annotations to enhance code reliability & readability
• Livekit: End-to-end WebRTC infrastructure to build live video and audio applications.
• Pusher: WebSockets solution for Realtime updates and bidirectional communication.
• TailwindCSS: Utility-first CSS framework
• Planetscale: Highly scalable, globally distributed database
• tRPC: Provides a simple, type-safe way to build APIs for TS & JS
• Prisma ORM: Modern, type-safe ORM for Node.js and TS
• APIs used: Browser's Webspeech API for transcribing and tts/speech synthesis. Google-translate-browser API for translating texts, OneAI's Summarizer for summarizations of meeting transcripts.

Note This project was bootstrapped with create-t3-app. Check package.json to know all the dependencies and tech stack we used.

Features

• Multilingual Meeting Support
• Real-time Translation and Transcriptions
• Automatic Meeting Minutes generation
• Support to upto 100 concurrent users
• Supports HQ Video streaming

Architecture

Installation steps

1. • Fork the repo
   • Clone the repo to your local system git clone https://github.com/swasthikshetty10/hackoverflow.git
   • Change current directory cd hackoverflow
2. Install latest version of Nodejs and install all the dependencies:

npm install

3. Generate prisma client

npx prisma generate

4. Copy and Rename the .env.example to .env, place it in the root directory and fill the essential vars.

cp .env.example .env

Warning Do not rename the original .env.example, it is used to keep track for env vars list as .env with values is gitignored

5. Run the development server:

npm run dev

Known Issues

• There is latency of around 5-6 seconds between the original time and the time when translated audio is spoken out. The latency issue can be tackled by following ways:
  • Use native models for faster transcriptions
  • Use paid APIs for quicker translations like Whisper AI, Google translate etc.
  • Develop a learning model to predict the next moves of the speaker, which is a replacement for WebSpeech API's final transcription boolean.
• As the app relies on browser's native Web Speech API for transcribing and tts, few browsers like Brave with known issues for this API and microphone functions tend to fail at running the app successfully.

Future scope of development

• Use deepAI models to sync video/lip movements & translated audio.
• Resemble the spoken audio(translated) to source audio(speaker) pitch.

Team Members

Nagaraj Pandith	Swasthik Shetty	Tanishka Rao

License

This repository is licensed under Apache License 2.0

Attributions

• Icon from - Flaticon