[AI] A necessary step forward (or maybe lateral?)

[MarcoRavich]

Hi everyone,
1st of all a heartfelt/dutiful thanks to all devs/contributors of this project which I am a proud and satisfied user.

Given the increasing spread of so-called "artificial intelligence" (which at the moment is "just" machine learning, neural networks, large language models), I started to wonder what the future deserves for both fansubbers and softwares like this.

As IT tech - even if not a dev - I've always believed that technological evolution must be "ridden" (or, if you prefer, the "hare's race ahead" approach need to be adopted), so I began to look for what the open source scene offers in this field of usage.

There are of course many "voice to text" or "neural ocr" out there, but the step forward seems to be the "dubbing transpose" (aka translating+voice cloning+lipsyncing) IMHO.

I recently discovered some - even if I haven't yet personally tested any of - interesting open source projects which aims to perform such kind of "transposition":

• @jianchang512's pyVideoTrans;
• @JollyToday's GhostCut;
• @ThioJoe's Auto Synced & Translated Dubs
• @FlorianEagox's WeeaBlind;
• @pranauv1's AI-Video-Translation;
• @R3gm's SoniTranslate;

I would love to have an - both online and offline - "AutoDub" function in SE in the future...

...what do you think about ?

[niksedk]

SE already has a lot of AI - and has really had this for a long time.

Translate:

Audio to text (besides Vosk):

Text to speech:

OCR:

[MarcoRavich]

Thanks for the lightening-fast reply,
yes I know that SE uses many "AI" services for its features.

The question is if audio translation/transposition (= voice-isolation+voice-clone+text-to-speech+audio-merge) has been ever evaluated or it's just out of the project scope.

Thanks in advice.

[niksedk]

You can do this by first using "Audio to text" and then do "Text to speech", with an optional "Auto translate" between.

[MarcoRavich]

You can do this by first using "Audio to text" and then do "Text to speech", with an optional "Auto translate" between.

...only voice-cloning is missing, then...

[cvrle77]

Service ElevenLabs can clone your voice.

[MarcoRavich]

Service ElevenLabs can clone your voice.

Of course, but it's an online - closed source and paid - service.

Applio - which is opensource (MIT) - offers offline TTS/cloning (using your local computing power) instead.

Interesting "review" about: