Synchronization offset

[skittlesvampir]

Problem description: openai/whisper#1770 (comment)

I've uploaded the data at: https://ben.ist-toll.xyz/k/whisper-test-files/

[EtienneAb3d]

@skittlesvampir
Bug fixed: when the accurate text is also in SRT format, both timestamps were in output
🙃

[skittlesvampir]

Oh my god, now it works!! Thank you so much.

Just two small details:

1. The Whisper subtitles often quite long compared to the original subtitle that is broken into smaller pieces. (Screenshot 1) Is there a way to get shorter segments?
2. Sometimes, lot's of consecutive subtitles are combined into a single subtitle (Screenshot 2), because Whisper didn't detect anything for a while. Would it be possible to approximate the subtitles in-between using framerates or something like that?

Screenshot 1

Screenshot 2

[EtienneAb3d]

I think it would be very hard to do a good job when guessing timestamps interpolations. In-between texts could be partially fast or slow and may include some sub-parts without spoken text.

For point 2: the real solution is to improve the Whisper recognition. This can be obtained with WhisperHallu.
https://github.com/EtienneAb3d/WhisperHallu

For both points 1 and 2: I'm currently working on a solution using word-level timestamps and some complementary pre-/post-processing around WhisperHallu. I don't plan to release it fully open-source. We can discuss about it if you have a budget.

[skittlesvampir]

I will check WhisperHallu out, it seems cool.

Unfortunately, I don't have a budget, I'm just synchronizing my own shows so I can understand them better.

Anyways, I think the errors are acceptable, so thank you for your work! I wish your business much success in the future!