Support SSA/ASS styling

[jakobkukla]

Add support for SSA/ASS styling. Currently there is only "minimal" support for ASS subtitles since styling was removed from the initial PR. (See #2281)

It would be great to have full support of the spec. Many subtitles use these styling features to work directly with the video (e.g. to substitute text in a video). SSA/ASS styling is also very common in certain genres of media like Cartoons, Anime or Karaoke.

[andrewlewis]

@icbaker Tentatively marking as an enhancement, but please dupe or close as needed. Thanks!

[icbaker]

Thanks for the request.

Unfortunately we don't currently have any capacity to improve ExoPlayer's SSA/ASS styling support. However we would welcome high quality PRs in this area.

I'm also happy to discuss various approaches with PR authors if a new feature doesn't fit easily into the existing code structure (either before the PR, or in the review, whichever is easier).

[13steinj]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

(I'm not familiar with what exactly even is supported currently).

[icbaker]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

That sounds like a good idea - the best way to do that is probably to compare the SsaDecoder to a copy of the spec. I've found the SSA/ASS spec to be fairly loose, and I'm not sure there's a canonical definition, but I've generally referred to this when working with these files in the past: http://moodub.free.fr/video/ass-specs.doc (linked from Matroska's page on SSA).

In case it's helpful, we would probably prioritise things roughly in this order:

• Features that look like they're implemented but don't work properly (basically bugs).
• Features that are similar to existing ones, and so can be implemented with small incremental changes.
• Features that don't fit into the existing code architecture - so implementing them requires significant refactoring.

I don't know if SSA has anything in the final category - but if it looks like it does then please come talk to us before spending ages refactoring SsaDecoder, as we might have some ideas about the best way to approach the problem.

[cosmo321]

I'm not a developer, so I can't exactly contribute a lot, but proper support of ass/ssa in Exoplayer is something I'd very much would like to have, as Plex for Android currently makes me sad...
I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

[icbaker]

I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

Thanks for the pointer to libass, I took a quick look. I think you're possibly suggesting two alternative things:

• ExoPlayer could depend on libass and use it's logic for all the SSA rendering.
• People working on improving ExoPlayer's existing SSA rendering logic could use libass as a reference implementation to see how different pieces of content should be rendered, as an augmentation to the "spec".

In case anyone is considering trying the first of these, I'd like to pre-emptively warn against it. libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java - so it's extremely unlikely we'd accept a PR that introduced a native dependency to achieve this. There's more discussion about (not) using native deps for parsing subtitles in #7746 and #7749.

The second suggestion (use it as a reference implementation) makes perfect sense - and afaict it's used by VLC, so it should be enough to just play any content with VLC and compare to ExoPlayer's behaviour.

[cosmo321]

I leave that up to actual developers to decide. I just thought I'd point out that libass exists and is probably the best reference we have, and that rendering subs differently from libass could potentially make people like me wonder why the subs behave differently in ExoPlayer. The second alternative is probably the better option for ExoPlayer. :)

[szaboa]

hey @icbaker , I'll check what the current SsaDecoder supports and what is missing in regards of styling, based on the specs you linked above, then I could probably start and implement those step by step (with my limited time).

@jakobkukla did you have anything specific in mind? I mean is there a specific styling option which you would like to see first?

[icbaker]

Thanks for taking a look! Breaking down the work into small PRs definitely sounds good.

[jakobkukla]

@szaboa To be honest I'm not really aware what is currently supported and what not. But the ability to display fonts and colors properly would be nice!

[szaboa]

@icbaker
I've checked out and it seems that only the v4+ "alignment" is supported. Here is the list of all the style attributes of v4 and v4+ from the specs. Do you see specific styles that would be very cumbersome to implement at first sight? E.g. thinking of "fontname", and "encoding"..

Style name	v4	v4+	Note	Format line support	Override support	PR
Name	✅	✅	-	✅	-	-
Fontname	✅	✅	-	❌	❌	-
Fontsize	✅	✅	-	✅	❌	#8615
PrimaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	✅	❌	#8490
SecondaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
OutlineColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR Called OutlineColour in v4+, and TertiaryColour in v4	❌	❌	-
BackColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
Bold	✅	✅	-	✅	❌	#8654
Italic	✅	✅	-	✅	❌	#8654
Underline	❌	✅	-	✅	❌	#8851
Strikeout	❌	✅	-	✅	❌	#8851
ScaleX	❌	✅	-	❌	❌	-
ScaleY	❌	✅	-	❌	❌	-
Spacing	❌	✅	-	❌	❌	-
Angle	❌	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Outline	✅	✅	-	❌	❌	-
Shadow	✅	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Alignment	✅	❌	values may be 1=Left, 2=Centered, 3=Right. Add 4 to the value for a "Toptitle". Add 8 to the value for a "Midtitle"	❌	❌	-
Alignment	❌	✅	layout of the numpad (1-3 sub, 4-6 mid, 7-9 top)	✅	✅	-
MarginL, MarginR, MarginV	✅	✅	-	❌	-	#10169
AlphaLevel	❌	✅	-	❌	❌	-
Encoding	✅	✅	-	❌	❌	-

My plan is to start implement some easy ones first :)

[icbaker]

The spec I linked above (#8435 (comment)) seems to think that Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+. In fact are possibly only supported there and not in v4. Maybe those rows in your table are accidentally flipped? Or maybe I'm misreading the spec...

The SsaDecoder currently ignores v4 styling completely. It's not clear if that's sensible - I don't know how prevalent each version is in the wild.

If you're going to add v4 support we probably need to do think about whether we can reuse the SsaStyle class or split it into two. It's up to you, but for now imo it would be easiest to focus on v4+ only.

These all look like they should be relatively easy (I've referenced the relevant Android styling span where it seems obvious):

• Fontname (TypefaceSpan)
• PrimaryColour (ForegroundColorSpan)
• BackColour (BackgroundColorSpan)
• Bold (StyleSpan)
• Italic (StyleSpan)
• Underline (UnderlineSpan)
• Strikeout (StrikethroughSpan)
• AlphaLevel (read together with PrimaryColour, though the spec I'm reading says "SSA does not use this yet" so maybe not worth bothering with)
• MarginL, R and V (in fact there's already a TODO for these :)). These will feed into Cue's position & line fields. This might be a bit messy - suggest you don't start here.

The spec I'm looking at doesn't really define these clearly enough to be sure:

• Fontsize (I suspect you can use RelativeSizeSpan or AbsoluteSizeSpan but I don't know which is relevant)
  • EDIT: Small correction - if we work out how the spec defines this, we can use Cue.textSize and textSizeType for encoding size information from the "Format" section of the file that applies to the whole cue. We only need to use spans for overrides that affect substrings of the cue text.

These look harder. They have basically no support in the existing Cue interchange format:

• SecondaryColour
• OutlineColour
• ScaleX and ScaleY
• Spacing
• Angle
• BorderStyle
• Outline
• Shadow

These are hard for other reasons:

• Encoding - I'm not sure how you're supposed to read this since it's in a text file - so you've already had to guess the encoding (right?). Currently we assume it's always UTF-8.

[szaboa]

Yep, you are right, the Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+ only, and not in v4. I will correct the table.

Thanks for the tips, will definitely start with the easy ones.

[szaboa]

Initial PR for "Primary Colour" is here #8490.

[moneytoo]

Is there any option/API to disable, customize or detect the styling? If an app sets custom CaptionStyleCompat, it now accepts all styling while the text color is now variable based on line from subtitle file. Also, how about when user uses platform CaptioningManager to customize subtitle look - again, text color is now variable. With static background/border colors and variable text color, the subtitle line may end up low contrast on unreadable (right...?). Does my thinking here make any sense? (I've never used SSA/ASS outside some testing.)

[icbaker]

@moneytoo Assuming you're using the ExoPlayer UI components, you can force the SubtitleView to ignore 'embedded' styles (i.e. those encoded in subtitle files) by calling subtitleView.setApplyEmbeddedStyles(false). You can get the SubtitleView from playerView.getSubtitleView().

[szaboa]

I just want to note that I haven't forgotten about this, just need to find time to continue it :)
Updated the table above with the PrimaryColour support.

[szaboa]

Initial PR for "FontSize" is here: #8615

[lsl330]

@szaboa

OK,thank you very much!!!
I use plex and the exoplayer version is 2.17.1.

[david2tm]

Hi,
I'm not sure if it's a right place to write this -- but I want to try some coding related to SSA/ASS.
The list of stuff these subtitles can do is extensive. So I started looking at the code (mind it is the first time I'll try coding for Android), but I got env set-up already -- with example of subtitles which are not working as they should (looking at VLC rendering) and I want to make them work.
I wander, if it's possible to use WebView to render these Cue's?
I see that there's WebViewSubtitleOutput... so I guess it's already something.
Am I on the right path?

[david2tm]

@icbaker I looked into what SSA/ASS supports and what is implemented and what not. So what I think about it.
Currently there are two "subtitle" renderers. CanvasSubtitleOutput (the default) and WebViewSubtitleOutput (which I'm not sure how it's launched, I guess it's done by the app which uses ExoPlayer).
Anyway - to play with it I forced the WebViewSubtitleOutput, just because doing some styling is easier for me personally with CSS.
Many features ASS supports are not so easily done with CSS as well. I even played with doing stuff like BorderStyle, Outline, Shadow with SVGs, as CSS doesn't support BorderStyle-Outline the way ASS requires it.
And I'm not even talking about Effects -- which are basically animations.
I then went further down wanting to check how libass works and came across their project, which uses webassembly. Btw according to libass github page, Crunchyroll is using this project. I even entertained myself with using said webassembly with WebViewSubtitleOutput, but it's a no go.
As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file. This makes sense. Check this example page. It showcases some capabilities of SSA/ASS which I'm not sure are at all possible with cues.
So what are next steps?

1. One can hope that someone ports libass to Java -- which, probably will never happen :)

2. Quoting you from above. I now think that parsing is indeed easy, but rendering is next to impossible (with cues).

libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java

3. Keep it as it is -- as in, not ever adding the full list of features.

wdyt?

[buliaoyin]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

[jsravn]

Plex has now side stepped this by bundling libass in their android client, presumably using ffmpeg or something to mux on the client side before passing to exoplayer.

[mugiritsu]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

That would be not only the easiest way, but also the only way to correctly render all those obscure and advanced feature-rich ASS files that typesetting enthusiasts can come up with. There are a lot .ass files with over 25MB of animated signs, and at this point not even those long-standing libass alternatives like vsfilter or its derivates can render them properly. Trying to recreate libass would be a neverending task and a huge magnet for complaints about any kind of advanced signs not rendering correctly. Not to mention, libass has years of performance optimizations.

Just adopting libass instead of making a new subtitle renderer could solve a lot of problems.

[Sparh4wk]

I would love to vouch for this one as well.. this is awesome player, but its rly bad for anime without full support for ssa subtitles :(
Please make this happen.

[GlassedSilver]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

That would be not only the easiest way, but also the only way to correctly render all those obscure and advanced feature-rich ASS files that typesetting enthusiasts can come up with. There are a lot .ass files with over 25MB of animated signs, and at this point not even those long-standing libass alternatives like vsfilter or its derivates can render them properly. Trying to recreate libass would be a neverending task and a huge magnet for complaints about any kind of advanced signs not rendering correctly. Not to mention, libass has years of performance optimizations.

Just adopting libass instead of making a new subtitle renderer could solve a lot of problems.

I wholeheartedly agree.

If this issue hadn't been such a long-standing one and been actually looked into by Google itself rather than them watching from the sidelines and hoping for the community to fully do this job by the looks of it, I'd be a lot more optimistic about exoplayer re-implementing and keeping up with changes in-time or mostly in-time. But apparently relying on constantly staying in-sync with ass extensions is a futile effort, so just using libass seems like the most reliable, future-proof-ish effort there is from my limited understanding.

It's a shame, but the dynamic nature of ass is both a curse, but also a blessing because it offers features not found elsewhere. The cost may be annoying. But when libass is doing the work needed why don't we just rely on it like what I think any sensible player does with proper ass-support?

Why re-invent the wheel? It's not like there's much to gain from it right? Even if using libass might not be as performant as a fully-optimized re-implementation, what sorts of differences that can be felt are we talking about here? Subtitle rendering isn't something that needs to scale endlessly, especially not when it's per player, at humanly digestible play-speeds, etc...

To anyone who is investing any sort of work into this effort: you have my most honest gratitude. The day Android has player-independent ACE ass support will be a good day! Not only for us anime and FOSS app fans.

[FongMi]

Is there way to fix overlapping subtitles ?

[GlassedSilver]

Is there way to fix overlapping subtitles ?

It's a known issue, any fix would be supplied by exoplayer extending ass support. I'm not aware of user settings that can fix this.

[Sparh4wk]

so its been two years already and we still dont have proper ssa support.. if using libass can solve this, can some dev do something about it? I think a lot of ppl would be rly happy. I know you are doing this for free, but still. thanks

[happyTonakai]

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

[FongMi]

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

#10169

[happyTonakai]

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

#10169

I'm not sure if this can be called as a progress since nearly two years have passed and this pr is still not merged. In addition, there are still many styles that are not supported. Low priority label makes me desperate...

[e1ker]

I'd love to get this implemented as it would make this player the only one with DV and proper .ass support in Android besides Kodi which is way too heavy for anything other than an Nvidia Shield.

[zhuhang-jasper]

Looking forward to this!