Accessibility support for linux needs to be implemented

[webczat]

introduction

As a blind user which daily uses linux, I am unable to use any avalonia based applications because they are inaccessible to screenreaders, while such support exists for windows and mac. I would like accessibility support for linux to be added.

Some details

I don't know how avalonia architecture looks like, so can't comment on that part, however I can share some things I gathered about the linux accessibility stack.

Because avalonia does not use any ui toolkit under the hoot and is based directly on X (and possibly wayland, doesn't really matter that much), there are two options to implement accessibility:

1. Use ATK library as a dependency and implement ATK interfaces.
2. Expose application directly to at-spi, the actual linux accessibility stack.
   Option 2 is recommended for cases like avalonia, even gtk4 stopped using ATK in it's implementation, even though GTK was atk's main user. However, below I will try to describe both options, especially that option1 is probably better documented.

Option 1: use ATK

Basically, that requires adding ATK as a dependency. In this case, for each avalonia control, you need a peer object implementing accessibility. This peer object is a GObject-based object inheriting from AtkObject and implementing one or more atk interfaces as appropriate for the object. The avalonia control itself doesn't need to be a GObject object, but one instance of a control needs to be bound to one instance of the AtkObject subclass appropriate for it. This probably can be done dynamically based on avalonia's way to indicate accessibility support for custom controls, if any.

Properties of a control need to be exposed via the atk object as appropriate, and also, any change in properties, state changes or events happening which influence the accessibility state need to be propagated through the atk peer.

ATK itself is not enough because the objects and events need to be exposed to at-spi. So, the library needs to load the at-spi2-atk bridge and initialize it first. I'm just not sure how do you integrate it with something that doesn't use the glib main loop, but you should be able to do it. I haven't done such implementation myself, I've just implemented a fake custom gtk widget with fake accessibility tree once, so my hands-on experience with that is limited.

Option 2: using at-spi directly

So, basically, we have a special accessibility bus, on which applications are exposed, and there is at-spi registry which knows about these applications. There is a library called libatspi, but it's intended for use by other side of the equation, like screenreaders and such like to inquire applications, and it can't be used to expose them. It's done directly over dbus.

Avalonia's job would be to find the accessibility bus (instructions: https://gitlab.gnome.org/GNOME/at-spi2-core/-/blob/main/bus/README.md?ref_type=heads), to connect to it and to expose it's accessibility tree as dbus objects. The last step seems to be registering an application's root object in the registry. There are probably few other special objects to be implemented like a cache...

Generally, dbus objects for accessibility tree need to implement the Accessible interface and one or more other interfaces as appropriate. The root application object also implements the Accessible interface along with Application interface, at least from what I can tell. Introspection data for all the interfaces along with some documentation can be found here https://gitlab.gnome.org/GNOME/at-spi2-core/-/tree/main/xml?ref_type=heads but at least in one place (the event.xml file) it's said that method signatures might be incorrect, so documentation might or might not be complete, and there is lots of info missing like what's the registry's bus name or object path that you need to use when registering an app. However, an app does not need a well known bus name.

According to documentation of accessible's GetParent method and based on local dbus introspection, the root application object should be called "/org/a11y/atspi/accessible/root" with an accessible role of application. All other objects are in it's subpaths, direct descendants seem to be windows (probably with a frame role).

There is also a special object /org/a11y/atspi/cache that is to be exposed by application and that should implement the cache interface, that seems to be for getting dump of the whole accessibility tree, unsure if it's implementation is required.

Registration of an application in registryis done using the socket interface (https://gitlab.gnome.org/GNOME/at-spi2-core/-/blob/main/xml/Socket.xml?ref_type=heads). One can probably detect registry disappearing and reappearing and re-register an app if needed. Unregistering is not necessary, closing dbus connection is sufficient. Socket is implemented on /org/a11y/atspi/accessible/root object of well known bus name org.a11y.atspi.Registry, or at least that's what introspection says.

I would happily provide more info but I have never tried to implement such a thing myself without ATK. But it could be an interesting exercise which I'd prefer to do without avalonia context if at all.

Testing

Testing of course requires playing with a screenreader like orca, but there is also a tool showing accessibility tree called accerciser which can be used to debug things. It can also show accessibility events based on filters and do other fancy things which can be helpful when debugging, but note that some errors will only appear when using an actual screenreader as how it reacts to some events is not always that obvious.

[grokys]

I did start on an implementation of accessibility support for Linux using AT-SPI but I quickly hit a few issues:

• As mentioned elsewhere, there are no docs. When I tried even the XML definitions were in a non-working state. Looking at them now, they seem to be in a better state.
• I asked if there's any documentation and was basically told "the documentation is the GTK source code". I notice that there are now documentation comments in the XML definitions at least
• I had difficulty following and debugging what the GTK source code was doing
• The way that AP-SPI works with DBus seems to be rather unusual and at the time https://github.com/tmds/Tmds.DBus didn't seem to be able to handle it, which is where I gave up

It's unlikely that I'll personally have any time soon to resume work on this (unless a customer wishes to pay for the feature), but the aborted work I was doing can still be found on the ui-automation-atspi branch if anyone is interested in picking it up. Note that this code is probably really outdated and would need updating to the final implementation of the automation APIs in Avalonia.

[webczat]

can you explain the last point about something not being supported by tmds.dbus? that's the most interesting one.
As for lack of docs, maybe they started to improve it after they switched their recommendations to use at-spi directly. Probably some reading sources will still be necessary.

[kekekeks]

can you explain the last point about something not being supported by tmds.dbus?

IIRC, Tmds.DBus didn't support variants properly.

[maxkatz6]

IIRC it's supported now with Tmds.DBus.Protocol and source-generator. cc @affederaffe to confirm.

[kekekeks]

Yes, the protocol library supports manually constructed variants.

[khanshoaib3]

May I suggest using AccessKit. It does all the low level work so the only things to be done in avalonia would be to create and send the gui tree and updates.

[webczat]

At the point where accessibility is already implemented directly for all desktop targets? except linux it probably makes no sense. also linux support is not yet mature in access kit.

[khanshoaib3]

I think we can use AccessKit only for linux, but yeah, thinking about it, it will essentially result two systems for a11y rather than having the existing system be compatible with linux.
And yup, just looked at the docs stating an issue with Orca.

[khaledmahmoud78]

Excuse me, there are C# cross-platform frameworks compatible with Orca screen reader on Linux, such as GTK#, Eto.Forms and AlterNET UI.
How did they solve this problem?!

[maxkatz6]

@khaledmahmoud78 they didn't solve this problem. They never had it fundamentally. At least GTK# and Eto.Forms map their controls onto GTK controls, which is a semi-native Linux framework. Just like MAUI is mapped onto mobile controls. You lose customization and flexibility, but get native controls look and build-in accessibility.

Can't say much about AlterNET.

[khaledmahmoud78]

@maxkatz6 Thanks for explanation!