Word navigation not working with screen reader

[isidorn]

From @RobGallo

Depending on which edit control is being used, the problem is either in UIAutomations’s Text pattern implementation or IAccessible2’s IAccessibleText implementation. Either way, the text returned for the word at the carat position is incorrect.

Here are the basic rules:

The text of words returned by the accessibility API must match keyboard word navigation (ctrl+leftArrow and ctrl+rightArrow) exactly.
The behavior of ctrl+leftArrow and ctrl+rightArrow must be symmetrical (i.e., the word breaks must not change regardless which command is used).
The placement of the carat should never be on white space, except at the end of a line
Word navigation should stop at all punctuation marks

Here is a line from a file I’m working in. I started at the beginning of the line and I placed vertical bars (|) where the ctrl+rightArrow stops.

import| {| NamedSFC| }| from| '../../|common|/react|/named|-sfc|';|

based on this, the words returned by the accessibility API should be:

“import”
“ {”
“ NamedSFC”
“ }”
“ from”
“ '../../”
“common”
“/react”
“/named”
“-sfc”
“';”

If this doesn’t look right to you, good. The places where the keyboard stops, and thus what is considered a word, make no sense. And as a coder myself, I find these word breaks not useful because I am often placed on space characters. That would be a problem even if I weren’t using an access technology.

I would arrange the words in the line this way:

“import “
“{ “
“NamedSFC “
“} “
“from “
“'”
“..”
“/”
“..”
“/”
“common”
“/”
“react”
“/”
“named”
“-“
“sfc”
“'”
“;”

Visual studio breaks up the line this way, which is also acceptable:

“import “
{ “
NamedSFC “
“} “
“from “
“'../../”
“common”
“/”
“react”
“/”
“named”
“-“
“sfc”
“’;”

Furthermore, the behavior of ctrl+leftArrow and ctrl+rightArrow in VSCode is not symmetrical. For example, here are the word breaks starting from the left and moving right:

from| ‘../../|common

Here are the line breaks for the fragment starting at the end and moving to the left.

from |‘../../|common

Notice the positioning of the line break between “from” and “’” has changed. That is disorienting.

[isidorn]

@RobGallo thanks a lot for providing feedback.

I just tried this and could not reproduce the exact behavior you are seeing.
Do you have "editor.accessibilitySupport": "on" turned on in your settings?

What I am seeing is that ctrl + left and ctrl + right are symetrical. They do not navigate ideally as in your example, but navigate the only way that NVDA properly reads the whole line.

Please note that you can configure between two word navigation behaviors, we have
cursorWordAccessibilityRight and cursorWordRight. You can navigate between what to use by simple disabling keybinding assigned to them in the File > Preferences > Keyboard Shortcuts menu.

When we detect that a screen reader is used on Windows we automatically switch to an "accessible" way of navigating words. The reason we do that is becuase NVDA expects that words get navigated in a way Chrome navigates them (because VS Code is detected as a Chrome app). So when we detect the Screen Reader cursorWordAccessibilityRight takes precendece over the regular cursorWordRight.

We first have to check if any of those navigation commands work for you.
Since the cursorWordAccessibilityRight works for all our NVDA users, however it might not be good for JAWS.
Thus I would like if you experiment with both. And if none work we can look into introducing a different one. I would just have to check if we can detect what screen reader is attached to distinguish between Jaws and NVDA.

Same what I just said for cursorWordAccessibilityRight also holds for following commands:

• cursorWordAccessibilityLeft - cursorWordLeft
• cursorWordAccessibilityLeftSelect - cursorWordLeftSelect
• cursorWordAccessibilityRightSelect - cursorWordRightSelect

Once you try this out let us know and we can works on a PR that will improve this.
Thanks!

fyi @Neuronne @derekriemer

[RobGallo]

@isidorn

I'd like to challenge your assertion that, "...the cursorWordAccessibilityRight works for all our NVDA users..." The text editing in VSCode does not behave like any other editor I have used on Windows. If there is a screen reader user out there who finds this behavior satisfactory, I'd like for them to please chime in to this conversation so I can understand what they expect from the next and prior word commands.

NVDA and JAWS work identically in the code I tested; so there does not appear to be much difference between the screen readers.

editor.accessibilitySupport was set to auto, And I can tell that the screen reader is being detected by VSCode. Even still, I change the setting to "on". No change.

I also disabled editor.accessibilitySupport to see if the default navigation worked better, but I found them to be basically the same.

"...but navigate the only way that NVDA properly reads the whole line..."

What is the relevance of the line here?

VSCode does seem to be providing the expected text at word boundaries from IAccessibleText.textAtOffset based on cursor navigation most of the time. The problem is that the cursor boundaries themselves are largely unexpected. As I mentioned in my original message, the cursor needs to stop at each punctuation mark when moving by character. Check out Visual Studio to see what it does.

Here are two specific suggestions to get started:

1. When the cursor stops on white space, the textAtOffset() function should return only white space.
   You can see this problem by moving the cursor to the end of any line. NVDA will announce the previous word boundary, not the white space.

2. When navigating by word, two consecutive white space characters should always be skipped, even if the two characters are different.
   You can see this problem if you move to the end of a line of text which is followed by a blank line, then more text. The cursor will land on the blank line, and that should never happen.

Lastly, my original message is part of this conversation, but as an image. Any other screen reader user who wants to read it, can't.

[isidorn]

@RobGallo first of all sorry for the slow reponse and thanks for your reply.

1. Disabling editor.accessibilitySupport should change how word navigation behaves.. Are you not seeing any difference at all, or simple both do not work for you?
2. "Relevance of the line": I mistyped, what I tried to say is the only way for NVDA to properly read each word on the line when navigating word by word.
3. We are not implementing the textAtOffset() function, since VS Code is built on top of Electron which is built on top of Chromium we can not control this.
4. We can fine tune the cursor navigation. However this is very user dependent, what you are saying I think does not make sense for 100% of our users. What we would have to do is introduce a third cursor navigation style. If that navigation style works good for NVDA then we can consider replacing the accessibility navigation commands with that. Here's a code pointer in case you might be interested in providing a PR

   vscode/src/vs/editor/contrib/wordOperations/wordOperations.ts

   Line 193 in 33e78e0

   return super._move(getMapForWordSeparators(EditorOptions.wordSeparators.defaultValue), model, position, wordNavigationType);

I appologize for the image in the orginal message, I made the mistake of Copying from Outlook who copies to the clipboard as an image. I have fixed the original message to be text.

[webczat]

In my experience it was reading more than it should. when I would navigate word by word I would hear parts of next line when being at the end of the previous line, if I remember correctly.

[RobGallo]

Neither cursor mode worked for me. Isidor, more than half the problem is in TextAtOffset. It may be in Chromium, but if we can’t fix it, the experience is just going to continue to be poor. You said, “this is very user dependent, what you are saying I think does not make sense for 100% of our users.” Visual Studio has had the keyboard behavior I described for 20 years. Most Windows applications have this keyboard behavior. Given these two facts, I don’t think you can successfully make an argument that what I’ve described is not more effective than the current behavior.

[ahicks92]

I am having similar issues. Going to start here. Word navigation with C++ will be i.e. foo->. Navigating over a function will repeat the whole function back as a word multiple times, i.e. foo() { and seems to sometimes not be moving through it. Sometimes move by word includes the next line. Sometimes selecting when moving by word selects parts of the next line.

The net result is I spend a lot of time second guessing word navigation and making sure it did what I want it to have done. Additionally this issue explains why configuring word separators doesn't take effect, I believe (which was the first thing I tried).

Will try the above suggestions.

[ahicks92]

Firstly I'm on NVDA. Should have said.

I tried the above on the line

void MemoryLookaheadStream::reset() {

and it's actually kind of hard to tell what's going on. Selection announcement is always accurate in the sense that it's not reporting things that aren't selected, but definitely acts surprising. Disabling the accessibility versions of selection fixes selection to be roughly what I'd expect, and still lets NVDA announce correctly.

Disabling the accessible versions of word navigation however doesn't fix word navigation. It fixes word navigation to do what it should but the announcement of the current word is massively incorrect.

Even with the accessible version of word navigation, the following code snippet is broken:

	this->recording = false;
}

as "false;\n}" is reported as a word 3 times as opposed to as separate entities.

Perhaps someone knows if/how hard it'd be to get the NVDA side of things playing nice here, because I'm not sure that this can be fixed in VSCode itself either, and it'd be really amazing to have this working. There's small ergonomics issues beyond this but even with this bug VSCode is enough of a good value proposition that I'm using it.

[isidorn]

@camlorn first of all thanks a lot for your feedback and sorry fo the slow response.
We can fine tune the word navigation in VS Code however we want. However that will probalby not fix the overall issue. The problem is how NVDA expectes word navgiation is done as in Chrome.
I am open to changing the word navigatino in whatever way we think is best and will work best for each Screen Reader separatly.

@LeonarddeR this is actually the biggest NVDA - VS Code issue which we would love to fix the most.
@jvesouza @ajborka are you happy with Word Navigation using Orca? Can you give it a spin and provide feedback so we try to get to a concensus here.

[LeonarddeR]

Strange enough, I"m unable to reproduce this with a11y mode on. It looks like things have improved lately, is this true?

The following example:
int cheese = 3; // Comment
Reads the following, as expected to where the cursor is positioned:

• int
• cheese =
• 3; //
• Comment

[ajborka]

@isidorn , word navigation in the editor works according to the standard Linux conventions. So, looks like everything is good to go here.

[RobGallo]

The Linux conventions do not work well at all for screen reader users, which after all, is the point of the bug. The text as given above is not an acceptable experience for blind and low vision users. I reiterate: the punctuation is not part of the word; The cursor should __only__ land on white space at the end of a word when the white space is a line break. One way you can tell the spoken text with navigation is poor (and there is more than one way) is by deleting a word using ctrl + delete. While I haven’t made an exhaustive study of the behavior, I know for certain there are cases where the text deleted by ctrl + delete and word navigation is different. That is by definition astonishing. Those word boundaries should be __exactly__ the same. The bug cannot be resolved without a change to the IAccessibleText support in Chromium. It may not be the responsibility of the VS Code team to fix it. But without that change, editing support for screen reader users is poor.

[ajborka]

@isidorn , @RobGallo , If I take the following text:

int x=1

Try to navigate it by word, I end up with the following results: Starting at the beginning of the line, and pressing ctrl+right arrow, Orca speaks the word it crosses: 'int'. When checking cursor position, it is located after the last character in the word 'int'. Further presses of ctrl+right arrow results in 'x=1' being spoken as a single word where the curser is placed after the number 1. Testing Orca in other environments such as Libre Office Writer, Pluma, Mate terminal, and others, navigation by word works as expected: Orca speaks the word and places the curser at the beginning of the word. Your answer seems correct then, that it is a Chromium/Chrome bug that Google needs to fix. Maybe @joanmarie can help since she has connections to the Chromium developers.

[ahicks92]

I know some of the following is known, but I think it's useful to explain this long form for anyone who isn't following some of this, so apologies to anyone who's already on the same page:

@LeonarddeR's example is going to work. But try it on int foo() and I think it's noticeable for everyone.

I am in agreement with everyone else: the issue is almost certainly not a VSCode problem. The described Chrome issue would explain it, since from the screen reader perspective the algorithm is "ask the text control for the current word", and the text control (1) comes from Chromium and (2) has an entirely different idea of word.

Even if this is declared fixed as-is, it doesn't respect custom word separators. It would be ideal if it was possible to have those be respected, because sometimes words are too big for lack of a better way of putting it, where the last step of getting your cursor to the part of the line of code you want to edit either means way overshooting or way undershooting and arrowing (if I recalled, for instance, I tried adding _ to my word separators). I know there's also navigation by word part, which might be useful, and though I haven't tried it I am almost sure that it probably leads to this bug.

I think it's possible for there to be a minimal definition of fixed that is something to the effect of we understand the behavior, but VSCode provides a lot of customization here and just what a meaningful word is changes language to language and person to person. I, for example, consider it ideal if foo() is actually foo ( ) or foo( ) because it's a special case of an empty argument list and between the parens is a very useful stopping point.

Unfortunately I think this ends at screen reader scripting perhaps with VSCode-side annotations on the DOM element via custom attributes to expose the settings, which on the whole I'd be personally fine with, because otherwise we'd need something new in Aria or one of the upcoming accessible object model specs. Unfortunately I don't know enough about NVDA to do the add-on (not to mention all the others) and it's possible we don't have access to custom HTML attributes anyway, so I will leave further discussion along those lines to others.

[joanmarie]

So in the case of Orca, there are multiple things taking place, one of which I will definitely fix in Orca, and two of which will potentially need to be addressed in Chromium.

Taking the "int x=1" case:

• It is Orca that is combining "x=1" into a single word. I will fix that in Orca.

• When I get the words via AtkText/AtspiText, Chromium says these are the words:

  • "int "
  • "x="
  • "1"

  The fix I will make in Orca to stop the combining of "x=1" into a single word will be "always trust the word reported by Chromium". Thus Orca will treat "x=" as a single word -- in Browse mode. If that is bad, then let's file a bug against Chromium and explain why it is bad and fix it there.

• Without Orca running, here's what happens when I load the following test case data:text/html,<div contenteditable>int x=1</div> (i.e. in Chromium's omnibox/location bar) and press Ctrl+Right. I move:

  • After the "t" but before the space in "int"
  • Between the "x" and the "="
  • After the "1"

  This is potentially relevant to VSCode -- assuming VSCode does not provide its own caret navigation (does it?).

  When Orca users are using VSCode, they should be in Focus mode. In Focus mode, Orca does NOT control the caret. It turns that over to the web app and browser and then tries to present something sane in response to the accessibility events it gets (here, object:text-caret-moved). In other words, if VSCode does not control what happens when Ctrl+Right is used, the Orca user (and I assume all users) will move to the positions I stated above.

[isidorn]

@joanmarie yes VS Code provides its own caret navigation. In short we have a highly customised word navigation based on user feedback and we have a second word navigation which we switch to when we detect a Screen Reader to be more aligned with what screen readers expectes from Chrome.
We can customise this other word navigation in any way we want to make the screen reader experience good.
I just designed this other caret navigation to behave as close as possible to what Chrome does in input boxes. Also there were some community PRs that tried to improve this.

[jvesouza]

@isidorn
just out of curiosity, does the claim that VSCode provides its own caret apply also when we are editing?
Thanks.

[isidorn]

@jvesouza our carret navigation is applied in all editors we use in VS Code. So yes also in the main editor where you are doing editing.
If that does not answer your question can you be more precise?

[mohammad-suliman]

Hello all,
since this is the first time I interact with the VS code team, I would like to thank you for your great efforts to provide such an accessible advanced code editor! Personally, this has transformed the way I code, and made it significantly more easier and enjoyable. And I think this also seems to be the case with other blind devs, so, in short, thanks for this very important work you did and still doing.
I very much agree that word navigation is one of the most significant few accessibility issues when using code. So, for me, giving it a top priority is a good idea.
I think also that this problem is caused by the way Chromium implements word navigation, I tested with Narrator, and it also has the same exact issue that NVDA has for me when navigating by word in the code editor.
Here is the issue I am facing in steps to reproduce format:

1. Open VS code and run NVDA.
2. Type the following lines in to the code editor:
   int x = 1;
   // This is a comment
3. Place the cursor at the beginning of the first line and navigate by word til the cursor reaches the start of the word "this".

Expected behavior:
NVDA reads the words as it does in other text editors.
Actual behavior:
NVDA reads the following:
int
x =
1;//
1;//
This

Note that NVDA reads the beginning of the second line when the cursor is still present in the first one.
Narrator also has the exact same issue, which leads to the conclusion that this is a Chromium bug and not an NVDA one. I am not sure you can modify the behavior of the text and cursor management accessibility API, but I think that this also needs to be done in Chromium, because chrome has other accessibility issues with text inputs, so this needs to be addressed there IMO.
Note also that other issues are present with word navigation as others pointed out, and I think they need to be addressed also, but this one seems to be the most one affecting the experience negatively? Not sure others agree in fact.
I have another example for the same issue with HTML code which is more real, but presenting it is more complex, so I chose that one. However, the HTML issue is more severe from my side.

[mohammad-suliman]

One more observation: I tested the issue with Google Chrome and message body of the compose email in gmail. The same exact issue can be reproduced, although cursor placement is different a bit there as this got customized here.

[isidorn]

Thanks for all the details. I can also reproduce this behavior with Chrome and a web page that just has a textarea element. Thus I have created this issue on the Chromium side
https://bugs.chromium.org/p/chromium/issues/detail?id=1063815

Feel free to comment there if I did not capture something properly.
Adding important label to this issue so we keep it on our radar. For now let's see what the Chromium engineers will say.

@mohammad-suliman thanks for the kind words.
fyi @deepak1556 since this is one of the biggest accessibility issues for us that could be fixed in Chromium. Ideally if Chrome is able to fix this we could potentially back port this fix?

[isidorn]

Ok, I have pushed an improvement in word navigation that we basically let Chrome handle word navigation and we just react on selection changes of the underlying textArea, so we will be aligned with what NVDA expects. I have tried this out and seems to work nicely.

So the behaviour in Insiders from tomorrow will be better than it is today, however it will be perfect once we cherry pick the commit that @joanmarie found. Hopefully we can do this this week, we'll see.

Because of my change you can still use the following commands if you have set custom keybindings for them but from now on they are deprecated and we will probably remove them in the future:
cursorWordAccessibilityLeft, cursorWordAccessibilityLeftSelect, cursorWordAccessibilityRight, cursorWordAccessibilityRightSelect

Thanks all for help, nice community effort :)

[deepak1556]

Thanks for narrowing down the issue, I will look at backporting the change today.

[isidorn]

Lovely, thanks.

[deepak1556]

Fix has been backported https://domoreexp.visualstudio.com/Teamspace/_git/electron-build/pullrequest/316782 (MS-only) , will update here once the fix has been consumed by vscode.

[nirmalsahoo]

GitHubTags:#A11y_VSCode;#A11ySev1;#A11yTCS;#A11yMAS;#DesktopApp;#WCAG1.3.1;#Win10;#Visual Studio Code;#NVDA;#JAWS; #CAITestPass_May2020;#FTP;#STP;

[isidorn]

@deepak1556 thanks for meging in the fix.

I just tried this and works very nicely for me. I tried all the examples that @ahicks92 provided and they all work much better now. The only corner case I found I have captured with this new Chromium issue. And this time I made sure I can repro with latest Chrome Canary https://bugs.chromium.org/p/chromium/issues/detail?id=1181643 fyi @joanmarie

@RobGallo @ahicks92 @mohammad-suliman if you have time it would be great if you can get latest vscode insiders and provide feedback if this is working as expected now https://code.visualstudio.com/insiders/
Thanks

[ahicks92]

I should probably go over to the NVDA issue, but the one thing that would be super nice to have is working customization of word separators. Word navigation is extra useful if you can customize to the programming language since dialing in a specific character while having to move by character is quite painful. I mention this because it sounds like @joanmarie's proposed fix for NVDA would actually solve this issue as well.

@isidorn I should be doing nontrivial C++ things with VSCode again this weekend, can try it then. Can't adopt VSCode at work yet because I haven't had time to make a line number bell and we have style requirements that need one, so I can't get you fast turnaround times.

[isidorn]

@ahicks92 vscode provides the option to customize word navigation however for that to work with NVDA, NVDA has to fix this issue nvaccess/nvda#12091
Please comment there. Thanks

[RobGallo]

@isidorn How does one customize word navigation please?

[isidorn]

@RobGallo you can change the following setting
"editor.wordSeparators"
That is a list of separators that basically define a word.
However until NVDA fixes that issue we ignore this list of separators on Windows when a screen reader is detected and we fall back to Chrome native word navigation. This setting is respected on other platforms for example macOS with VoiceOver or Linux with Orca.

Also there are mutliple commands to navigate words, you can try mutliple of them and try to find the right one. Just go to Preferences > keyboard shortcuts and type "cursorWord" - you can then find a list of keybindings for these various commands or you can assign your own keybinding.
Also there are probably lots of extensions that add cursor navigation - however I did not explore tthis at all. I just noticed there is an extension called "subword navigation".

But the important thing: with NVDA none of those cursor navigations will read the right thing, only the native chrome word navigation will read things properly - which we do by default.

[LeonarddeR]

I'm happy to look into this further, but pointed out the following in nvaccess/nvda#12091 (comment)

As a workaround, I'm still not sure how to do this, as NVDA's word navigation reads the current word you're at, not the word you passed over. Therefore, the difference in caret position before and after the cursor movement doesn't make sense to how NVDA behaves.

Also note that the behavior of ctrl+leftArrow and ctrl+rightArrow isn't symmetrical in Chrome.

[isidorn]

@LeonarddeR great, for the NVDA fix let's continue the discussion in the NVDA issue. Thanks!

[RobGallo]

I should point out that it is not NVDA or JAWS or any other screen reader where problems lie or where fixes should be made. The problem, the actual, fundamental problem, is in Chrome.

The problem we're discussing occurs when the word boundaries for keyboard navigation and text retrieval through IAccessibleText are not consistent. In other words, if the number of characters jumped when moving by word with the keyboard does not equal the number of characters returned when retrieving text at the word boundary through IAccessibleText, a screen reader user will receive either more or fewer characters than are visually represented by a word on screen.

The problem has two parts:

1. Keyboard navigation of text by boundary type, specifically word
2. The IAccessibleText interface, part of the IAccessible2 suite of interfaces, supported by Chrome

IAccessibleText supports retrieving text from an edit field by a specified boundary type, e.g., character, word, line, paragraph, etc. So when a screen reader reads the word at the cursor, this is how it is done:

1. The screen reader asks for the current offset of the cursor.
2. The screen reader asks for the text in the word boundary of that offset.

When navigating by word, the following occurs:

1. The screen reader captures the word navigation keystroke (ctrl+rightArrow or ctrl+leftArrow on Windows).
2. The screen reader most often and ideally passes the keystroke through to the underlying application.
3. The application moves the cursor.
4. The application sends an event indicating the cursor has moved.
5. Based on the cursor moved event combined with the stored knowledge of what keystroke generated the event, the screen reader then requests the word at the cursor as described above.

You will notice in the above description that the application is entirely responsible for both moving the cursor by word and determining what text is within the word boundary when requested through the IAccessibleText interface. Ergo, the problem needs to be fixed in the application, in this case Chrome. Modifying NVDA and/or VSCode may cover over the problem, but it isn't an actual fix.

[ahicks92]

The insiders build helps, but I've got other weird examples now:

char *dest = (char *)out;

Skips the ( when reading by word from left to right, but not when reading from right to left. And:

auto ret = stream->read(count, dest);

Reads -> twice when going from left to right, rather than reading read. When going from right to left, it's fine.

This is the following insiders version:

Version: 1.54.0-insider (user setup)
Commit: e590188f17162393f50feec19263398e6fe02d13
Date: 2021-02-26T20:39:10.253Z (19 hrs ago)
Electron: 11.3.0
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0
OS: Windows_NT x64 10.0.19042

[isidorn]

@ahicks92 thanks for sharing those example. Both examples are already covered with this issue I have created last week https://bugs.chromium.org/p/chromium/issues/detail?id=1181643

So please comment on it so we get some movement by the Google engineers. Hopefully they can fix it on their side and once they do we would just cherry-pick the fix onto vs code.
Thanks!