-
Notifications
You must be signed in to change notification settings - Fork 8.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor UiaTextRange For Improved Navigation and Reliability #4018
Conversation
- remove _degenerate - _start and _end are now COORDs - _end is always exclusive - de-static-fy functions - all COORDs are in the text buffer coordinate system
Get some tests running again
Make sure you say “closes x, closes y, closes z” to make sure GitHub can figure out what you mean! |
…ange Missing tests for... - GetText - Move - MoveEndpointByUnit Still need to think of how to approach Move and MoveEndpointByUnit since these should be very similar and should reuse a lot of code.
bdd75bd
to
ea22c8f
Compare
Hi @josephsl. You reached out last year about providing assistance with NVDA. We're experiencing an issue where NVDA is successfully getting the text from our TextRangeProvider, but, for some reason, it fails to actually read it. Think you could take a look or provide some guidance here? We've updated NVDA to 2019.3beta2 and enabled the enhanced debugging logs. We're seeing this issue:
It successfully extracts the text but it seems to be getting a null pointer exception somehow. Prior to this PR, NVDA was able to read the text, so I'm definitely doing something in this PR that is causing the issue. What's particularly confusing is that Narrator is not experiencing this issue. Please let us know if you have any ideas for next steps. We want to make sure we do this right. Thanks! |
Hi, try enabling UIA support for consoles (NVDA menu/Preferences/Settings/advanced, check “I understand that changing these settings may cause NVDA to function incorrectly” checkbox, then choose “prefer UIA” and click OK and see if this makes a difference). CC @codeofdusk
From: Carlos Zamora <notifications@github.com>
Sent: Friday, January 17, 2020 4:05 PM
To: microsoft/terminal <terminal@noreply.github.com>
Cc: Joseph Lee <joseph.lee22590@gmail.com>; Mention <mention@noreply.github.com>
Subject: Re: [microsoft/terminal] Refactor UiaTextRange (#4018)
Hi @josephsl <https://github.com/josephsl> . You reached out last year about providing assistance with NVDA. We're experiencing an issue where NVDA is successfully getting the text from our TextRangeProvider, but, for some reason, it fails to actually read it. Think you could take a look or provide some guidance here?
We've updated NVDA to 2019.3beta2 and enabled the enhanced debugging logs. We're seeing this issue:
IO - inputCore.InputManager.executeGesture (14:49:22.266) - winInputHook (17428):
Input: kb(desktop):numpad7
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.299) - MainThread (3932):
_getTextWithFieldsForUIARange
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.301) - MainThread (3932):
rootElement: TerminalControl
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.304) - MainThread (3932):
full text: -a---- 1/16/2020 1:05 PM 106764 OpenConsole.sln
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.306) - MainThread (3932):
Fetching parents starting from enclosingElement
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.310) - MainThread (3932):
Hit root
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.311) - MainThread (3932):
Done fetching parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.311) - MainThread (3932):
Generating controlFields for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Done generating controlFields for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Yielding control starts for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Done yielding control starts for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
Yielding balanced fields for textRange
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
no children
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
Yielding text
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.313) - MainThread (3932):
_getTextWithFields_text start
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.314) - MainThread (3932):
Walking by unit None, with further units of: [1, 2, 0]
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.317) - MainThread (3932):
Chunk has text. Fetching formatting
ERROR - scriptHandler.executeScript (14:49:22.320) - MainThread (3932):
error executing script: <bound method GlobalCommands.script_review_previousLine of <globalCommands.GlobalCommands object at 0x0D2CB9B0>> with gesture 'numpad 7'
Traceback (most recent call last):
File "scriptHandler.pyc", line 205, in executeScript
File "globalCommands.pyc", line 1038, in script_review_previousLine
File "speech\__init__.pyc", line 952, in speakTextInfo
File "NVDAObjects\UIA\__init__.pyc", line 704, in getTextWithFields
File "NVDAObjects\UIA\__init__.pyc", line 692, in _getTextWithFieldsForUIARange
File "NVDAObjects\UIA\__init__.pyc", line 482, in _getTextWithFields_text
File "NVDAObjects\UIA\__init__.pyc", line 177, in _getFormatFieldAtRange
File "UIAUtils.pyc", line 219, in __init__
File "comtypesMonkeyPatches.pyc", line 26, in __call__
OSError: exception: access violation reading 0x00000000
It successfully extracts the text but it seems to be getting a null pointer exception somehow. Prior to this PR, NVDA was able to read the text, so I'm definitely doing something in this PR that is causing the issue. What's particularly confusing is that Narrator is not experiencing this issue.
Please let us know if you have any ideas for next steps. We want to make sure we do this right. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#4018?email_source=notifications&email_token=AB4AXEA6QPHAHEIWBSF6URTQ6JBQ7A5CNFSM4J4YXXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJJKC3I#issuecomment-575840621> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4AXECZUOTEGQ54DEAXWPLQ6JBQ7ANCNFSM4J4YXXRA> .
|
Hmmm, quite odd. Maybe something in ConsoleUIATextInfo is broken here? Perhaps could someone generate a build that uses the standard UIATextInfo class in consoles for testing? CC @michaelDCurran
…Sent from my iPhone
On Jan 17, 2020, at 22:11, Joseph Lee ***@***.***> wrote:
Hi, try enabling UIA support for consoles (NVDA menu/Preferences/Settings/advanced, check “I understand that changing these settings may cause NVDA to function incorrectly” checkbox, then choose “prefer UIA” and click OK and see if this makes a difference). CC @codeofdusk
From: Carlos Zamora ***@***.***>
Sent: Friday, January 17, 2020 4:05 PM
To: microsoft/terminal ***@***.***>
Cc: Joseph Lee ***@***.***>; Mention ***@***.***>
Subject: Re: [microsoft/terminal] Refactor UiaTextRange (#4018)
Hi @josephsl <https://github.com/josephsl> . You reached out last year about providing assistance with NVDA. We're experiencing an issue where NVDA is successfully getting the text from our TextRangeProvider, but, for some reason, it fails to actually read it. Think you could take a look or provide some guidance here?
We've updated NVDA to 2019.3beta2 and enabled the enhanced debugging logs. We're seeing this issue:
IO - inputCore.InputManager.executeGesture (14:49:22.266) - winInputHook (17428):
Input: kb(desktop):numpad7
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.299) - MainThread (3932):
_getTextWithFieldsForUIARange
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.301) - MainThread (3932):
rootElement: TerminalControl
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.304) - MainThread (3932):
full text: -a---- 1/16/2020 1:05 PM 106764 OpenConsole.sln
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.306) - MainThread (3932):
Fetching parents starting from enclosingElement
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.310) - MainThread (3932):
Hit root
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.311) - MainThread (3932):
Done fetching parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.311) - MainThread (3932):
Generating controlFields for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Done generating controlFields for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Yielding control starts for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.312) - MainThread (3932):
Done yielding control starts for parents
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
Yielding balanced fields for textRange
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
no children
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFieldsForUIARange (14:49:22.313) - MainThread (3932):
Yielding text
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.313) - MainThread (3932):
_getTextWithFields_text start
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.314) - MainThread (3932):
Walking by unit None, with further units of: [1, 2, 0]
DEBUG - NVDAObjects.UIA.UIATextInfo._getTextWithFields_text (14:49:22.317) - MainThread (3932):
Chunk has text. Fetching formatting
ERROR - scriptHandler.executeScript (14:49:22.320) - MainThread (3932):
error executing script: <bound method GlobalCommands.script_review_previousLine of <globalCommands.GlobalCommands object at 0x0D2CB9B0>> with gesture 'numpad 7'
Traceback (most recent call last):
File "scriptHandler.pyc", line 205, in executeScript
File "globalCommands.pyc", line 1038, in script_review_previousLine
File "speech\__init__.pyc", line 952, in speakTextInfo
File "NVDAObjects\UIA\__init__.pyc", line 704, in getTextWithFields
File "NVDAObjects\UIA\__init__.pyc", line 692, in _getTextWithFieldsForUIARange
File "NVDAObjects\UIA\__init__.pyc", line 482, in _getTextWithFields_text
File "NVDAObjects\UIA\__init__.pyc", line 177, in _getFormatFieldAtRange
File "UIAUtils.pyc", line 219, in __init__
File "comtypesMonkeyPatches.pyc", line 26, in __call__
OSError: exception: access violation reading 0x00000000
It successfully extracts the text but it seems to be getting a null pointer exception somehow. Prior to this PR, NVDA was able to read the text, so I'm definitely doing something in this PR that is causing the issue. What's particularly confusing is that Narrator is not experiencing this issue.
Please let us know if you have any ideas for next steps. We want to make sure we do this right. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#4018?email_source=notifications&email_token=AB4AXEA6QPHAHEIWBSF6URTQ6JBQ7A5CNFSM4J4YXXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJJKC3I#issuecomment-575840621> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4AXECZUOTEGQ54DEAXWPLQ6JBQ7ANCNFSM4J4YXXRA> .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Can a .exe build of this new console be provided for testing? I suspect this PR will fix a number of issues we had to work around in NVDA. |
I guess one important question is: does NVDA treat windows terminal the same way it treats conhost.exe? The implementation for UIA is shared between the two of them, but if the behavior is different that could be a good starting point. @carlos-zamora: you might try running Host.EXE and doing NVDA against that to reduce the number of variables :) |
No, not yet. See nvaccess/nvda#10305 |
…llptr when attribute not found. Don't know why this is an issue _now_
Turns out, NVDA works on ConHost but not Windows Terminal with this change. @DHowett-MSFT and I used that to trace an issue to our wrapper class for the text ranges. @codeofdusk This change should affect ConHost as well so we're hoping this will make things on your end easier. Let us know if you expect any different behavior though. Here's a build of ConHost.exe for you to play with. NOTE: All I really have left on my radar for this PR is adding polish to word navigation. You can expect the following in the coming commit(s)
|
- disable debug log
I get an error stating that |
Never mind, copying the file to |
This build of NVDA is good for testing these changes – UIA in consoles is preferred when available and our workarounds and word navigation have been disabled. Internally, it is identical to nvaccess/nvda#10716 with a Windows version check that always returns With changes applied, NVDA and the console hang when using the new native word navigation across lines. |
I'm glad to hear that this fixes some of your workarounds! If there's any more workarounds that you're still going through or find, let me know and I'll see if I can fix them in this PR too (or at least track them on the repo). As for the new word navigation, I'm currently working on optimizing that properly here. That should come in in the upcoming commits. Right now it's scanning the entire buffer to decide if it is on the last word and I'm sure there's a better approach haha. With regards to the timeline, this is where things get tricky. This change will affect Windows Terminal in the upcoming release. Though this code is shared with ConHost (the regular Console), the in-box console will not have these changes in Windows 10 version 2004. However, if somebody builds/deploys the ConHost from GitHub or somehow uses the one that comes with Windows Terminal, then it'll still have the changes. @DHowett-MSFT can confirm or provide more clarity on that and correct me if I'm wrong. |
Yes – IUIAutomationTextPattern::GetVisibleRanges in consoles returns junk ranges when the window is maximized and subsequently restored or when it receives a lot of new text.
…Sent from my iPhone
On Jan 24, 2020, at 13:44, Carlos Zamora ***@***.***> wrote:
This build of NVDA is good for testing these changes – UIA in consoles is preferred when available and our workarounds and word navigation have been disabled. Internally, it is identical to nvaccess/nvda#10716 with a Windows version check that always returns True.
With changes applied, NVDA and the console hang when using the new native word navigation across lines.
I'm glad to hear that this fixes some of your workarounds! If there's any more workarounds that you're still going through or find, let me know and I'll see if I can fix them in this PR too (or at least track them on the repo).
As for the new word navigation, I'm currently working on optimizing that properly here. That should come in in the upcoming commits. Right now it's scanning the entire buffer to decide if it is on the last word and I'm sure there's a better approach haha.
With regards to the timeline, this is where things get tricky. This change will affect Windows Terminal in the upcoming release. Though this code is shared with ConHost (the regular Console), the in-box console will not have these changes in Windows 10 version 2004. However, if somebody builds/deploys the ConHost from GitHub or somehow uses the one that comes with Windows Terminal, then it'll still have the changes. @DHowett-MSFT can confirm or provide more clarity on that and correct me if I'm wrong.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
I don't follow. What's the repro steps and the expected behavior? Are you calling |
|
Narrator's word navigation has regressed. This PR will not be ready until that is fixed.
Moved `FindText` to `UiaTextRangeBase`. Now that Search is a shared component (thanks #3279), I can just reuse it basically as-is. #3279 - Make Search a shared component #4018 - UiaTextRange Refactor I removed it from the two different kinds of UiaTextRange and put it in the base class. I needed a very minor change to ensure we convert from an inclusive end (from Search) to an exclusive end (in UTR). Worked with `FindText` was globally messed with in windows.h. So we had to do a few weird things there (thanks Michael). No need for additional tests because it _literally_ just sets up a Searcher and calls it.
Glad to see that this was merged! Any idea when this will ship to Windows 10 conhost? |
@codeofdusk unfortunately, this one missed the train for this year’s Windows releases. We’ll see what we can do about that, but it’s looking like it won’t be out for desktop users for a while. |
Thanks for looking into it – I'd really appreciate seeing this sooner rather than later, as it's blocking an NVDA feature (see nvaccess/nvda#10716). |
🎉 Handy links: |
@carlos-zamora @DHowett-MSFT About when will this PR make it into conhost in insider builds? 30 April? |
"Soon", but I can't officially give any dates. 😄 |
@codeofdusk well, lol, it's our lucky day. This was just released for the inbox console in insider build 19603! |
Hi, let’s do some tests then – installed 19603, so let me know if there are scenarios you want me to test. Thanks.
|
Fix for microsoft/terminal#9239 Summary of the issue: Windows 10 21H1 has been released to insiders with significantly reduced scope (i.e. excluding the UIA changes it was originally set to include), making our internal names misleading. Additionally, there are plans to "undock" conhost from Windows, removing the guarantee that a given Windows version will run a particular version of the console (it's technically already possible to run newer console on older Windows, I do it for testing). Description of how this pull request fixes the issue: Renames: - NVDAObjects.UIA.winConsoleUIA.is21H1Plus -> NVDAObjects.UIA.winConsoleUIA.isImprovedTextRangeAvailable - NVDAObjects.UIA.winConsoleUIA.consoleUIATextInfo -> NVDAObjects.UIA.winConsoleUIA.ConsoleUIATextInfo (Start class name with upper case) - NVDAObjects.UIA.winConsoleUIA.consoleUIATextInfoPre21H1 -> NVDAObjects.UIA.winConsoleUIA.ConsoleUIATextInfoWorkaroundEndInclusive - The implementation works around both end points being inclusive (in text ranges) before microsoft/terminal#4018 - Workarounds for expand, collapse, compareEndPoints, setEndPoint, etc Also moves the text override to the workaround textInfo as #10036 is no longer reproducible in new console. Co-authored-by: Reef Turner <reef@nvaccess.org>
…(Windows 11 Sun Valley 2) (#10964) Supersedes #9771 and #10716. Closes #1682. Closes #8653. Closes #9867. Closes #11172. Closes #11554. Summary of the issue: Microsoft has significantly improved performance and reliability of UIA console: * microsoft/terminal#4018 is an almost complete rewrite of the UIA code which makes the console's UIA implementation more closely align with the API specification. * microsoft/terminal#10886, microsoft/terminal#10925, and microsoft/terminal#11253 form a robust testing methodology for the UIA implementation, including bug fixes in response to newly added tests based on Word's UIA implementation. * microsoft/terminal#11122 removes the thousands of empty lines at the end of the console buffer, significantly improving performance and stability. Since all console text ranges are now within the text buffer's bounds, it is no longer possible for console to crash due to the manipulation by UIA clients of an out-of-bounds text range. * Countless other accessibility-related PRs too numerous to list here. We should enable UIA support on new Windows Console builds by default for performance improvement and controllable password suppression. Description of how this pull request fixes the issue: This PR: * Exposes all three options for the UIA console feature flag in the UI (replaces the UIA check box with a combo box). * Adds a runtime check to test if `apiLevel >= FORMATTED`, and use UIA in this case when the user preference is auto. This is the case on Windows 11 Sun Valley 2 (SV2) available now in beta and set for release in the second half of 2022.
Summary of the Pull Request
This pull request is intended to achieve the following goals...
This also required a change to the wrapper class
XamlUiaTextRange
that has been causing issues with Narrator and NVDA.See below for additional context.
References
#3976 - I believe this might have been a result of improperly handling degenerate ranges. Fixed here.
#3895 - reduced the duplicate code. No need to separate into different files
#2160 - same as #3976 above
#1993 - I think just about everything is no longer static
PR Checklist
UiaTextRange
#3895, Closes Task: Explore de-static
-fyingUiaTextRange
functions #1993, Closes All forms of UIA navigation appear to skip entries when going back and are unable to go forwards #3976, Closes Narrator line navigation gets stuck on some lines #2160Detailed Description of the Pull Request / Additional comments
UiaTextRange
start
is inclusive,end
is exclusive. A degenerate range is when start == end.MoveByUnit()
functions now rely onMoveEndpointByUnit()
functionsEndpoint
,ScreenInfoRow
, etc..TextBuffer
andViewport
XamlUiaTextRange
GetAttributeValue()
must return a special HRESULT that signifies that the requested attribute is not supported. This was the cause of a number of inconsistencies between Narrator and NVDA.FindText()
should returnnullptr
if nothing was found. Introduce UiaTextRangeBase::FindText() for Accessibility #4373 properly fixes this functionality now that Search is a shared moduleTextBuffer
TextBuffer
for proper abstractionAs an example, consider a buffer with this text in it:
" word other "
In selection, a "word" is defined as the range between two delimiters, so the words in the example include [" ", "word", " ", "other", " "].
In accessibility , a "word" includes the delimiters after a range of readable characters, so the words in the example include ["word ", "other "].
Additionally, accessibility word navigation must be able to detect if it is on the first or last word. This resulted in a slight variant of word navigation functions that return a boolean instead of a COORD.
Ideally, these functions can be consolidated, but that is too risky for a PR of this size as it can have an effect on selection.
Viewport
EndExclusive
is added. This is used by UiaTextRange'send
anchor as it is exclusive. To signify that the last character in the buffer is included in this buffer,end
must be one past the end of the buffer. This isEndExclusive
COORD
is in bounds, a flag must be set to allowEndExclusive
as a validCOORD
that is in bounds.Testing
Validation Steps Performed
Tests pass
Narrator works
NVDA works