UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964

codeofdusk · 2020-04-08T23:19:57Z

Link to issue number:

Supersedes #9771 and #10716. Closes #1682. Closes #8653. Closes #9867. Closes #11172. Closes #11554.

Summary of the issue:

Microsoft has significantly improved performance and reliability of UIA console:

Refactor UiaTextRange For Improved Navigation and Reliability microsoft/terminal#4018 is an almost complete rewrite of the UIA code which makes the console's UIA implementation more closely align with the API specification.
Improve UIA movement testing methodology microsoft/terminal#10886, Add more UIA word navigation tests microsoft/terminal#10925, and Fix UIA Word movement tests microsoft/terminal#11253 form a robust testing methodology for the UIA implementation, including bug fixes in response to newly added tests based on Word's UIA implementation.
[A11y] Treat last character as 'end of buffer' microsoft/terminal#11122 removes the thousands of empty lines at the end of the console buffer, significantly improving performance and stability. Since all console text ranges are now within the text buffer's bounds, it is no longer possible for console to crash due to the manipulation by UIA clients of an out-of-bounds text range.
Countless other accessibility-related PRs too numerous to list here.

We should enable UIA support on new Windows Console builds by default for performance improvement and controllable password suppression.

Description of how this pull request fixes the issue:

This PR:

Exposes all three options for the UIA console feature flag in the UI (replaces the UIA check box with a combo box).
Adds a runtime check to test if apiLevel >= FORMATTED, and use UIA in this case when the user preference is auto. This is the case on Windows 11 Sun Valley 2 (SV2) available now in beta and set for release in the second half of 2022.

Testing performed:

Tested the IMPROVED and FORMATTED console implementations as my primary command console for over a year (since February 2020) and worked closely with the Windows Console team (especially @carlos-zamora) to verify stability and report/fix issues. Sent test builds of the new console to several others to verify stability and functionality.

When UIA console is set to auto in advanced settings, UIA is used in the Windows 11 SV2 console and legacy in all older versions.
When UIA console is set to UIA in advanced settings, UIA is always used if available (console has a UIA provider and legacy mode in properties is unticked).
When UIA console is set to legacy in advanced settings, legacy is always used.

Known issues with pull request:

In the event that this PR must be reverted, please do a git revert or the GitHub revert button to roll it back completely (I'm happy to help with conflict resolution should it be necessary). In other words, please do not change the default value of the winConsoleImplementation option in the config spec in an attempt to back out this change similar to #10682, as the "auto" option will then not match default behaviour as expected.

Change log entry:

== New Features ==

In the Windows Console Host used by Command Prompt, PowerShell, and the Windows Subsystem for Linux on Windows 11 version 22H2 (Sun Valley 2) and later:
- Vastly improved performance and stability. (UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964)
- When pressing ctrl+f to find text, the review cursor position is updated to follow the found term. (Efficiently reviewing the output buffer in terminals #11172)
- Reporting of typed text that does not appear onscreen (such as passwords) is disabled by default. It can be re-enabled in NVDA's advanced settings panel. (Passwords spoken in WSL #11554)
- Text that has scrolled offscreen can be reviewed without scrolling the console window. (UI Automation in Windows Console: use UIATextInfo in FORMATTED consoles #12669)
- More detailed text formatting information is available. (Expose Text Attributes to UI Automation microsoft/terminal#10336)

== Changes for Developers ==

In builds of Windows Console (conhost.exe) with an NVDA API level of 2 (FORMATTED) or greater, such as those included with Windows 11 version 22H2 (Sun Valley 2), UI Automation is now used by default. (UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964)
- This can be overridden by changing the "Windows Console support" setting in NVDA's advanced settings panel.

josephsl · 2020-04-08T23:31:44Z

Hi, actually you might want to clarify that legacy consoles will be in effect for 20H2 and earlier so Version 2004 can be covered (note that even in Version 2004, newer Windows Terminal preview release may contain UIA console improvements, and checking for 21H1 property says “True”). Thanks.

josephsl · 2020-04-08T23:36:02Z

Hi, if possible, revert the submodule commit, do git submodule update based on master branch, and commit the rest. Thanks.

codeofdusk · 2020-04-08T23:36:04Z

note that even in Version 2004, newer Windows Terminal preview release may contain UIA console improvements, and checking for 21H1 property says True.

How so? That property is on WinConsoleUIA, not WinTerminalUIA.

josephsl · 2020-04-08T23:44:55Z

Hi, if I do `focus.UIATextPattern.GetVisibleRanges().length == 1` from Windows Terminal preview, it returns True (but then again, I might be doing something wrong). Another way to specify 21H1 when we don’t know the build is specifying the first build where the new routine appeared (in this case, 19603, and I can confirm the new console UIA support can be enabled). Thanks.

codeofdusk · 2020-04-09T00:08:15Z

Yes. Since Windows Terminal always has a correct UIA implementation, ConsoleUIATextInfo (not ConsoleUIATextInfoPre21H1) is always used, and there's no need for an is21H1Plus property.

feerrenrut · 2020-04-09T07:17:46Z

@codeofdusk could you update the description to summarize what is rolled back in the two PR's you link to and also summarize the two microsoft/terminal PR's.

codeofdusk · 2020-04-10T05:57:19Z

I've updated the description.

codeofdusk · 2020-05-01T11:10:31Z

@feerrenrut How can I re-label this PR as Windows Console, not Windows Terminal?

feerrenrut · 2020-05-01T11:25:58Z

Most of the PR seems to be referring to terminal and the microsoft/terminal repo. Can you make the description clearer?

codeofdusk · 2020-05-01T11:30:05Z

Yes, because the terminal repo contains both Windows Terminal and Windows Console source code.

dpy013 · 2020-06-16T10:35:36Z

Will this PR merge NVDA2020.2?
thanks

codeofdusk · 2020-06-16T11:38:42Z

Will this PR merge NVDA2020.2?
thanks

Probably not, as there are still blocking issues on the conhost side. Once the conhost issues are fixed we can look at merging this.

codeofdusk · 2020-10-09T15:54:38Z

@feerrenrut @michaelDCurran Could this PR please be labelled as blocked until the blocking PRs are merged?

Also CC @carlos-zamora, @DHowett.

codeofdusk · 2022-05-19T04:30:03Z

@seanbudd Thanks for approving! I'm not sure why the system tests are failing as my changes are unrelated.

source/UIAHandler/utils.py

It was requested in nvaccess#10964 (comment) that the ternary option be exposed separately to the change in default behaviour to ease revertability. This commit exposes the ternary option as requested and updates the user guide in preparation for PR nvaccess#10964. Co-authored-by: Łukasz Golonka <lukasz.golonka@mailbox.org>

…preparation for usage by default (#13807) Summary of the issue: It was requested in #10964 (comment) that the ternary option be exposed separately to the change in default behaviour to ease revertability. Description of how this pull request fixes the issue: Exposes the ternary option as requested and updates the user guide.

Follow-up of nvaccess#13807. Co-authored-by: Łukasz Golonka <lukasz.golonka@mailbox.org>

codeofdusk · 2022-06-21T03:08:35Z

@seanbudd I've updated this PR now that #13807 has been merged. It is once again ready for review.

Qchristensen

Reads well, great work!

Fix-up of #10964. Summary of the issue: The link to Expose Text Attributes to UI Automation microsoft/terminal#10336 was broken by t2t preproc directives. Additional clarification on "NVDA API levels" might be appreciated by some users.

This comment has been minimized.

Sign in to view

codeofdusk force-pushed the cmduia-default-21H1 branch from 93323c3 to 66c56b9 Compare April 8, 2020 23:37

josephsl mentioned this pull request Apr 8, 2020

Introduce support for Windows 10 Version 2004 (May 2020 Update/build 19041) #10922

Merged

codeofdusk force-pushed the cmduia-default-21H1 branch from 66c56b9 to 1d001b5 Compare April 9, 2020 00:03

feerrenrut added app/windows-terminal New terminal app, potentially supersedes app/windows-console (repo: microsoft/terminal) feature labels Apr 9, 2020

feerrenrut added app/windows-console Legacy console in Windows, potentially will superseded by the app/windows-terminal and removed app/windows-terminal New terminal app, potentially supersedes app/windows-console (repo: microsoft/terminal) labels May 1, 2020

codeofdusk mentioned this pull request May 1, 2020

Support UI Automation in Microsoft Edge Chromium and Google Chrome #10555

Closed

codeofdusk force-pushed the cmduia-default-21H1 branch from 1d001b5 to b2910a5 Compare May 4, 2020 00:45

codeofdusk mentioned this pull request Jul 1, 2020

When NVDA is started from source with python.exe it is not possible to interact with any console window. #8653

Closed

codeofdusk mentioned this pull request Jul 20, 2020

Properly handle and test "end of buffer" in UiaTextRange microsoft/terminal#6986

Closed

codeofdusk force-pushed the cmduia-default-21H1 branch from b2910a5 to 49f8936 Compare August 7, 2020 23:03

codeofdusk mentioned this pull request Oct 12, 2020

Passwords spoken in WSL #11554

Closed

This comment was marked as off-topic.

Sign in to view

codeofdusk force-pushed the cmduia-default-21H1 branch from 1075633 to 9c212e6 Compare May 8, 2022 20:41

This comment was marked as off-topic.

Sign in to view

seanbudd approved these changes May 19, 2022

View reviewed changes

seanbudd reviewed May 20, 2022

View reviewed changes

source/UIAHandler/utils.py Show resolved Hide resolved

seanbudd removed their assignment May 23, 2022

seanbudd self-requested a review May 23, 2022 05:48

codeofdusk marked this pull request as draft June 17, 2022 08:28

codeofdusk mentioned this pull request Jun 17, 2022

UI Automation in Windows Console: expose all three config options in preparation for usage by default #13807

Merged

5 tasks

This comment was marked as outdated.

Sign in to view

codeofdusk and others added 2 commits June 20, 2022 20:01

Additional GUI/documentation fixes for UIA console by default.

6b48d3a

Follow-up of nvaccess#13807. Co-authored-by: Łukasz Golonka <lukasz.golonka@mailbox.org>

Implement the runtime check for UIA console by default.

c80f421

codeofdusk force-pushed the cmduia-default-21H1 branch from c2f83a6 to c80f421 Compare June 21, 2022 03:02

codeofdusk marked this pull request as ready for review June 21, 2022 03:08

seanbudd approved these changes Jun 21, 2022

View reviewed changes

seanbudd requested a review from Qchristensen June 21, 2022 03:23

Qchristensen approved these changes Jun 21, 2022

View reviewed changes

update changes

0ec7808

seanbudd merged commit b817b03 into nvaccess:master Jun 21, 2022

nvaccessAuto added this to the 2022.3 milestone Jun 21, 2022

codeofdusk mentioned this pull request Jun 21, 2022

UI Automation in Windows Console: fix-up changes for UIA by default #13828

Merged

5 tasks

codeofdusk mentioned this pull request Jul 1, 2022

Remove most uses of CompareInBounds microsoft/terminal#13244

Merged

codeofdusk mentioned this pull request Aug 2, 2022

UI Automation in Windows Console and Windows Terminal: Make POSITION_FIRST and POSITION_LAST relative to visible text in FORMATTED consoles #13671

Closed

5 tasks

seanbudd mentioned this pull request Aug 22, 2022

Unable to interract with the interactive prompts in shell #14042

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964

UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964

codeofdusk commented Apr 8, 2020 •

edited

Loading

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

This comment has been minimized.

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

codeofdusk commented Apr 8, 2020

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

codeofdusk commented Apr 9, 2020

feerrenrut commented Apr 9, 2020

codeofdusk commented Apr 10, 2020

codeofdusk commented May 1, 2020

feerrenrut commented May 1, 2020

codeofdusk commented May 1, 2020 •

edited

Loading

dpy013 commented Jun 16, 2020

codeofdusk commented Jun 16, 2020 •

edited

Loading

codeofdusk commented Oct 9, 2020 •

edited

Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

codeofdusk commented May 19, 2022

This comment was marked as outdated.

codeofdusk commented Jun 21, 2022

Qchristensen left a comment

UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964

UI Automation in Windows Console: make default in FORMATTED consoles (Windows 11 Sun Valley 2) #10964

Conversation

codeofdusk commented Apr 8, 2020 • edited Loading

Link to issue number:

Summary of the issue:

Description of how this pull request fixes the issue:

Testing performed:

Known issues with pull request:

Change log entry:

josephsl commented Apr 8, 2020 via email • edited by feerrenrut Loading

This comment has been minimized.

josephsl commented Apr 8, 2020 via email • edited by feerrenrut Loading

codeofdusk commented Apr 8, 2020

josephsl commented Apr 8, 2020 via email • edited by feerrenrut Loading

codeofdusk commented Apr 9, 2020

feerrenrut commented Apr 9, 2020

codeofdusk commented Apr 10, 2020

codeofdusk commented May 1, 2020

feerrenrut commented May 1, 2020

codeofdusk commented May 1, 2020 • edited Loading

dpy013 commented Jun 16, 2020

codeofdusk commented Jun 16, 2020 • edited Loading

codeofdusk commented Oct 9, 2020 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

codeofdusk commented May 19, 2022

This comment was marked as outdated.

codeofdusk commented Jun 21, 2022

Qchristensen left a comment

Choose a reason for hiding this comment

codeofdusk commented Apr 8, 2020 •

edited

Loading

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

josephsl commented Apr 8, 2020 via email •

edited by feerrenrut

Loading

codeofdusk commented May 1, 2020 •

edited

Loading

codeofdusk commented Jun 16, 2020 •

edited

Loading

codeofdusk commented Oct 9, 2020 •

edited

Loading