Preliminary support for UI Automation in Windows Console #9614
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
At the moment, it merely disables NVDA's existing console support, falling back to what UIA support is available in NVDA. The option is currently called `ConsoleUIA`, but we'll need to change this once autodetection is implemented (as there will be three options at that point). Typed characters are now spoken in UIA consoles, but characters are doubled when typing quickly, leading to incorrect typed word reporting.
This fixes the doubled characters bug, making speaked typed words function as intended, but it may have other implications.
…TextInfo classes to implement automatic readout and work around a caret movement bug in Windows 10 version 1903. The consoleUIA class inherits from NVDAObjects.behaviors.Terminal, which serves as the foundation for automatic readout support. At the moment, it is likely that such support will either be not functional at all or report extraneous text from user input.
…nfo->consoleUIATextInfo for consistency. See the source code comments in this commit for some steps still incomplete.
…le a user is typing. This does not regress NVDA functionality, since we already cancel speech when the user types.
…GUI to inform the user that restarting NVDA is required to switch console implementations.
…Ranges. This may improve performance when large amounts of text are written to the console, but possibly at the cost of less reliable autoread (particularly when the screen is refreshed or scrolled).
feerrenrut
reviewed
May 24, 2019
The user guide will need to be updated once auto-detection is introduced (and therefore the NVDA preference is changed).
LeonarddeR
suggested changes
May 25, 2019
|
@feerrenrut @LeonarddeR I think I've addressed all review items. Could you please have another look? |
LeonarddeR
reviewed
May 25, 2019
This fix adds a new textInfo instance variable, _expandCollapseBeforeReview, which stops review from expanding and immediately collapsing the textInfo during review. For some reason, this was causing object review to malfunction in this instance.
michaelDCurran
requested changes
May 27, 2019
This allows for UIA console support to be dynamically enabled/disabled. Note: _UIAHandler.badUIAWindowClassNames has been removed. To check if a UIA windowClassName is bad, use the new _UIAHandler.UIAHandler._isBadUIAWindowClassName method.
…s with friendly version numbers, and only register for UIA textChange on Windows 10.
|
Could you please have another look @michaelDCurran ? |
michaelDCurran
requested changes
May 27, 2019
LeonarddeR
reviewed
May 27, 2019
There was a problem hiding this comment.
Arrg, really frustrating that GitHub calls every way to comment on an ongoing conversation for an inline comment that is outdated. I'm taking the discussion about the config spec to the main threat here.
Ah, I probably should've clarified a few things:
- Windows 10 has had UIA console support since version 1709, but it was incomplete in early builds (see Add ConsoleWindowClass to badUIAWindowClasses as the UIA implementation for win32 Console windows is not usable yet. #7497). The final PR will set a minimum good Windows version for UIA, which likely won't be 1709 (even though 1709 has limited support). That is to say, we won't turn on UIA console support automatically until a Windows version where we know it works completely, but we should give users the option of forcibly enabling it if our default is too aggressive.
I agree that this makes sense.
- To accomplish this, I proposed the provisional setting during development (and then adding the auto-detection and new setting once it's final), but maybe there's a better way?
As noted, a provisional setting requires a config spec and scheme update as soon as it gets changed. Especially if you already know what the final setting is going to look like, I'd prefer just adding the setting to the spec as it is supposed to be in the final pr. You could map UIA to True and legacy to False and just make auto act like False/legacy for now.
source/winVersion.py
Outdated
| try: | ||
| return winVersion.build >= win10VersionsToBuilds[version] | ||
| except KeyError: | ||
| log.warning("Unknown Windows 10 version {}".format(version)) |
There was a problem hiding this comment.
I'd rather see this raise an exception, but good to know opinions from others as well here.
source/winVersion.py
Outdated
| return winVersion.build >= win10VersionsToBuilds[version] | ||
| except KeyError: | ||
| log.warning("Unknown Windows 10 version {}".format(version)) | ||
| return False |
There was a problem hiding this comment.
Please add an empty line below this line.
michaelDCurran
approved these changes
May 27, 2019
This was referenced May 31, 2019
This was referenced Jun 5, 2019
|
@beqabeqa473 Fixed in #9647. |
This was referenced Jun 17, 2019
UI Automation in Windows Console: Remove "Text area", replace isAtLeastWin10, and code cleanup
#9761
Merged
Closed
This was referenced Jul 8, 2019
This was referenced Jul 16, 2019
This was referenced Jul 27, 2019
feerrenrut
pushed a commit
that referenced
this pull request
Jul 31, 2019
…9944) This fixes an issue in UIA consoles, invoking the "review start of line" script (`shift+numpad 1` by default) does not move the review cursor. Expanding and collapsing the textInfo is essential for the functionality of this script (to move the textInfo to the start of the line), we should ignore the _expandCollapseBeforeReview flag. The _expandCollapseBeforeReview flag was added in #9614. It is True everywhere except in UIA consoles.
feerrenrut
pushed a commit
that referenced
this pull request
Aug 1, 2019
Previously: NVDA failed to announce typed characters and/or words in Mintty, and spells output close to the caret in legacy Windows consoles. This commit factors out much of the code for handling typed characters in UIA consoles into a new `NVDAObjects.behaviors.TerminalWithoutTypedCharDetection class`. The class is now used in Mintty (PuTTY, Git Bash) and legacy Windows consoles on Windows 10 version 1607 and later. In legacy Windows consoles, the old keyboard handling code is disabled when the class is in use, and the new support can be disabled in the advanced preferences in case it is incompatible with some programs or if suppression of passwords is critical. Since legacy Windows consoles fire textChange rather slowly, this commit prefers faster responsiveness at the cost of offscreen characters (such as passwords) always being reported. Users may disable "speak typed characters" and/or "speak typed words" (using the existing scripts) when entering passwords to suppress this output. On Windows 10 version 1607 with the new keyboard support enabled, spurious characters are reported when the dead key (if available) is pressed. Fixes #513 Fixes #1348 Related to #9614
feerrenrut
pushed a commit
that referenced
this pull request
Aug 1, 2019
…e checks (PR #9957) Builds on #9614 Supersedes #9735 and #9899 Closes #9891 Previously, after the console window was maximized (or the review cursor is otherwise placed outside the visible text), text review is no longer functional. The review top line and review bottom line scripts do not function as intended. This commit changes: - The isOffscreen property has been implemented as UIAUtils.isTextRangeOffscreen. - When checking if the text range is out of bounds, we now also check that oldRange is in bounds before stopping movement. - Re-implemented POSITION_FIRST and POSITION_LAST in terms of visible ranges.
feerrenrut
pushed a commit
that referenced
this pull request
Aug 2, 2019
… 1803 and later (PR #9771) NVDA's UIA console support significantly improves performance and stability, so we should enable it by default. While UIA in consoles was added in Windows 10 1709, the initial implementation left much to be desired, particularly concerning caret movement and expansion to character. A new `UIAUtils.shouldUseUIAConsole` function has been added which determines whether to use UIA in consoles: * If the user has explicitly chosen to use UIA or legacy console support, the function returns `True` or `False` respectively. * Otherwise, if the user is running at least Windows 10 1803, the function returns `True`, or `False` if not. The "use UI Automation in the Windows Console when available" GUI setting has been changed to expose all options in the config spec (i.e. to automatically detect UIA/legacy or override this autodetection). builds on #9614 closes #673 closes #6291
feerrenrut
pushed a commit
that referenced
this pull request
Aug 12, 2019
Closes #10035 Related to #9614 Identical to #10043 Works around a UIA bug on Windows 10 1803 and later that means we can not trust the "end" endpoint of a collapsed (empty) text range for comparisons. The console incorrectly reports the "end" as being past the "start" endpoint. This stops braille being able to properly track the system caret in Windows Console. Instead use getText(1) on the textRange to attempt to fetch 1 character. getText returns an empty string for a collapsed range. By definition, the "start" and "end" endpoints for a collapsed range are equivalent, thus read from the "start" endpoint of a collapsed range instead of the "end" endpoint.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to issue number:
This PR shouldn't yet close any issues, but it is related to #513, #673, #1682, #6291, and #7497.
Summary of the issue:
NVDA provides support for the Windows command console used by Command Prompt, PowerShell, and the Windows Subsystem for Linux. It allows users to read the console's contents using review cursor commands, and automatically reports console output as it arrives. While functional, this support has some shortcomings, leading to decreased NVDA performance and stability and, by extension, user productivity.
Description of how this pull request fixes the issue:
This PR re-implements NVDA's Windows Console support using Microsoft's UI Automation API (UIA). At the moment, typed characters and words are read to the user as well as incoming text. Review of console output is functional on Windows 10 1803 and 1809. The new console support is implemented in the
NVDAObjects.UIA.winConsoleUIA.winConsoleUIAobject, and can be enabled with a new option in NVDA's advanced preferences.Since the UIA console interface was first implemented in Windows 10 version 1709, it is safe to use the
keyboardHandlermodule to receive typed characters in UIA consoles. The console sometimes sends duplicateWM_CHARmessages for typed characters, so the in-process approach makes "speak typed words" unreliable.On windows 10 version 1903 and later, the UIA caret position is off-by-one. To compensate,
consoleUIATextInfomoves new UIA text ranges around the caret one character to the right.A mapping from UIA's event for text changes to NVDA's
textChangeevent was present, but commented out, in_UIAHandler. This mapping has been uncommented to allow automatic readout of console text to be implemented.Automatic text readout uses the
NVDAObjects.behaviors.LiveTextfunctionality. Since UIA adds thousands of empty lines to console output, it was necessary to re-implement_getTextLinesto filter the output.UIA fires
textChangewhenever the console's text changes, even during user input. WhileLiveTextalready filters out many typed characters, some duplicates (particularly the beginning of new words) were being reported. To compensate,winConsoleUIAdoes not report any new text while the user is typing. This is not a regression of NVDA's functionality, as speech is already cancelled during user input, even when "speak typed characters" and "speak typed words" are disabled.Testing performed:
Tested the new support on Windows 10 versions 1803, 1903, and 2003. In particular, tested reporting of typed characters and words (when the corresponding settings were enabled), reporting of new text, text editing and selection, and object review.
Known issues with pull request:
winConsoleUIA._isTypingis cleared), reporting of new text and speak typed words may not function correctly._isTypingflag is not cleared on the console, so new text is not announced immediately.Change log entry:
== New features ==
== Changes for Developers ==
isAtLeastWin10function to thewinVersionmodule which returns whether or not this copy of NVDA is running on at least the supplied release version (such as 1809 or 1903) of Windows 10. (Preliminary support for UI Automation in Windows Console #9614)