Use auto select detection for UIA when listening for TextChanged and TextSelectionChanged events

[LeonarddeR]

Link to issue number:

Fixes #9749
Fixes #7554

Summary of the issue:

1. When editing in Excel, braille doesn't always follow correctly. This also happens in other UIA text fields, such as the search field in Win 10 settings.

2. Suggestions aren't read in the non-chromium Microsoft Edge address bar.

As of pr #9614, on Windows 10, UIA TextChanged events are once again received by NVDA. They were once disabled in 9025c04. As part of this commit, we switched from the NVDAObjects.behaviors.EditableTextWithAutoSelectDetection ovrlay to the EditableTextWithoutAutoSelectDetection overlay for UIA. This was a necessary step, as text change and selection change events were no longer received from UIA.

Description of how this pull request fixes the issue:

We are already listening for TextChange events. This pr also requests TextSelectionChange events as before 9025c04. A, but only for Windows 10. When on Windows 10, UIA text controls now use the EditableTextWithAutoSelectDetection overlay.

Testing performed:

1. Tested that in UIA text controls, such as in the Windows start menu and settings app, selection is reported as expected. Value changes are ignored correctly.
2. Tested selection support in Word with UIA enabled.3. Tested that braille follows much more accurately when editing Excel cell contents.
3. According to @josephsl, suggestions in the Edge address bar are now handle correctly.

Known issues with pull request:

None

Change log entry:

It is probably a bit difficult to come up with an entry that is understandable by end users.

• Changes

  • In UIA text controls on Windows 10, NVDA once relies on the UIA TextSelectionChange event to report selection changes, rather than detecting them manually.

• Bug fixes

  • On Windows 10, braille will now follow when editing cell contents in Microsoft Excel and in other UIA text controls were it was lagging behind. (No braille feedback in Excel when editing directly in cell #9749)
  • NVDA will once again report suggestions in the Microsoft Edge address bar. (Nvda is not reading the site suggestions in the address field in Microsoft Edge #7554)

[josephsl]

Hi,

One additional bug fix: address omnibar in EdgeHTML-based Edge is announced correctly if there are suggestions.

Two things observed when testing it with Edge:

1. In GitHub edit fields, typed text is announced despite my setting to turn typed characters/words off.
2. On some websites (such as Google), selection isn't announced if I use focus mode inside an edit field and attempt to select text.

Tested with PR-9660 binary build from Appveyor.

Thanks.

[LeonarddeR]

Thanks for pointing me at these, both should be fixed by now. I need to do some additional testing in Edge in the without autoselect detection. What bug were you referring to about the omnibar?

[josephsl]

Hi, until now omnibar content wasn’t announced at all – I wrote a workaround in my add-on which someone helped test and was found to be working. I used similar principle in there, but this PR should do the trick too. I’ll do some more tests with Excel and Outlook in just a moment. Thanks.

[josephsl]

Hi,

EdgeHTML-based Edge issues are fixed. It breaks in Excel, however:

1. Open Excel.
2. When a new sheet opens, press F2 to edit a cell.
3. Locate the just edited cell and press F2 to edit it again.
4. Try selecting cell contents by using Shift+arrow keys.

Expected: selected character/word/line, etc. are announced.
Actual: no announcement.

Tested on Excel 365.
No problem with Outlook at the moment.

Thanks.

[LeonarddeR]

Thanks for reporting again, could be related to my last pr disabling several additional win events for UIA. I will investigate further next week.

[codeofdusk]

Test notes for UIA consoles:

Windows 10 1803:
No noticeable changes.

Windows 10 1903:

• Performance of selection change reporting has been significantly improved.
• When editing text, NVDA repeatedly announces "selected" and "unselected". If it's the console's UIA implementation that's misbehaving here (rather than a bug in this PR) I think I can work around it.

[LeonarddeR]

@josephsl: I couldn't reproduce your issue with Excel with Excel 2016. I tested from source, commenting out the version check from _UIAHandler. Made sure that WithAutoSelectDetection is on the dynamic class of the edit box. Both the formulla bar and the in cell edit field work as expected.

[LeonarddeR]

@codeofdusk: I've only looked quickly at this, but it looks like the autoselect detection code triggers on this because even though nothing is selected and the text range seems to be collapsed, isCollapsed on the selection returns False.

[LeonarddeR]

@codeofdusk: No matter what I do, collapsing a text info for UIA consoles on Windows 10 1903 keeps returning False for isCollapsed. Really looks like there's something severely broken in there.

[LeonarddeR]

@leonardder commented on 3 Jun 2019, 07:20 CEST:

@josephsl: I couldn't reproduce your issue with Excel with Excel 2016. I tested from source, commenting out the version check from _UIAHandler. Made sure that WithAutoSelectDetection is on the dynamic class of the edit box. Both the formulla bar and the in cell edit field work as expected.

Could reproduce it in Excel 365. The reason is that the edit window is of window class Excel7, and that class is in badUIAWindowClassNames, so events for that class are forcefully ignored.

[josephsl]

Hi,

Excel: works. Thanks.

[LeonarddeR]

@michaelDCurran: I really would like to know what you think about this.
Note that issues regarding command consoles are still there, but that UIA implementation in 1903 realy has its brokenness.

[michaelDCurran]

There are definitely issues with the way Microsoft has implemented UIA textRanges for consoles. Specifically:

• In many cases, fetching the range for the caret results in a textRange one character previous to what it should be. Bill has worked around this bug I think.
• When fetching the selection from a UIA text pattern when there is no selection (only a caret), the resulting textRange is not collapsed. I think it is expanded to one character. This is a bug which we really can't work around. I am in the process of getting this reported to Microsoft.
  I think all we can do for now is for consoles, we should still use editableTextWithoutAutoSelectDetection. I.e. check for autoSelectDetection class in findOverlayCasses and remove it, and make UIAWinConsole inherit from editableTextWithoutAutoSelectDetection.
  I do understand the advantages of using autoSelectDetection, but could you clearly define from a users' point of view how this pr is going to be bennoficial?

[LeonarddeR]

  I do understand the advantages of using autoSelectDetection, but could you clearly define from a users' point of view how this pr is going to be bennoficial?

I don't think that there are much benefits, apart from selection being reported when performed with the mouse There is also #9660 (comment) where @codeofdusk noted massive performance changes, but it looks like we have to disregard these.

[LeonarddeR]

@feerrenrut: Just wanted to point out that from a user's perspective, I think this one is important to have in 2019.3. It will also benefit from a long testing period.

[LeonarddeR]

@codeofdusk: how is this now behaving with regard to your console work?

[josephsl]

I think it'd be best to hold off until 2019.3 due to vast changes needed. I did run into editable field issues in Start menu and such, but no need to worry, as these problems are seen with Windows 10 App Essentials enabled (I'll write a fix for it from my add-on once this PR comes to NVDA Core proper). Thanks.

[codeofdusk]

@codeofdusk: how is this now behaving with regard to your console work?

Initial testing looks quite promising (no longer says "selected" repeatedly when typing on Windows 10 1903)!

[LeonarddeR]

@feerrenrut; This pr fixes issues related to braille and UIA. It also needs some wider testing before going into an official release. Therefore if you could give this some attention in the near future that would be awesome!

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit dd9ed2dd7c

[LeonarddeR]

@feerrenrut: The linter raises the following warning for _UIAHandler:

2     F405 'UIA_Text_TextChangedEventId' may be undefined, or defined from star imports: comtypes, comtypes.gen.UIAutomationClient, ctypes, ctypes.wintypes

I can explicitly import it, but I'm afraid this will make _UIAHandler pretty messy in the end. Therefore I have the feeling that we should exclude this warning. For new modules, warning F403 will still trigger.

[LeonarddeR]

There is also the following issue for the following code:

	UIAEventIdsToNVDAEventNames.update({
		UIA_Text_TextChangedEventId: "textChange",
		UIA_Text_TextSelectionChangedEventId: "caret",
		})

If the line has 2 tabs at the start, the warning is: 1 ET126 (flake8-tabs) unexpected number of tabs at start of expression line (expected 1, got 2). However, if I remove one tab, we get the under indented for hanging indent warning. I have the feeling that flake8-tabs fails here.

[AppVeyorBot]

PR introduces Flake8 errors 😲

See test results for Failed build of commit 855139b269

[feerrenrut]

Error F405 I believe is quite valuable.

• F403: If we wish to import * so that clients of this module / package are exposed to everything in some other module then we can leave that and put a #noqa comment to explain why. Ideally this is rare.
• F405: The functions from an import * that we use in the current module should be imported by name. This can be in addition to those the import * line.

I'm looking into the indentation error.

[feerrenrut]

I'm not sure what is causing E121, note that this should be emitted, E121 is supposed to be replaced by ET121 from Flake8-tabs. I can't reproduce it with a minimal test file. So that the closing brace does not need to be moved when adding a new item, to make it easier to see that the brackets match, and indentation levels are correct, I would prefer the following style:

	UIAEventIdsToNVDAEventNames.update({
		UIA_Text_TextChangedEventId: "textChange",
		UIA_Text_TextSelectionChangedEventId: "caret",
	})

[feerrenrut]

Ah, I think I made a mistake with the config. I don't think we want use-pycodestyle-indent = True

[feerrenrut]

Or maybe we do, but need to ignore E121 and any others similar....

[LeonarddeR]

What is it supposed to do?

[feerrenrut]

flake8-tabs replaces many, but not all of pycodestyle indent checks. When you enable flake8-tabs it disables the pycodestyle indent checks, this allows you to re-enable them. However, then some of them are contradictory to the flake8-tabs checks and should be manually disabled. I think we want some of them, but not all. I am looking at them now, and will update #10020 when I work out a good combination.

[josephsl]

To @LeonarddeR: any plans for dealing with linting?

Thanks.

[codeofdusk]

To @LeonarddeR: any plans for dealing with linting?

Thanks.

You might also wish to test how the latest changes affect UIA consoles and if selection is reported more consistently.

[LeonarddeR]

I somehow thought that this was already merged. Should have been a dream.

[LeonarddeR]

I fixed linting issues by applying a suggestion from @michaelDCurran from #10110 (comment)