Terminal crashes when exiting a bash shell

[j4james]

Windows Terminal version

1.15.1862.0

Windows build number

10.0.19044.1889

Other Software

WSL with Ubuntu 20.04.4 LTS

Steps to reproduce

1. Start Windows Terminal
2. Open two tabs, one of which is a bash shell.
3. Exit the bash shell (with exit).

Expected Behavior

The terminal should remain open after the bash shell exits, with the other tab still working.

Actual Behavior

The terminal crashes.

I was able to reproduce this in the debugger with a build derived from commit c12987a, and the stack trace for the crash looked like this:

ntdll.dll!TppRaiseInvalidParameter()	Unknown
ntdll.dll!TppTimerpValidateTimer()	Unknown
ntdll.dll!TpSetTimerEx()	Unknown
Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_terminalScrollPositionChanged(const int viewTop, const int viewHeight, const int bufferSize) Line 1308	C++
[External Code]	
Microsoft.Terminal.Control.dll!Microsoft::Terminal::Core::Terminal::_NotifyScrollEvent() Line 1305	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::_EraseScrollback() Line 1952	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::AdaptDispatch::EraseInDisplay(const Microsoft::Console::VirtualTerminal::DispatchTypes::EraseType eraseType) Line 614	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::OutputStateMachineEngine::ActionCsiDispatch(const Microsoft::Console::VirtualTerminal::VTID id, const Microsoft::Console::VirtualTerminal::VTParameters parameters) Line 492	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_SafeExecute<`Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch'::`2'::<lambda_1>>(Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch::__l2::<lambda_1> && lambda) Line 2039	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::_ActionCsiDispatch(const wchar_t wch) Line 480	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessCharacter(const wchar_t wch) Line 1778	C++
Microsoft.Terminal.Control.dll!Microsoft::Console::VirtualTerminal::StateMachine::ProcessString(const std::basic_string_view<wchar_t,std::char_traits<wchar_t>> string) Line 1853	C++
Microsoft.Terminal.Control.dll!winrt::Microsoft::Terminal::Control::implementation::ControlCore::_connectionOutputHandler(const winrt::hstring & hstr) Line 1706	C++
Microsoft.Terminal.Control.dll!winrt::impl::delegate<winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler,`winrt::Microsoft::Terminal::TerminalConnection::TerminalOutputHandler::implementation<winrt::Microsoft::Terminal::Control::implementation::ControlCore,void (__cdecl winrt::Microsoft::Terminal::Control::implementation::ControlCore::*)(winrt::hstring const &)>'::`1'::<lambda_218_>>::Invoke(void * output) Line 174	C++
[External Code]	

The problem, as far as I can make out, is that the ControlCore instance is deleted at this point in time. If you look at ControlCore::Close method, we're closing the connection asynchronously, which I think means we're still receiving VT sequences (in this case \e[3J) after ControlCore is gone. Assuming I've understood that correctly, that would probably explain the crash.

[j4james]

This might be a duplicate of #12068. Possibly also #13337 (they initially said they weren't doing anything when the terminal crashed, but later crashes occurred when "attempting to exit restored tabs" which sounds similar to what I'm seeing).

I'm surprised we don't have had more reports like this, though, because it's really easy for me to produce, but I guess there's an element of timing involved. I've also only noticed this recently, so I'm wondering if there's something we've changed that has made it easier to trigger.

[lhecker]

This commit seems to be related (although I don't immediately see why that would be): 23e4d31

It tries to work around a deadlock issue by calling UpdatePatternLocations from a background thread. Even if I don't see how it might cause this, it does seem to be related because:

• It's a recent change (included in c12987a)
• It's in ControlCore
• _updatePatternLocations is called in _terminalScrollPositionChanged which calls TpSetTimerEx

This PR is my attempt to fix one bug caused by that commit: #13859
But yours seems to be another: The TppRaiseInvalidParameter call seems like an issue that might be caused if the til::throttled_func has been destroyed, making the wil::unique_threadpool_timer handle invalid. But why would this have worked before? The only difference I can think of is that the previous ThrottledFunc uses a shared_ptr to keep itself alive until the scheduled callback returns, but I don't see how that would be related here since no such ThrottledFunc is involved in _connectionOutputHandler. Did we just coincidentally get lucky up until now?

In either case I believe the underlying issue is this:

terminal/src/cascadia/TerminalControl/ControlCore.cpp

Lines 99 to 100 in a85d9e6

	// This event is explicitly revoked in the destructor: does not need weak_ref
	_connectionOutputEventToken = _connection.TerminalOutput({ this, &ControlCore::_connectionOutputHandler });

The comment is wrong. Just because an event has been revoked, doesn't mean it's not currently being called by another thread, right? TerminalOutput is called from a background thread and so the callback's execution might outlive the main thread's strong reference to the ControlCore instance. The fix can't be to just use get_weak() instead of this, because ControlCore is WinUI-land and thus inherently thread-unsafe and can't be destroyed on a background thread. If I'm correct we have to block in ~ControlCore until the ITerminalConnection can promise that all calls to TerminalOutput have returned.

[j4james]

This commit seems to be related (although I don't immediately see why that would be): 23e4d31

I don't think that's to blame, because I'm also getting this crash in preview release 1.15.1862.0, which I think predates that commit.

Did we just coincidentally get lucky up until now?

I was wondering that too. I don't fully understand what's going on in that code, but my initial impression was that it shouldn't ever have worked.

If I'm correct we have to block in ~ControlCore until the ITerminalConnection can promise that all calls to TerminalOutput have returned.

Yeah, I was thinking the same thing.

[ghost]

🎉This issue was addressed in #13882, which has now been successfully released as Windows Terminal Preview v1.16.252.:tada:

Handy links:

• Release Notes
• Store Download

[ghost]

🎉This issue was addressed in #13882, which has now been successfully released as Windows Terminal v1.15.2874.:tada:

Handy links:

• Release Notes
• Store Download