Swift Target Crashes with Multi-Threading

[rmehta33]

• I am not submitting a question on how to use ANTLR; instead, go to antlr4-discussion google group or ask at stackoverflow
• I have done a search of the existing issues to make sure I'm not sending in a duplicate

Reproducing Error

Running via Xcode for iOS 13
Project Swift Version: 4.2
Compiled swift target from 4.9.2 release

Grammar

grammar math;

NUMBER : [0-9] ;
WS : [ \r\n\t] + -> skip ;

operation : l=NUMBER op='+' r=NUMBER  ;

Swift Code

for _ in 0...10 {
    DispatchQueue.global().async {
        let chars = ANTLRInputStream("1+1")
        let lexer = mathLexer(chars)
        
        let tokenStream = CommonTokenStream(lexer)
        let parser = try! mathParser(tokenStream)
        
        let node = try? parser.operation()
    }
}

What Happened

The program crashes in LexerATNSimulator.swift on line 721 with EXC_BAD_ACCESS. There was a double free at some point in the stack.

Any help is greatly appreciated! Hopefully this is just user error.

[martinvw]

Hello @rmehta33, I fear that I'm bumping in the same issue, but I did not yet have a reproduction path, does you Backtrace look a bit like this:

0  CoreFoundation               	0x18a642708 __exceptionPreprocess + 220
1   libobjc.A.dylib               	0x19f14c7a8 objc_exception_throw + 59
2   CoreFoundation                	0x18a545bfc -[NSObject+ 183292 (NSObject) doesNotRecognizeSelector:] + 143
3   CoreFoundation                	0x18a645260 ___forwarding___ + 1443
4   CoreFoundation                	0x18a647560 _CF_forwarding_prep_0 + 95
5   libswiftCore.dylib            	0x18e26aa8c __CocoaDictionary.lookup+ 789132 (_:) + 35
6   *redacted*                  	0x10e204dd0 specialized Dictionary.subscript.getter + 79
7   *redacted*                  	0x10e1aecac closure #1 in LexerATNSimulator.addDFAState+ 437420 (_:) + 91
8   *redacted*                  	0x10e1ac26c LexerATNSimulator.addDFAState+ 426604 (_:) + 767
9   *redacted*                  	0x10e1abb70 LexerATNSimulator.matchATN+ 424816 (_:) + 187
10  *redacted*                  	0x10e1ab754 LexerATNSimulator.match+ 423764 (_:_:) + 359
11  *redacted*                  	0x10e2007fc Lexer.nextToken+ 772092 () + 795

If so that would be awesome I'm already looking at this for days without a reproduction path, I did not think I was running multiple threads but there might also be a small issue there. Thanks for your input.

[martinvw]

Thinking out loud I think I see (part of) the problem.

1. the decisionToDFA field inside LexerATNSimulator is private

antlr4/runtime/Swift/Sources/Antlr4/atn/LexerATNSimulator.swift

Lines 71 to 73 in 5e5b6d3

	public private(set) final var decisionToDFA: [DFA]

2. But not constructed internally but received

antlr4/runtime/Swift/Sources/Antlr4/atn/LexerATNSimulator.swift

Lines 99 to 106 in 5e5b6d3

	public init(_ recog: Lexer?, _ atn: ATN,
	_ decisionToDFA: [DFA],
	_ sharedContextCache: PredictionContextCache) {
	self.decisionToDFA = decisionToDFA
	self.recog = recog
	super.init(atn, sharedContextCache)
	}

3. from the generated parser

antlr4/tool/resources/org/antlr/v4/tool/templates/codegen/Swift/Swift.stg

Lines 357 to 362 in 5e5b6d3

	override <accessLevelNotOpen(parser)>
	init(_ input:TokenStream) throws {
	RuntimeMetaData.checkVersion("4.9.2", RuntimeMetaData.VERSION)
	try super.init(input)
	_interp = ParserATNSimulator(self,<p.name>._ATN,<p.name>._decisionToDFA, <parser.name>._sharedContextCache)
	}

4. In that class, it is a static field

antlr4/tool/resources/org/antlr/v4/tool/templates/codegen/Swift/Swift.stg

Lines 238 to 246 in 5e5b6d3

	internal static var _decisionToDFA: [DFA] = {
	var decisionToDFA = [DFA]()
	let length = <parser.name>._ATN.getNumberOfDecisions()
	for i in 0..\<length {
	<!// decisionToDFA[i] = DFA(<parser.name>._ATN.getDecisionState(i)!, i);!>
	decisionToDFA.append(DFA(<parser.name>._ATN.getDecisionState(i)!, i))
	}
	return decisionToDFA
	}()

5. However the Mutex guarding access

antlr4/runtime/Swift/Sources/Antlr4/atn/LexerATNSimulator.swift

Lines 712 to 723 in 5e5b6d3

	return dfaStatesMutex.synchronized {
	if let existing = dfa.states[proposed] {
	return existing
	}
	let newState = proposed
	newState.stateNumber = dfa.states.count
	configs.setReadonly(true)
	newState.configs = configs
	dfa.states[newState] = newState
	return newState
	}

6. Is defined on class level:

antlr4/runtime/Swift/Sources/Antlr4/atn/LexerATNSimulator.swift

Lines 76 to 84 in 5e5b6d3

	///
	/// mutex for DFAState change
	///
	private let dfaStateMutex = Mutex()
	///
	/// mutex for changes to all DFAStates map
	///
	private let dfaStatesMutex = Mutex()

I do believe that this is likely to be the cause of the issue, but please correct me if I'm wrong!

@ewanmellor, @ericvergnaud, and/or other (Swift) maintainers your opinion is greatly appreciated

Edit ----

I did a quick comparison with Java which has comparable code, however, the synchronized works there on top of the object reference so similar problem do not occur there, I fear we have to pass in the mutex-es externally as well.

[rmehta33]

Hello @rmehta33, I fear that I'm bumping in the same issue, but I did not yet have a reproduction path, does you Backtrace look a bit like this:

0  CoreFoundation               	0x18a642708 __exceptionPreprocess + 220
1   libobjc.A.dylib               	0x19f14c7a8 objc_exception_throw + 59
2   CoreFoundation                	0x18a545bfc -[NSObject+ 183292 (NSObject) doesNotRecognizeSelector:] + 143
3   CoreFoundation                	0x18a645260 ___forwarding___ + 1443
4   CoreFoundation                	0x18a647560 _CF_forwarding_prep_0 + 95
5   libswiftCore.dylib            	0x18e26aa8c __CocoaDictionary.lookup+ 789132 (_:) + 35
6   *redacted*                  	0x10e204dd0 specialized Dictionary.subscript.getter + 79
7   *redacted*                  	0x10e1aecac closure #1 in LexerATNSimulator.addDFAState+ 437420 (_:) + 91
8   *redacted*                  	0x10e1ac26c LexerATNSimulator.addDFAState+ 426604 (_:) + 767
9   *redacted*                  	0x10e1abb70 LexerATNSimulator.matchATN+ 424816 (_:) + 187
10  *redacted*                  	0x10e1ab754 LexerATNSimulator.match+ 423764 (_:_:) + 359
11  *redacted*                  	0x10e2007fc Lexer.nextToken+ 772092 () + 795

If so that would be awesome I'm already looking at this for days without a reproduction path, I did not think I was running multiple threads but there might also be a small issue there. Thanks for your input.

Yes!

[martinvw]

@rmehta33 I created a PR: #3276