Odd crash when deallocating OC_PythonArray

[ronaldoussoren]

Original report by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

I have an odd crash that only happens occasionally on exit.

Unfortunately I cannot post the code to reproduce it right now, but what I do is basically store a python list in NSUserDefaults by using the setObject_forKey_ method like this:

#!python

self._user_defaults = objc.lookUpClass('NSUserDefaults').standardUserDefaults()

# ...

self._user_defaults.setObject_forKey_([str(v) for v in value], name)

I only have pyobjc-core installed.

Here is the crash report.

#!

Process:               Python [2033]
Path:                  /usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.4.2 (3.4.2)
Code Type:             X86-64 (Native)
Parent Process:        zsh [722]
Responsible:           Terminal [570]
User ID:               501

Date/Time:             2015-07-16 18:36:45.573 -0400
OS Version:            Mac OS X 10.10.4 (14E46)
Report Version:        11
Anonymous UUID:        319B57F5-91FC-214A-1DF0-430F5EACDFA4

Sleep/Wake UUID:       B243AE87-DEA6-4EDB-B021-6D08CA8E8FCA

Time Awake Since Boot: 1100000 seconds
Time Since Wake:       360000 seconds

Crashed Thread:        2  Dispatch queue: com.apple.root.default-qos

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Application Specific Information:
abort() called
/SourceCache/libpthread/libpthread-105.10.1/src/pthread.c:pthread_exit:1316: pthread_exit() may only be called against threads created via pthread_create()

Thread 0:: Dispatch queue: com.apple.main-thread
0   org.python.python             	0x000000010ac49963 tupledealloc + 106
1   org.python.python             	0x000000010ac1be36 code_dealloc + 93
2   org.python.python             	0x000000010ac298c8 func_dealloc + 91
3   org.python.python             	0x000000010ac35e71 insertdict + 405
4   org.python.python             	0x000000010ac3ef44 _PyModule_ClearDict + 393
5   org.python.python             	0x000000010acb3902 PyImport_Cleanup + 834
6   org.python.python             	0x000000010acbe524 Py_Finalize + 95
7   org.python.python             	0x000000010acd2615 Py_Main + 3393
8   org.python.python             	0x000000010abf6e27 0x10abf5000 + 7719
9   libdyld.dylib                 	0x00007fff932075c9 start + 1

Thread 1:: Dispatch queue: com.apple.libdispatch-manager
0   libsystem_kernel.dylib        	0x00007fff85be6232 kevent64 + 10
1   libdispatch.dylib             	0x00007fff93225a6a _dispatch_mgr_thread + 52

Thread 2 Crashed:: Dispatch queue: com.apple.root.default-qos
0   libsystem_kernel.dylib        	0x00007fff85be4c82 __kill + 10
1   ???                           	0x0000000000000001 0 + 1
2   libsystem_c.dylib             	0x00007fff92373b53 abort + 129
3   libsystem_pthread.dylib       	0x00007fff89762cef __pthread_abort + 49
4   libsystem_pthread.dylib       	0x00007fff89762da3 __pthread_abort_reason + 180
5   libsystem_pthread.dylib       	0x00007fff897603dc pthread_exit + 63
6   org.python.python             	0x000000010acd00f5 PyThread_exit_thread + 21
7   org.python.python             	0x000000010ac9aced PyEval_RestoreThread + 75
8   org.python.python             	0x000000010acbd531 PyGILState_Ensure + 89
9   _objc.so                      	0x000000010c092da3 -[OC_PythonArray release] + 19
10  com.apple.CoreFoundation      	0x00007fff93ae2db0 CFRelease + 304
11  com.apple.CoreFoundation      	0x00007fff93c8e4b6 __67-[CFPrefsPlistSource sendFullyPreparedMessage:settingValue:forKey:]_block_invoke_2 + 166
12  libxpc.dylib                  	0x00007fff8a8dd52c _xpc_connection_reply_callout + 47
13  libxpc.dylib                  	0x00007fff8a8dd4b8 _xpc_connection_call_reply + 36
14  libdispatch.dylib             	0x00007fff93222c13 _dispatch_client_callout + 8
15  libdispatch.dylib             	0x00007fff9322588f _dispatch_root_queue_drain + 935
16  libdispatch.dylib             	0x00007fff93233fe4 _dispatch_worker_thread3 + 91
17  libsystem_pthread.dylib       	0x00007fff8975f637 _pthread_wqthread + 729
18  libsystem_pthread.dylib       	0x00007fff8975d40d start_wqthread + 13

Thread 3:
0   libsystem_kernel.dylib        	0x00007fff85be594a __workq_kernreturn + 10
1   libsystem_pthread.dylib       	0x00007fff8975d40d start_wqthread + 13

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

Worth to mention that I have neither explicit NSAutoreleasePool nor run loop.

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

I updated the code to the following:

#!python

self._user_defaults = objc.lookUpClass('NSUserDefaults').standardUserDefaults()
self._nsarray_class = objc.lookUpClass('NSArray')

# ...

with objc.autorelease_pool():
    self._user_defaults.setObject_forKey_(self._nsarray_class.arrayWithArray_([str(v) for v in value]), name)

But it still crashes, with slightly different exception:

#!

Thread 2 Crashed:: Dispatch queue: com.apple.root.default-qos
0   libsystem_kernel.dylib        	0x00007fff85be4c82 __kill + 10
1   ???                           	0x00007ff14c587be1 0 + 140674344713185
2   libsystem_c.dylib             	0x00007fff92373b53 abort + 129
3   libsystem_pthread.dylib       	0x00007fff89762cef __pthread_abort + 49
4   libsystem_pthread.dylib       	0x00007fff89762da3 __pthread_abort_reason + 180
5   libsystem_pthread.dylib       	0x00007fff897603dc pthread_exit + 63
6   org.python.python             	0x00000001001690f5 PyThread_exit_thread + 21
7   org.python.python             	0x0000000100133ced PyEval_RestoreThread + 75
8   org.python.python             	0x0000000100156531 PyGILState_Ensure + 89
9   _objc.so                      	0x0000000101535913 -[OC_PythonUnicode release] + 19
10  com.apple.CoreFoundation      	0x00007fff93ae2db0 CFRelease + 304
11  com.apple.CoreFoundation      	0x00007fff93b03bbd -[__NSArrayI dealloc] + 125
12  libobjc.A.dylib               	0x00007fff8958289c objc_object::sidetable_release(bool) + 236
13  com.apple.CoreFoundation      	0x00007fff93ae2db0 CFRelease + 304
14  com.apple.CoreFoundation      	0x00007fff93c8e4b6 __67-[CFPrefsPlistSource sendFullyPreparedMessage:settingValue:forKey:]_block_invoke_2 + 166
15  libxpc.dylib                  	0x00007fff8a8dd52c _xpc_connection_reply_callout + 47
16  libxpc.dylib                  	0x00007fff8a8dd4b8 _xpc_connection_call_reply + 36
17  libdispatch.dylib             	0x00007fff93222c13 _dispatch_client_callout + 8
18  libdispatch.dylib             	0x00007fff9322588f _dispatch_root_queue_drain + 935
19  libdispatch.dylib             	0x00007fff93233fe4 _dispatch_worker_thread3 + 91
20  libsystem_pthread.dylib       	0x00007fff8975f637 _pthread_wqthread + 729
21  libsystem_pthread.dylib       	0x00007fff8975d40d start_wqthread + 13

I'm going to try to use NSString instead of OC_PythonUnicode.

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

The final versions seems to work, but since bug is occasional it's hard to be sure:

#!python

self._user_defaults = objc.lookUpClass('NSUserDefaults').standardUserDefaults()
self._nsarray_class = objc.lookUpClass('NSArray')

# ...

self._user_defaults.setObject_forKey_(self._nsarray_class.arrayWithArray_([self._nsstring_class.stringWithString_(str(v)) for v in value]), self._nsstring_class.stringWithString_(name))

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Not sure what's going on here. The first crash log seems to indicate that this happens during/after interpreter shutdown (see PyEval_RestoreThread in Python/ceval.c).

What I think happens is that the .app calls Py_Finalize and after that either the main thread or some other thread cleans up Cocoa and that calls back into Python again. That fails because the Python interpreter is already gone.

BTW. Which version of PyObjC do you use? And how do you build the application bundle? Using py2app or something else?

I have to think about this a bit more, but think I'll work around this in py2app by not calling Py_Finalize. That's not ideal (because the usual orderly shutdown for Python objects is not run), but better than the alternative.

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

I'm using pyobjc-core 3.0.4, bundle is created by using pyqtdeploy. The code responsible for starting (and finishing) python interpreter can be found here: https://github.com/GreatFruitOmsk/pyqtdeploy/blob/master/pyqtdeploy/builder/lib/pyqtdeploy_start.cpp

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Pyqtdeploy does the same as py2app: it calls py_finalize at the end, and the orderly shutdown of cocoa can happen after that call.

I'm not sure if there is a way to avoid crashes here. For Python 3 it might be possible to invalidate all proxy objects, but that will likely have other unwanted side effects (such as silently erasing a user setting instead of crashing).

--
On the road, hence brief.

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

If it's improper way to finish, I can alter it, but I do not know how.

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

I think I can fix this particular crash on the PyObjC side by adding more code to -[OC_Python* release] and dealloc, those basically need to check if the interpreter is initialised before calling into Python. That will have a small performance impact, but that cannot be helped.

I don't know yet how to fix this in general though. There is a race condition between shutdown of Python and cleanup of Cocoa. I can work around this in the management of reference counts, but not when Cocoa tries to call into Python code that does something more substantial.

I have a (partial) workaround in changeset 68804363ebf4 (not yet pushed to the repository)

[ronaldoussoren]

Original comment by Kentzo (Bitbucket: Kentzo, GitHub: Kentzo).

---

@ronaldoussoren Maybe Python should finalize after Cocoa's run loop dies?

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

It already does, but not before all other threads have died. What seems to happen in the first crash dump is that Py_Finalize is working on cleaning up the interpreter (and has already marked it as being no longer valid), while another thread (Thread 2) is still doing some work with a OC_PythonArray object (reducing the refcount).

The only way you can be sure to not have crashes in this case is to remove the call to Py_Finalize in pyqtdeploy_start.cpp. That has an unwanted side effect though: the interpreter is not cleanly shut down and that means the destructors of objects aren't run. One possible result of that is that open files aren't closed cleanly, in particularly buffers aren't flushed (more so in Python 3 than in Python 2, in the latter the buffers are in stdio and that will still clean up after itself).

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Workaround for a race condition between interpreter shutdown and Cocoa cleanup when exiting an application.

See issue #124

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Removing version: 3.0 (automated comment)

[ronaldoussoren]

Closing this issue because there is a workaround for this in PyObjC and there's nothing I can do beyond that.