Mac OS bundle are rejected by Apple's notarization process

[darad]

We create a bundle of our python project using the bdist_mac command from cx_freeze. It seems that Apple now requires bundles to be notarized to be able to pass the Gatekeeper on target machine.

Our app is currently being rejected by the notarisation process seemingly because cx-Freeze places all the files (ie executables, libraries, scripts, data) in the MacOS folder. That by itself violates many of rules and recommendations by Apple's codesigning guide. For example, all the data and script files need to be in Contents/Resources and although codesign is able to sign the bundle, running the verification (ie codesign -vvv --strict /path/to/bundle) fail due to existence of data/script files in code locations.

We've been trying to implement a post-cx-freeze step in our build process that tries to restructure the bundle to conform to requirements by the notarisation process. However, the implementation is very hacky and error-prone to future changes of libraries and python version.

It sounds like a problem that needs to be fixed at cx-Freeze's level of operations.

[marcelotduarte]

Hi @darad,
Can you contribute with patches or with your post cx-freeze step to me or other adapt to cx-freeze?

[darad]

Hi @marcelotduarte

Here you can find the implementation of our wrapper command around bdist_mac. I have modified the code slightly to remove some company-specific information (mostly variable naming). You might wonder what the FileManager module contains (ie from customapi import FileManager as FM). It's mainly a wrapper around python's os module for convenient purposes.

We are currently using this command to restructure the cx-frozen bundle to make it notarizable by Apple.

Feel free to contact me for any questions regarding the gist. Hope it helps.

[marcelotduarte]

Thanks!

[TechnicalPirate]

Hi @marcelotduarte

I have added an updated Gist for you here the main change from @darad's previous gist is that with > cx_Freeze 6.4 we need to handle Qt framework paths slightly differently

Please let us know if you have questions re: the Gist / Apples requirements .

Equally we'd be very happy to help test new versions of cx_freeze as we have a cross-platform product & supporting CI all ready to go.

[TechnicalPirate]

Hi @marcelotduarte

Just a head's up im picking up this issue now - We're updating our whole dependency chain so i'm investigating the output of cx_freeze since we developed the workaround linked above ( Which is now slightly stale with other cx_freeze changes ),

I'll keep you posted with progress - For now I've been running into other reported issues.

Right now I'm encountering something like this issue : #1511 with cx_freeze 6.15.4.

[2023-07-27T13:11:59.694Z] copying /usr/local/opt/libiodbc/lib/libiodbc.2.dylib -> /Volumes/jenkins/workspace/nvestigate_osx_bundle_structures/buildandassemble/samples/PySide2/build/exe.macosx-10.14-x86_64-3.9/lib/libiodbc.2.dylib

[2023-07-27T13:11:59.694Z] error: [Errno 2] No such file or directory: '/usr/local/opt/libiodbc/lib/libiodbc.2.dylib'

Was the issue closed due to resolution or lack of response from the author? :)

( I'm currently trying to understand why its picking up libs from this location / whats changed dependency wise for why its now being included )

[marcelotduarte]

Hi @TechnicalPirate
The issue was closed due to a lack of response and because the initial question differed from the other problem being addressed.
But issue #1671 is still open.

I'm doing a patch for v6.16, which will reevaluate some dependencies, mainly Qt, so you should work with that in mind. When I publish it, I'll let you know.
I usually do it for Linux, but testing on other systems via CI.

[TechnicalPirate]

Just minor comment to aggregate some notes that may be beneficial to resolving this issue

Already reported issues with code sign (Which in turn will cause notarization fails)

• MacOS codesign fails in tcltk subcomponent: 'bundle format unrecognized, invalid, or unsuitable' #1671
• Code-sign breaks on macOS with python 3.11 #1861

On the notarization front - above a user linked a issue from onionshare - looking at their resolution it appears they are monkey patching the DistributionCache... might warrant closer inspection once we clear code-sign issues :)

[TechnicalPirate]

Hi @marcelotduarte resuming conversation from the other issue.

Problem Statement: .apps generated via bdist_mac have an invalid bundle structure

To quote apples docs Placing Content in a Bundle

Important

If you put content in the wrong location, you may encounter hard-to-debug code signing and distribution problems. These problems aren’t always immediately obvious. For example, when building a Mac app, incorrectly placed code might work during day-to-day development, but might cause problems during notarization.

This i believe explains most of the reports cx_freeze gets re: signing / notarization issues.

Looking at the output of cx_freeze-6.15.5 the main issue is the putting of the lib folder inside {appname}.app/Contents/MacOS

Skimming the docs - the expected structure is:

• {appname}.app/Contents/MacOS should just contain the app binary
• {appname}.app/Contents/Frameworks should contain all the .dylib and .so files
• {appname}.app/Contents/Resources should contain all the .pyc files ( lib basically! )

The workaround Darad and i were using was a post script that did the restructuring to be compliant. ( But obviously this isn't working perfectly with changes - this is what im trying to fix up now )

Even with the discovery of zip_includes_packages as a means to minimize the amount of code signing issues - the output structure in general is still invalid from Apples POV so is highly likely to cause future support issues with cx_freeze depending on how aggressively apple enforce their app structures.

Edit: Some useful links here

• Packaging Mac Software for Distribution
• Creating Distribution Signed Code for Mac

With regards to pyside2 vs pyqt5 + pip vs conda I dont think this is relevant to this issue - as its the output of bdist_mac that's incompatible with code signing / notarization in general.

As mentioned in the other issue - i'm still hoping to fix up our restructuring code to get a clean build, then work with you to contribute back to ensure the native output of cx_freeze for the bdist_mac command is compliant.

... i hope that all makes sense 😅

[TechnicalPirate]

Mini update

After some more head scratching / head desk interactions - I was stumped for ages on afinal error i was getting from the notarization process + our bundle restructuring post step

"The signature of the binary is invalid." error for the actual app inside /macOS/MyappName

Eventually i stumped on this forum post link and low and behold - cx_freeze now outputs frozen_application_license.txt into the MacOS folder - I nuked that file in our bundle restructuring and now it passes notarization...

... but the app dosen't yet run - so some more fiddling required - will update once i get a running, clean build and then start dissecting all the mess of changes so far :)

[TechnicalPirate]

Hi @marcelotduarte!

Another mini update -> My teammate is back from vacation and we're now co-dev'ing a solution to this on a fork.

Couple of questions:

• Are you actively doing development work on the bdist_mac implementation? ( so we can avoid conflicts in future! )
• How best to contribute? - One PR that addresses the overall issue -> which if needed we can break into smaller chunks after reviewing the overall change? - or would you prefer us to contribute PR's for each bespoke step we can identify?

[marcelotduarte]

Generally what I did for bdist_msi was minimal, just a few tweaks. If I have to do something, only if something different from what you're already looking at comes up.

To review, I think following this is best, but if the change is focused on bdist_mac, and doesn't affect other modules, you can do whatever is best for production and testing.

[TechnicalPirate]

Hi @marcelotduarte

Making progress but hit a snag when building locally on mac vs on our CI setup.

How do you build .whl's for cx_freeze when releasing? - Using setup.py bdist_wheel outputs a valid product but there appears to be platform / architecture differences in the output metadata and a problem with the resulting product trying to load Python from an unexpected place ( /Library/Frameworks - rather than the venv im working in - which is what the official whl's do )

i suspect I'm not invoking the build process correctly / I'm missing environment variables or context - Can you tell me what commands you call when building a release? / what env vars have been set? :)

Edit1: Right now im trying to locally build this commit as its the last stable version - but when i generate the .whl and use that - it doesn't work as expected

• If i fetch the official release i get cx_freeze-6.15.6-cp39-cp39-macosx_10_9_x86_64
• If i build locally i get an output of cx_freeze-6.15.6-cp39-cp39-macosx_10_9_universal2

Edit2: Inspecting the .whl's themselves - when i build locally the entire lib folder is missing from within the bases directory

[marcelotduarte]

I use cibuildwheel.
workflow: https://github.com/marcelotduarte/cx_Freeze/blob/main/.github/workflows/build-wheel.yml#L44-L51
pyproject: https://github.com/marcelotduarte/cx_Freeze/blob/main/pyproject.toml#L218-L238

Locally you can use:

python -m pip install cibuildwheel
export CIBW_BUILD="cp39*"
cibuildwheel --output-dir wheelhouse

[marcelotduarte]

Release 6.15.7 is out!

[TechnicalPirate]

Heya!

So I'm now porting our actual apps ( not just the samples ) to the latest cx_Freeze and I'm stumbling on some corner cases that i think warrant this issue being reopened to address :)

For transparency - Apple can be quite cryptic / ambiguous on why notarization can fail - I just realized upon porting that it will give misleading no timestamp / no hardened runtime warnings + errors on the Contents/MacOS/*appname* path - they are actually triggered because of none executable files inside that directory.

For the record - the errors are:

Removing the extra files from our core app fixed this issue - where does this leave us w/cx_Freeze? - well

• include_files parameter - will leave the specified files inside Contents/MacOS ( should be Contents/Resources )
• Executable(... icon="someicon.png") - Will also leave the icon inside the Contents/MacOS directory ( should be Contents/Resources )

There is perhaps more corner cases that we will discover - But for now documenting what i've found and preparing a fix.

[TechnicalPirate]

Ok after a lot of digging i've come to the conclusion i can work past these issues without further changes -but i will report them here for clarity ( they may need turning into their own issues )

• include_files parameter -> accepts a tuple allowing control over where they get placed inside the bundle

• Potential Improvement: Anything added to include_files is placed inside the /Resources dir

• Exectuable(... icon="someicon.png") - So by default there is no way to control where cx_Freeze will output this on disk - it will appear next to the executable and give the above errors on notarization - however you can just not set this parameter as a work around and instead use a info.plist file to add a icon to your app

• Potential Improvement: Change implementation of set icon code inside cx_Freeze to output into /Resources

... im still testing - but im veering away from needing to reopen this issue - as the blockers so far i can workaround - but we could improve the out of the box experience of cx_Freeze by making some these changes, i also feel we may want to raise an exception / warn the user if the /Contents/MacOS folder contains anything other than the executable + lib symlink 👍

[marcelotduarte]

Hi!

/Contents/MacOS folder contains anything other than the executable + lib symlink

From what I understand, executable and symlink are acceptable, so I had an idea, I'll post it later.

[TechnicalPirate]

Correct yep - executables + symlinks are allowed anything else will cause hard to debug notarization errors :)

[marcelotduarte]

PR #2048
You can test the patch in the latest development build:
pip install --upgrade --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze

Basically:
Copy the full build_exe to Contents/Resources
Move only executables in Contents/Resources to Contents/MacOS
Continue with symlib to lib

include_files parameter -> accepts a tuple allowing control over where they get placed inside the bundle

With is patch, files in include_files will be in the Resource

Potential Improvement: Anything added to include_files is placed inside the /Resources dir

bdist_mac already has include_resources but you won't need to use.

Exectuable(... icon="someicon.png") - So by default there is no way to control where cx_Freeze will output this on disk - it will appear next to the executable and give the above errors on notarization - however you can just not set this parameter as a work around and instead use a info.plist file to add a icon to your app

Now, icon goes to Resource too.

Potential Improvement: Change implementation of set icon code inside cx_Freeze to output into /Resources

bdist_mac already has iconfile that is added to icon.icns, but we have check if it work correctly.

[TechnicalPirate]

Hey just confirming - have tested this, honestly great improvement - wish i'd thought of it myself :D

[marcelotduarte]

Release 6.15.8 is out!