feat: kubo 0.29 and native apple silicon

[lidel]

This is exploration into better support of Apple Silicon.

electron-builder supports producing either universal package or one for each architecture.

In case of Desktop, things are more complicated: we also bundle go-ipfs binary.
This means that unless the go-ipfs binary is also universal or per-arch, we can't improve much.

Due to the way we depends on go-ipfs via npm-go-ipfs universal binaries all the way sound like way easier route (downside being the disk size), however dist.ipfs.io already has amd64 and arm64 for darwin.

Universal binary test

Main cost of universal package is size, even with out go-ipfs (still being amd64) it adds ~75 MB:

• 195M IPFS Desktop-0.16.0-universal.dmg (electron amd64&arm64 + go-ipfs arm64)
• 119 MB IPFS-Desktop-0.16.0.dmg (amd64, M1 will use Roseta)

TODO

• decide if we want to provide single universal binary or one for amd64 (legacy intel) and one for arm64 (new apple silicon)
  • Data point from Brave:
    • atm IPFS integration is using amd64: go-ipfs binary is amd64, it runs on M1 via Rosetta. Brave plans on using arm64 in the near future, as they already provide arm64 version of Brave itself.. In that contenxt universal binary is wasteful (IPFS component is fetched on-demand, so small size matters).
• based on previous decision, decide how we handle go-ipfs dependency (right now it downloads kubo only for arch of CI worker, which is amd64).
• make Universal DMG and ZIP
• replace Kubo binary with universal one
  • postinstall script that detects darwin build on ARM, and fetches missing intel binary + merges both into universal and puts it at node_modules/kubo/bin/ipfs
• upstream kubo+0.28.0.patch to npm-kubo repo
  • opened feat: expose standalone download function npm-kubo#73
• test DMG from this PR on both
  • Intel
  • ARM: tested on M1 macMini with macOS Sonoma (14), works fine with universal Kubo binary
• update Electron (if needed) does not seem to be needed on ARM, tbd if Intel works
• wait for https://github.com/ipfs/npm-kubo to ship with exposed light download function
  • switch this PR to kubo dependency with feat: expose standalone download function npm-kubo#73 (0.29.x)
  • remove package-patch for kubo dependency
• fix powershell bug from Auto update failed while executing the powershell command electron-userland/electron-builder#7127
• resolve crash from feat: kubo 0.29 and native apple silicon #1856 (review)

Closes #2813

powershell.exe -NoProfile -NonInteractive thing should be also fixed by electron-userland/electron-builder#7127 (comment), so:

Closes #2335
Closes #2636
Closes #2752
Closes #2778
Closes #2779
Closes #2780
Closes #2784

[AlexxNica]

@lidel, do we know where the difference in size comes from, specifically? We could find where they are and try something to minimize the impact of shipping the universal package.

[unbeatable-101]

@lidel, do we know where the difference in size comes from, specifically? We could find where they are and try something to minimize the impact of shipping the universal package.

Not a dev, but I'm pretty sure the size increase is just the binaries about doubling in size due to basically containing two apps.

[jenkijo]

Hello, any update?
I've committed fix for go-ipfs ipfs/npm-kubo#44

[dvcrn]

Thanks for the PR! I wanted to build this locally but ran into issues with the build step trying to download a arm ipfs binary that obviously didn't exist. How did you build this?

[jenkijo]

It work fine with me, what error that you got?

[hacdias]

I think that Universal would be easier for the end users despite the storage size. In general, storage is cheap and 75 MB won't probably affect many people.

Iff storage size is very important, we could either build universal + download go-ipfs on first run, or ship different binaries.

[SgtPooki]

@lidel Have we looked into https://github.com/tauri-apps/tauri at all? It claims significant improvements over electron across all platforms, though I haven't investigated more deeply. I also haven't found any apps that have migrated from electron to tauri

[SgtPooki]

Regarding universal app vs rosetta on M1.. the speed increases you get by running M1 native is significant in every benchmark I've seen. However, it would be great to see a breakdown of the risks & benefits for our specific application.

I don't think we should decide to just force rosetta forevermore. If the decision is to wait for some period until we have more bandwidth, I think sticking with rosetta and the smaller bundle for now is fine. Ultimately, we should be moving to the type of app bundle the platform we're targeting recommends.

For Apple/macOS, that is the universal build.

[hacdias]

@SgtPooki an issue you may want to look at too is #867

[lidel]

Have we looked into https://github.com/tauri-apps/tauri

I think we looked at it briefly, but it was very alpha back then. IIRC the main downside was lack of tooling for end-to-end autoupdate solution, which is raison d'être of ipfs-desktop. It seems they work on it, but it is still undocumented? tauri-apps/tauri#2776

Long term, we want to move away from fetching updates from centralized HTTP server like github and leverage IPFS+IPNS instead (see old notes in #789), so if we have working setup for that, we could move away from Electron (literally the only value it brings is https://www.electron.build/auto-update working on all three platforms: macOS+Windows+Linux(AppImage))

75 MB won't probably affect many people.

Usually I would be against doubling the size of download, but we talk Apple here – it is fair to assume that if someone uses mac they most likely don't care about additional download cost over the mobile carrier (just being realistic here).

Ultimately, we should be moving to the type of app bundle the platform we're targeting recommends.
For Apple/macOS, that is the universal build.

Yeah, sounds like universal binary is the way to go. The caveat here is that we don't have "universal go-ipfs binary" – iiuc this PR (in current form) still bundles the intel version of go-ipfs.

Figuring out how to include both binaries (arm64 and amd64) for darwin is the key to unblocking this.
Someone familiar with Apple hardware and ecosystem needs to investigate if ipfs-desktop should fetch independent binaries during the build of DMG, or if dist.ipfs.io website should add universal binary upstream (if it is even possible for standalone CLI binaries).

[lidel]

I wrote a plan for unblocking this in ipfs/distributions#770 (comment).
Not a macOS user myself, but if someone wants to work on this, I'm around to advise.

[SgtPooki]

I wrote a plan for unblocking this in ipfs/distributions#770 (comment).

Not a macOS user myself, but if someone wants to work on this, I'm around to advise.

I read that comment and the approach seems reasonable and clear. Thanks for the backlink!

[whizzzkid]

@lidel should this be closed as ipfs/distributions#770 (comment) is closed.

I was also wondering if this could be solved as part of #2441 where instead of "shipping" a binary, we download and validate checksums and then use that?

[lidel]

There are two separate topics at play (mind I am not macOS user, below is my rough understanding on Apple Silicon situation)

• architecture of go binary (kubo daemon)
  • What we did on dist.ipfs.tech is we provide separate Kubo binaries for amd64 (intel) and arm64 (arm/apple silicon) (feat: kubo builds for arm64 distributions#838, fix(site): default apple downloads to arm64 distributions#841).
  • Brave downloads correct binary at runtime.
  • There is no universal binary for Kubo, but you could build one as noted in Add Universal Mac OS build distributions#770 (comment) – afaik nobody is doing that atm.
    • IPFS Desktop DMG bundles NPM version pulled in during the build – afaik this is still amd64, so it runs emulation.
    • IPFS Desktop needs to either stop embeedding and fetch correct binary at runtime, or include both and run correct one on macOS.
• architecture of electron app (ipfs desktop)
  • this feels unresolved, but maybe not a big deal, since the majority of work happens in kubo?
  • iiuc what happens on macOS is that we produce DMG that is amd64 and when run on Apple Silicon it runs emulation

I agree we should close this, as universal DMG does not solve it end-to-end.
If we want to use native arch on macOS, need to fill new issue and make a new plan for both topics above.

[SgtPooki]

please see #2681 for a continuation of this work

[lidel]

Reopening because of #2813

I will prototype Universal DMG that includes Universal Kubo binary as well.

[SgtPooki]

Should this be version 0.36.0? code changes look fine. app shows universal. but I tried running the app (from latest CI job) and it crashes immediately.

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGBUS)
Exception Codes:       KERN_PROTECTION_FAILURE at 0x0000007483f72aa9
Exception Codes:       0x0000000000000002, 0x0000007483f72aa9

Termination Reason:    Namespace SIGNAL, Code 10 Bus error: 10
Terminating Process:   exc handler [24263]

VM Region Info: 0x7483f72aa9 is in 0x74005c0000-0x7500000000;  bytes after start: 2207984297  bytes before end: 2080953686
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      Memory Tag 255             7400540000-74005c0000   [  512K] rw-/rwx SM=ZER  
--->  Memory Tag 255             74005c0000-7500000000   [  4.0G] ---/rwx SM=ZER  
      Memory Tag 255             7500000000-7500004000   [   16K] rw-/rwx SM=ZER  

[lidel]

@SgtPooki yes, this will be released as 0.36.0 (2e891f3 for release-please, it will open release PR once this one is merged).

But the crash is concerning.

• Are you on Intel or Apple Silicon machine?
• What is the version of macOS?
• Is it crashing every time?
• Was it crashing with RC 2 (.zip of build here)?

I can get my hands on M1 macMini tomorrow, but having second data point would be useful.

[SgtPooki]

• Are you on Intel or Apple Silicon machine?

Apple silicon

• What is the version of macOS?

Sonoma 14.5 (23F79)

• Is it crashing every time?

Yep.

• Was it crashing with RC 2 (.zip of build here)?

I can get my hands on M1 macMini tomorrow, but having second data point would be useful.

that version is also crashing for me immediately. I remember a previous version you asked me to test a month ago or so worked fine. If that's the same one, it's not now and the problem is my system.

---

It seems like it might be related to me using nix-shell for nodejs and it's not letting the kubo binary download

[lidel]

ℹ️ This was done mostly to ensure macos update works fine on ARM, but also includes fix for powershell.exe -NoProfile -NonInteractive errors:
electron-userland/electron-builder#7127 (comment)

[lidel]

Thanks, sounds like it.

FWIW I've just tested both on M1 macMini and both work ok. Let's merge and ship.

If crashes of IPFS Desktop binary (electron) rather than kubo itself become common, then the first thing we should do is to upgrade to supported version of Electron (#2775)