-
Notifications
You must be signed in to change notification settings - Fork 846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NetBeans 12.5/12.6 missing ARM based binaries for MacOSX terminal support in NetBeans #3467
Comments
@tmulle can you build and test the binaries on a M1 Mac? oyarzun@5523c7f @vieiro FYI, we can cross compile the binaries on an Intel mac. |
@oyarzun I cloned your git REPO and ran "ant -Dcluster.config=basic" like I did during my testing and I don't see any arm binaries in the directory. I ran the netbeans build and the terminal was blank.
After the build finished:
|
@oyarzun Sorry, my bad. I didn't checkout the But even after switching to the macOS_arm branch and building, the binaries are still not being generated. Unless, I'm doing something wrong.. |
@tmulle the binaries are not built using ant. I believe ant will download precompiled binaries from http://netbeans.osuosl.org/binaries/06D8BC67B2F104F2DE773CB0F77AACE6B3DF553D-exechlp-1.0.zip To test my branch you would need to run FYI, I just pushed another commit to that branch since there was a typo in one of the makefiles. |
Hi @tmulle, as @oyarzun says it's just the The steps to verify that the library can be built in M1 would be: cd a-working-directory-you-want
git clone https://github.com/oyarzun/netbeans.git
cd netbeans
git checkout macos_arm
git pull # make sure we're in the last commit in this branch
cd ide/dlight.nativeexecution/tools/
sh ./buildall.sh
ls -ali ../release/bin/nativeexecution |
@oyarzun @vieiro ok, that might have worked! Just a small change:
The org.netbeans.modules.nativeexecution.api.util.HelperUtility.java class builds the file it's looking for as 'MacOSX-cpufamily_bitness' Here is where the path is used in that class..path is set to "bin/nativeexecution/MacOSX-arm_64/pty"
Once I had the correct directory, I then could open the terminal and according to Activity Monitor the PTY process was Apple instead of Intel which means it worked. I'm not sure why 2 PTY processes open when there is only 1 terminal visible in NetBeans, however. Here is the file info from the Inspector in Activity Monitor: One thing also, is if I close the terminal in NetBeans.. those processes don't seem to close. Here is a ps -ef | grep pty after I shutdown NetBeans and you can see the PTY processes still hanging around:
|
Hi @tmulle , thanks! Would you please run this as well so we understand what exact files are generated? cd a-working-directory-you-want # same places as above
cd netbeans
cd ide/dlight.nativeexecution/tools/
find ../release/bin/nativeexecution i.e., we want to know the exact names (i.e., if we're getting You may want to zip that directory and attach here. @oyarzun , we'll need at least another independent compilation (to ensure @tmulle is not adding an extra bitcoin miner to the binaries :-D ) and then upload the zip and new hash with these. |
@vieiro haha..no bitminer code here :) Here is the output of the find command.. I ran it inside the current macos_arm branch I previously built. The NetBeans code wants the directory to be called 'MacOSX-arm_64' You can see it's building to the wrong directory name that the code is looking for as mentioned in my previous comment a few mins ago.
|
Thanks @tmulle ! Well, it's a pity we don't have a bitcoin miner in NetBeans :-D. @oyarzun even though the binaries are added to the zip file, we still will need to modify the nbproject/project.properties file [1] so NetBeans adds them to the module (you can do that in |
Remove needs:triage - think we're sure this is a bug - @vieiro @oyarzun please remove that label on responding to an issue. Also added needs:discussion as something to follow up on dev@ here - we should move the location of the binaries to Maven if possible, similar to the Windows launcher, and wind down usage of OSUOSL - cc @ebarboni and https://lists.apache.org/thread/9tf28w5r3wxx9fwbl12m6x36sb3pfdry @oyarzun Swift launcher also going to be related to this? For NB13 I wonder whether we should make the quick fix change at least, if we can't get this in? |
@vieiro @oyarzun Did you see the issue about possible resource leak above comment? in #3467 (comment) Just want to make sure it's not missed in case it is a real issue.. |
@neilcsmith-net , let's move this to the mailing list then. |
I would say this is a feature rather than a bug as it is a new unsupported architecture. While we are adding this, we probably should add Linux arm as there are no binaries either.
IMHO, these changes can wait until after NB13.
I looked at this as well, for the launcher we can create a universal binary that contains both the x86-64 and arm-64. Currently the NetBeans launcher is built when the macOS installer is created. @mcdonnell-john has been building these on his Mac. It can be created with the following command with the latest Swift 5.4 tools, and maybe 5.3. swift build --configuration release --arch arm64 --arch x86_64
@tmulle I just pushed a commit which should fix building the
There are a few platform specific binaries that would benefit from moving the compilation and packaging to a better process. I will start a discussion thread on the dev mailing list. |
Just to add, I noticed this also fails when running with an x86_64 JDK on Apple Silicon. |
I think the JDK is irrelevant since it uses the Line 69 in c084119
|
@oyarzun Hi again... just wanted to let you know that this is still missing in NetBeans 14. I had to copy my modified hostinfo.sh and the ARM folder for Mac to the installation that was created by the above steps. Wasn't sure if this was included in the official releases yet. |
@tmulle correct no changes have been incorporated into an official release or master. |
I just installed NetBeans 16 and looks like the changes still aren't in for the terminal. Will this be in a future release? I know not everyone has M1 Macs.. I think there are companies you can rent M1 Macs in the cloud for build processes for those that don't or can't have the proper machines. |
Hi, any status on this? I just installed Netbeans 17 (working great so far!) - Although not a fan of the new MacOS icons. I don't like the outline folders...:( Anyway, in order to get my terminal to open in Netbeans I had to copy the files once again. Will this be going into master anytime soon? |
@oyarzun Hi, just wondering if this will make it into the official release process for NB18+ so I don't have to keep copying my files on each upgrade? Thanks! |
I am unable to tell what the blocker is on this thread, Jan 2022 this PR was raised, any status update? Netbeans Terminal not functional on Apple Silicon is not a minority problem anymore. |
@ctipper thanks for info. I'm trying to keep Rosetta of my dev machine so that I get immediate warnings from macOS if our universal app bundle for macOS fails to be recognized as a universal app. We have had issues with Info.plist settings and launcher scripts that trigger Rosetta (even though the project is built to be universal). This allows us to catch these kind of issues early and easily. |
@fixbugbroken I assume that you tried the suggested workaround for the error message on libjnidispatch? I guess this might be because loading an unsigned jnilib will fail except in Rosetta? That might rule out the zip package for macOS M1/M2 for good?! I'm happy to look at building a variant of the Codelerity installer with an aarch64 JDK and/or no JDK, if you'd be interested in testing? Contact me via email on GitHub profile. I'm not sure if there's any universal JDK yet? |
@neilcsmith-net Yes clearing the extended attribute removes the warning from macOS. However, as mentioned by the OP the zip file does not contain a directory for MacOSX-aarch64 binaries under bin/nativeexecution directory. I could certainly test a Codelerity installer if that will help Netbeans build an installer that does not prompt for Rosetta. I will reach out to you via e-mail. There are arm64 builds for all JDK versions as far back as JDK 8 (see OpenJDK, Adoptium and Azul). I'm currently running an arm64 distribution of JDK 20 with the Netbeans zip package. Of course the arm64 JDK distributions do not trigger prompts for Rosetta. |
@fixbugbroken take a look at the installer linked from #6052 (comment) The
That wasn't what I meant by universal JDK though. I'm aware of the arm64 JDKs. We (as in Codelerity) might ship a separate installer with an arm64 JDK bundled, although I'd rather see this issue resolved first. The Apache release will never bundle a JDK. The other thing is considering using NBPackage to build the Apache released macOS installer as well, with a universal build of the Swift launcher, as well as fixing the signing issues. That's being discussed in the other issue. |
I tried Netbeans 19 the work around has no effect now and this PR is more urgent than ever. Not a particular fan of the new pkg installer, it's only meant for distributing libraries and it prevents the work around I mentioned above. |
I'm not sure what you mean by the "only meant for distributing libraries" comment? Using We might look at shipping the app bundle directly in a dmg or zip (easier and preferred in my opinion). I doubt this will make a difference for you - everything is now signed with the hardened runtime and the launcher is universal architecture when no JDK is specified. So, altering files in the bundle is not possible, and the fact that the launcher architecture might have fooled macOS in some circumstances was a happy accident. You could use nbpackage to build from the zip yourself, forcing the launcher to x86_64 rather than universal, or just look at one of various macOS options to force use of Rosetta for the launcher? That might still work. Simply put, we need to fix this properly by finding a way to work with the correct architecture, ideally by NB20. |
So I think not triggering Rosetta is governed by how app is installed. DMG method before and I think it should be brought back, helps installing side by side. |
I think I'll stick to Netbeans 18, I really have no clue why this issue has been open for three years. |
The previous installer was still a |
I didn't see that that would explain it. |
Also an unnecessary change, and it stopped the terminal from working. |
The changes were a necessary part of fixing #6052 and other related issues. The changes were also tested and discussed at https://lists.apache.org/thread/gvp7s1l1s0lkltmm5q7q2q1gxc38wnll NetBeans does not yet fully support running on macOS Arm unfortunately. You can run on an Personally, I'd prefer we stopped shipping a macOS installer without bundled JDK runtime anyway - using the zip in those circumstances is preferable IMO for a multitude of reasons. Either directly, or by running NBPackage on the zip with a JDK of your choice. |
I agree, we need better support for macOS Arm. Each release I manually copy the binaries I built on my M1 mac to the new Netbeans folder to get the terminal to work. I run straight ARM jdk using SDKMAN and things work fine. If it would help I can make a GitHub repo with my folder that I built for the ARM support to get the terminal working. It's not perfect, but it will get your terminal working again. |
@tmulle what we probably need to do is extricate the existing source to a separate repo (similar to https://github.com/apache/netbeans-native-launchers ) then look at using GitHub actions to build them, and run a release vote on them. The NetBeans build can then use the released native binaries. I think your input into that would be very welcomed! I also have a local M1 mac, so happy to do some testing and helping out with this. cc/ @ebarboni also for input |
The problem still exists on Apple Silicon with NB19. Looks like the binaries are a "closed shop" and not open source. Where is the source code for these? Do we have to wait until a PMC member wants to use Netbeans on Apple Silicon? |
@sroeper of course it's open source - the source is at https://github.com/apache/netbeans/tree/master/ide/dlight.nativeexecution/tools If we can get a PR in that includes the relevant changes (eg. @oyarzun or @tmulle changes linked above??) then we can look at a similar workflow as for the profiler to release the binaries for consumption in the build. I for one would like to see this fixed in NB20. There is a lot that "simple contributors" can help out with here. |
I can submit a PR for the changes I did, but it is just a few lines added to As far as any other changes I did, I didn't do anything else but following the instructions at #3467 (comment) To test building on my M1. If you follow the comments from that link down you can see what I ran into. TL;DR - running |
To clarify the situation.
To solve the issue at once: a) Compile the binary once in the required platform using the "ide/dlight.nativeexecution/tools/buildall.sh" script (I assume that if you build this in an Apple M1 box it will work well for Apple M2 boxes) To solve the issue forever: a) Ensure that Apple becomes an sponsor of the Apache Software Foundation. D'oh! Apple is a platinum sponsor! Cheers, |
Thanks @vieiro but I wasn't intending to use OSUOSL or user compiled binaries (which would both be -1 anyway). We already have access to macOS cross-compilation and/or Apple Silicon via GitHub Actions. As long as the build process in the main repo works, I intend to fork the workflow I've just merged for the profiler binaries and release via dist.a.o - which is close to your second suggestion. See #6502 I'd like to see if we can fix both these for NB20. As well as merging required fixes, we need to clarify exactly which folders are required (ie. anything outside of Both Apple and GitHub are already ASF sponsors btw! 😄 |
... actually, I may have jumped the gun on availability of the architecture on GitHub actions. Had seen a reference to being in public beta. However, as @oyarzun points out above (and his commit seems to include) we can cross-compile on macOS. If those changes make it in, we might be OK just using an x86_64 runner as we are for profiler binaries, for now? |
@neilcsmith-net Yes, it's enough to have a x86_64 runner and cross compile the binaries. There is no need for a M1/M2/Mwhatever. The only change you have to do is to call clang with the -arch option. To build something which includes both (i.e. universal library/executable) call with -arch x86_64 -arch arm64 Works the same on a M1/M2 |
I've adapted the workflows of @tmulle and @oyarzun for #6513 - thanks both! Artefacts from the workflow can be found at eg. https://github.com/apache/netbeans/actions/runs/6382908341 ... or look for a later run. If anyone is willing to check those, that would be great! There's still some work to do on that, but it might be enough to get us some terminal support for Apple Silicon in NB20?! |
Should be working in NB20. A release candidate should be available next week. Please help testing that when it's ready. |
Apache NetBeans version
Latest release
What happened
When using the “Open in Terminal” option I get nothing but a blank terminal window when running on an M1 Silicon Pro MacBook Pro using native ARM based JDK
After debugging the source code of Netbeans itself it comes down to a missing resource in the “netbeans/ide/bin/nativeexecution/“ folder for “MacOSX-unknown_64/pty”
This is because the host information cannot figure out the CPUFAMILY and is returning “UNKNOWN”
Things I tried:
Add another check for CPUFAMILY and return ARM:
➜ tools git:(master) ✗ chmod +x ./buildall.sh
➜ tools git:(master) ✗ . ./buildall.sh
If I then make a directory called “netbeans/ide/bin/nativeexecution/MacOSX-arm_64” and copy the “buildall/“ output into that directory the terminal works and I they are running natively (Not Rosetta).
Again, this worked for me but obviously someone with more knowledge will know how to fix this.
And yes, I know that you need a M1 Silicon Mac to build the binaries :-(
Quick Fix
I was able to fix it temporarily with these steps but they are not ideal:
Step (1) from above is required
Copying the contents of “netbeans/ide/bin/nativeexecution/MacOSX-x86_64” to “netbeans/ide/bin/nativeexecution/MacOSX-arm_64”
This works, however, any terminal window is now using Rosetta and NOT native ARM code
How to reproduce
Try to open a Terminal window in either NetBeans 12.5 or 12.6 running on a native ARM JDK and you'll get a blank terminal window.
Did this work correctly in an earlier version?
Operating System
MacOS M1 Max Pro Monterey
JDK
openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12) OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode) also happens with JDK11
Apache NetBeans packaging
Apache NetBeans binary zip
Anything else
As requested from the email, here is the output:
Here is also "uname -m" on both my M1 Mac and Intel:
"uname -m" = arm64
"uname -m" = x86_64
Original Request Email:
Hi Tim,
Thanks for your report!
It seems we're not contemplating the new M1 Apple computers. Since not all of us have access to one of these, would you please add the following to this ticket?
a) The result of "uname -p" (this is what we call CPUTYPE in [1])
b) The result of "uname -n" (this is what we call HOSTNAME in [1])
c) The result of "uname -s" (this is what we call OS in [1]).
d) The result of "uname -a" (we use it elsewhere in [1]).
e) The result of "sysctl hw.cpu64bit_capable" (may require 'sudo', this is what we call BITNESS, I assume this is 64 bit).
f) The result of "hostinfo"
It would be great if we could compile "dlight.nativeexecution/tools" for the new M1 processors by running "build.sh" in [2] on one of these computers (I don't own one, though). This will probably require adding new #include's in different parts of the code. We'll talk to ASF Infra to see if we can have one of these to compile.
Thanks again,
Antonio
[1]
[https://github.com/apache/netbeans/blob/master/ide/dlight.nativeexecution/release/bin/nativeexecution/hostinfo.sh]
[2]
[https://github.com/apache/netbeans/tree/master/ide/dlight.nativeexecution/tools]
Are you willing to submit a pull request?
Code of Conduct
The text was updated successfully, but these errors were encountered: