FreeBSD support

[asomers]

Describe the enhancement
Support building the runner on FreeBSD

Additional information
I think FreeBSD has all the libraries that the runner needs. And while the dotnet-sdk isn't availble from Microsoft, there is already a package for it: lang/linux-dotnet-sdk . So it should be possible to build the runner on FreeBSD.

[TingluoHuang]

We can't really do that for mainly 2 reasons.

• Supportability, without office support from dotnet core team, debugging any native dependency problem will be a nightmare.
• Legal, we (GitHub) might run into legal problem when we consume and redistribute 3rd party open source project.

But, we are totally fine with forking this repository and try yourselves, we are generally doing do on holding server/client compat.

Hope this can help. 😄

[PrivatePuffin]

Addition:
lang/linux-dotnet-sdk is based on a very old version. Considering the DotNet Core team is working on version 5 by now, I doubt anyone is willing to spend time debuging it, as @TingluoHuang explained.

[TingluoHuang]

We can add support if we do #243 or Dotnet 5 supports FreeBSD

[Neilpang]

Sorry to bother you.
but I just created an action to make it possible to use FreeBSD in your workflow.

https://github.com/vmactions/freebsd-vm

[davidchisnall]

I'd love to see FreeBSD support, but there's a long tail of other platforms. For Microsoft/snmalloc, we support Windows, macOS, Linux, OpenEnclave, and FreeBSD and have had external contributions for NetBSD, Haiku, OpenBSD, DragonflyBSD and Solaris. We support x86, ARM, MIPS, and RISC-V, (most in both 32- and 64-bit variants) and have CHERI versions in development for ARM, MIPS and RISC-V.

We would love to spin up CI for more of these, and can run a lot of them in Azure to provide self-hosted runners (at least for a weekly test run, if not on every PR), but porting .NET to all of them and having supported releases is a phenomenal amount of work, particularly over the next couple of years as we grow the amount of CI that we want to run on experimental hardware.

It would probably be more valuable to the community to have the protocol that the runners need to speak documented so that third-party contributors can implement something for operating systems that don't have first-party support. The only thing that I need a runner to do for our CI to work is install some packages, download the code, and run a shell script, then report the results. A minimal C++ implementation that I can compile to a statically-linked binary to provide this minimal environment would be ideal for our purposes.

The fact that the only way of getting FreeBSD / OpenBSD / Solaris CI on GitHub actions currently is to spin up a Mac worker and run the desired OS in a VM is quite sad, especially since the initial launch announcement said that other operating systems would be supported.

[ChristopherHX]

You might be interested in a github action compatible runner (at least github.com works) which runs on freebsd and other systems without vm. It was a bit tricky to replicate the protocol, but finally it connected to the github action service.
You can use

• run steps
• composite run steps
• node actions if you have node in your PATH prior launching

[davidchisnall]

@ChristopherHX, thank you, this is fantastic! I can confirm that your runner works really well on FreeBSD. I have not yet done so, but it looks as if the --once flag should make it trivial to set up a jail that uses a ZFS clone that is discarded at the end of the run.

For anyone else looking to use GitHub actions with FreeBSD, I can thoroughly recommend @ChristopherHX's runner. It would be great if GitHub would provide a free runner pool using this for FreeBSD in addition to the Linux / Windows / macOS ones so that folks aren't using the macOS pool and vm-actions to run FreeBSD in VirtualBox on macOS and can just use it directly on a (much cheaper) Azure VM.

Any GitHub people reading this, feel free to ping me on Teams if you want help setting this up.

[davidchisnall]

I've written some scripts for running @ChristopherHX's version on FreeBSD. They use pot to create a jail backed by a ZFS dataset with the runner and any dependencies you want to install and then invoke this in a loop, with pot rolling back the ZFS dataset to the most recent snapshot in between each run. Scripts run with root privilege inside the jail and can do pretty much anything short of loading kernel modules (jails have a shared kernel). I've got it working on one of our repos with a VM providing runners in 12.2 and 13.0 FreeBSD jails. This may be useful for anyone else wanting self-hosted runners on FreeBSD. The scripts are very rough at the moment.

[michael-o]

I guess even though most issues/requirements were resolved, this is still not going forward?

[michael-o]

nvm

What's that?

[ykla]

I hope to support FreeBSD.

[jacob-carlborg]

Yet another third party solution. I've created a GitHub action [1] that allows to run commands on platforms not supported by GitHub (they run in a VM). It currently supports FreeBSD, OpenBSD and NetBSD. x86-64 and ARM64 are supported. Boot time is around 1 minute. It uses GitHub's hosted runners. It works on both Linux and macOS runners.

[1] https://github.com/marketplace/actions/cross-platform-action

[andrewrk]

Supportability, without office support from dotnet core team, debugging any native dependency problem will be a nightmare.

Then hire somebody to do it

Legal, we (GitHub) might run into legal problem when we consume and redistribute 3rd party open source project.

You have lawyers to deal with this problem

[michael-o]

Supportability, without office support from dotnet core team, debugging any native dependency problem will be a nightmare.

Then hire somebody to do it

Legal, we (GitHub) might run into legal problem when we consume and redistribute 3rd party open source project.

You have lawyers to deal with this problem

Yeah, pretty much pretexts.

[akhramov]

Hey there,

I'm wondering if this is the medium to voice feature requests like this.

It seems that certain teams at Microsoft could really benefit from having FreeBSD runners. A quick GitHub search reveals that Microsoft teams are currently having to deal with rather clunky VM workarounds for this purpose (Verona, snmalloc).

On top of that, a certain competitor (I won't name names here) has managed to port their CI runner to FreeBSD. Fortunately for Microsoft, they hasn't yet provided a SAAS option for this.

I get the feeling that this issue might be sitting on the back burner, perhaps out of politeness, but there's a clear demand for it both internally and externally.

Could we please consider taking this to the next level?

[davidchisnall]

Disclaimer: I am currently a Microsoft employee, but I have submitted my resignation and my last day is very soon. I definitely do not speak for Microsoft in this, or any other, regard.

We have had to do hacks work arounds for MS projects to make FreeBSD support work. We eventually moved to using the VM actions and run FreeBSD CI in a FreeBSD VM on the Mac runners. These cost a lot more than a native FreeeBSD runner, but probably less than the cost of GitHub properly supporting FreeBSD.

Anyone whose project is not in the Microsoft GitHub org can simply use Cirrus for CI, since they provide large quotas for open source project and fully support FreeBSD and their design makes it easy to bring your own base VM image, so they can be used with any other platform if you put in the effort.

For Microsoft to do this, GitHub needs to hear demand from customers. From conversation with folks in that bit of the org, FreeBSD support rarely comes up at all, and when it does it’s not near the top of priority lists. They need big potential customers to tell them that they are not buying GitHub services because of the lack of FreeBSD support. If Cirrus has a large number of customers that would be on GitHub actions because of FreeBSD support, this argument would be compelling.

Most of the demand for FreeBSD CI that I’ve seen has been from open source projects that are not paying for CI. Do more work so that you can give us things for free is not a great economic argument, sadly. Do more work so that you can sell us a pile of expensive things is. If companies like Netflix, NetApp, Juniper, and so on that build big FreeBSD products told GitHub that they wanted support, it might make a difference, but as far as I know they all have solid in-house infrastructure.

[Thefrank]

This sounds like a bit of a "chicken or egg" problem. Support for FreeBSD is not going to get added because there is not enough people asking for it so people won't add/use/want FreeBSD runners because of that.(*)

There might be an org size issue too. If your org is large enough to look around and go "FreeBSD would work great for THING-WE-DO because FIT/LICENSING/STAFF/ETC" your org already has the staff to support FreeBSD without outside help or will hire to fill gaps. FreeBSD's usage is, sadly, still rather niche (e.g., Netflix and NetApp use it mostly for getting the most out of KTLS, Sony uses it as a base for their PlayStation console, Apple originally used it for...)

From the above, as a financial standpoint for Microsoft, FreeBSD support would be the digital equivalent of a giant money pit. But with the pit, you can see you got a pit; with this all you see is red on your books.

*A current similar example for those that enjoy long reads is FreeBSD in dotNET.

[cederom]

+1 to add FreeBSD Actions / CI / Runner support. We are censored out from lots of projects where code / test / build / packaging could take place! Please add FreeBSD support! :-)

[bryanmacfarlane]

One solution is to port the runner to go. We already use it in many of our services in github/actions. We (actions engineering) have discussed and it's something on our radar.

[michael-o]

This sounds like a bit of a "chicken or egg" problem. Support for FreeBSD is not going to get added because there is not enough people asking for it so people won't add/use/want FreeBSD runners because of that.(*)

There might be an org size issue too. If your org is large enough to look around and go "FreeBSD would work great for THING-WE-DO because FIT/LICENSING/STAFF/ETC" your org already has the staff to support FreeBSD without outside help or will hire to fill gaps. FreeBSD's usage is, sadly, still rather niche (e.g., Netflix and NetApp use it mostly for getting the most out of KTLS, Sony uses it as a base for their PlayStation console, Apple originally used it for...)

From the above, as a financial standpoint for Microsoft, FreeBSD support would be the digital equivalent of a giant money pit. But with the pit, you can see you got a pit; with this all you see is red on your books.

*A current similar example for those that enjoy long reads is FreeBSD in dotNET.

That would also mean that Microsoft should stop using open source immediately because someone else paid (money or time) to develop it. Give and take.

[michael-o]

One solution is to port the runner to go. We already use it in many of our services in github/actions. We (actions engineering) have discussed and it's something on our radar.

That would be ideal, I never understood why C# was chosen.

[davidchisnall]

One solution is to port the runner to go

Good news everyone!

(Seriously, if you do decide on Go, please don't duplicate effort)

[cryptocode]

One solution is to port the runner to go. We already use it in many of our services in github/actions. We (actions engineering) have discussed and it's something on our radar.

Any updates on this? The project David mentioned seem to confirm its feasibility. Just say dotnot to dotnet for infra projects.

Users of GitHub (including Microsoft as mentioned earlier) currently have to resort to ugly kludges. As a result, countless projects are holding back on offering FreeBSD support when they could easily do so if only GH offered an easy runner solution.

[flavorjones]

Just dropping a note here for anyone who's interested, I've been using https://github.com/vmactions/freebsd-vm for the last few years and it's been a really solid option. It's slower than a native bsd runner would be, but it's sufficient for coverage for my use case.