Cannot boot RancherOS on Raspberry Pi 4

[ubergeek77]

RancherOS Version: (ros os version)
RancherOS 1.5.3 (rancheros-raspberry-pi64.zip)

Where are you running RancherOS? (docker-machine, AWS, GCE, baremetal, etc.)
Raspberry Pi 4 (4GB)

I realise that the Pi 4 is still fairly new, and lots of OSes still need to be updated to support it, but an issue on the main Rancher repository claimed to be running RancherOS on their Raspberry Pi 4, so I thought I was clear to try it out.

But when I flashed the image to the Pi 4 and plugged it in, nothing happened - literally. It was plugged in via Ethernet, but my router didn't detect it as a client. I had it plugged in via HDMI, but my monitor didn't detect any display signal. Absolutely no signs of life.

The same SD card was able to boot RancherOS successfully on a Pi 3 B+ (although the OS was completely broken... but I suppose that's for another issue), and I can even boot other OSes on my Pi 4 just fine.

Only RancherOS is giving me issues, which is bizarre, since someone is claiming to have it running already on a Pi 4.

Does RancherOS already support the Pi 4 and I'm doing something wrong?

[niusmallnan]

We have not tested RPi 4 yet. I can't easily buy RPi4 at my workplace.

This support will be postponed to the next milestone(may be 1.5.5)

[ubergeek77]

Thanks, that's what I figured, but I thought I should at least mention it, given someone was apparently already able to use RancherOS on a Pi 4.

It clearly doesn't work out of the box, so maybe they hacked it together with some boot files from Raspbian or something.

[bf8392]

Anyone else tried this?

[chrismathis]

Yes. Network (wired) does not seem to initialize. I.e. non blinking lights on the network connector. I'll have to get a micro hdmi cable before I can get console output.

[ubergeek77]

For what it's worth, I wasn't able to get the Pi to display over HDMI when booting RancherOS (I mentioned this above). It doesn't seem to do anything when trying to boot RancherOS.

[killua99]

Nothing extra to add, tested the 1.5.4 same result.

It's possible to compile the latest master to give it a shoot? I didn't search a lot about documentation to compile RancherOS for arch64 and mostly for PI.

[ubergeek77]

I haven't looked into RancherOS' boot files, but I know that the Pi 4 requires a very different kernel/kernel settings than a Pi 3 or Pi 3 B+ requires. In other words, you actually need to create something purpose-made in order for the Pi 4 to even boot.

With that in mind, and since a collaborator here says that work on the Pi 4 hasn't started, I don't think RancherOS, even up to the latest commit to this repository, will be bootable until that work has started.

And since others here have confirmed my findings of the Pi 4 not booting at all, either the issue I linked in my original issue is mistaken (a Pi 3 was used instead of a Pi 4), is lying (very unlikely), or the submitter of that issue supplied their own boot files. Perhaps they Frankenstein'd them from another OS, or compiled them manually on their own.

Given the state of RancherOS on the Pi 4 right now, I'm actually surprised a Rancher maintainer hasn't called attention to that in that linked issue. Especially considering they started a full bug fix on behalf of it.

[prom3t3us]

can confirm that 1.5.4 won't boot with rpi 4 ... any news on this ?

[btharper]

The current pi image is based on the "bcmrpi3_defconfig" and various support files are included for the older processor. Because of the significant changes to the SoC new files are needed as @ubergeek77 mentioned. The kernel version also would need to be bumped from 4.14 to 4.19 to include more recent support for the rpi4.

I've gotten a one-off build to boot without issue on multiple (4.19, 5.2) kernel versions (just booting and entering simple docker containers, far from fully tested). Adding support can be done either by expanding the boot partition from 30MB to something larger and making a single rpi-64bit release, or as two releases, one for each SoC.

Raspbian doesn't have any arm64 builds, but puts all builds in 1 release.
Sakaki-'s Gentoo Build is a 64-bit combined pi3/pi4 release.
BalenaOS uses separate releases for each version of the pi, with 64-bit capable boards supporting both 32/64 bit containers.

It's also possible to separate builds based on GPU usage, I suspect a large number of instances would be run headless, so you can squeeze out a bit more memory by using the cd (cutdown) images and only allocating 16MB of memory to the GPU. When camera support is needed, the x (expanded) image is required with a higher minimum GPU memory of 128MB. Minimum for the standard image seems to be 32MB and default is 64MB. Selecting a single (default, _x, or _cd) set of boot files can also help slim down the boot partition some.

Does anyone feel strongly about a particular type of release? For the sake of simplicity a single image seems like a safe starting point.

[bf8392]

@prom3t3us also confirmed that 1.5.4 not booting on pi 4

[btharper]

Definitely beta (no video output, not yet sure why), but should let a few other people play with it.

Contains pi4 image and build files.
https://github.com/btharper/os-rpi-kernel/releases/tag/v4.19.80-rpi4-beta

Adding

hdmi_force_hotplug=1
hdmi_force_hotplug:0=1
hdmi_force_hotplug:1=1

to config.txt fixes video for me, I'm not sure if this is due to my setup or the image, let me know what works for you.

[pudgypaw]

v1.5.5 also expresses issue as reported by @ubergeek77.

Edit: Confirming image from @btharper works for me, before appending to config.txt.

[orangedeng]

@btharper I tried your way but the hdmi output is not working for me. Just want to let you know.

[dudezilla]

@btharper - boots. SSH Works. Video output works (only with config to manually enable HDMI)

ideas - one one think that board is a constant- but I did use the newer beta firmware from Hexxeh? I hope I have the name right. It is a potential variable that could account for users experiencing different outcomes.

I do like the idea of a build image. - SBC Specific PI4. Something like Yocto - looks interesting. I, personally, am still in pursuit of the goal of k3s cluster.

Needs: 64 bit kernel with small footprint. For KVM and what not.
Highly desirable - Netboot all nodes.

[btharper]

New Pi4 only image posted based on RancherOS 1.5.5 https://github.com/btharper/os-rpi-kernel/releases/tag/20200301

HDMI workaround is built into the config.txt by default. Wifi firmware is not working, copying the full /lib/firmware directory from a raspbian image got things working though.

I like the idea of netbooting, I would have to see what's required, but hosting the files from the image on a PXE server may be sufficient once the Pi4 has the EEPROM update for netbooting. Bootstrapping into a cluster would be the more challenging portion.

[blitmap]

New Pi4 only image posted based on RancherOS 1.5.5 https://github.com/btharper/os-rpi-kernel/releases/tag/20200301

HDMI workaround is built into the config.txt by default. Wifi firmware is not working, copying the full /lib/firmware directory from a raspbian image got things working though.

I like the idea of netbooting, I would have to see what's required, but hosting the files from the image on a PXE server may be sufficient once the Pi4 has the EEPROM update for netbooting. Bootstrapping into a cluster would be the more challenging portion.

Okay, so in response to this:

I (net-)booted your image with the copied raspbian firmware. The PXE server is raspbian-lite, hosting the firmware and kernel. cmdline.txt points it back at the same server for the root fs. I wish I could figure out how to make this read-only. I cannot seem to get ssh enabled for password or passwordless logins. :( I'm just not as familiar with rancher I'm afraid. There's also this issue...

raspberrypi/linux#3286

...where the sd card is not present and it polls it repeatedly. It just fills the console with spam/debug text, but otherwise I can ping the machine and it does seem to be working. (even if i can't get into it, hah)

Just wanted to confirm I did get this to boot with the above image on an RPi4

[dudezilla]

Hey guys been out of the loop for a while - I am back with a minimal test env. Just Pi on Pi was a bit slow for iteration. I have no exp. with CI/CD - I want this working to build into that.

But I cannot get that far until I have a working "egg" to hatch into a "chicken"

My Egg:

But maybe I can configure rancher-os on a PI3 to boot from an NVME. -

Sort of like my pi4 config.

I can move the files I suspect do not relate to the pi-3 boot sequence into both
a "pi4" boot sequence and a "pi2B" boot sequence and split the Pi-64 image into three parts
and rewrite the so that NO DECISSIONS are made in the boot config's.
Just Sequence - No Flow Control.

Motivation:
This could be fun. - I will see what I can do to publish what I've found on Git so that we can take this issue to a place where things are understood in a more objective fashion. Because Wiki's and Six year old documentation is a problem from my perspective. Maybe others are experiencing that.

SO: Today's experiment will involve a git hub page that shows the result of a boot. - And the files in the boot directory as the machine is booted a number of times.

End Result:
-Refactoring the boot to include Only Pi3 boot kernel- Config script boots only the pi3.
-Pi3 Boot will contain Pi3 Boot
-Pi2B Boot will contain Pi2 Boot
-Pi4B Boot will contain Pi4 Boot

Point on leaving:
The end result of the experiment could be a bash-script "Prune My Berry Tree" --Please help with a better name. Thx.

[dudezilla]

dudezilla/PiBoot

Text Feedback on experiment conducted.

[bf8392]

Has someone tried it with 1.5.6? Would love to see it on pi :-) I use it on all my servers because it's the best system for docker! Keep up the amazing work :-)

[olljanat]

Rancher OS is EOL from Rancher side #3000 but I'm trying to build up community to keep it alive. Please share you use cases on burmilla#6

Basically bcm2711-rpi-4-b.dtb firmware is needed by Raspberry 4B. Look DieterReuter/rpi64-kernel#11

PS. I'm writing this one using 64 bit version of Raspbian on Raspberry 4 (8GB) Model B 😉

[srbala]

Rancher has the another project k3s is designed for PI and IOT devices. checkout https://github.com/rancher/k3os/releases for amd64 and arm64 releases. setting up a cluster out of multiple PI using this is possible.

[srbala]

One more thing to make note is the direction of kubernetes releases 1.20 and above

https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/

[olljanat]

FYI. BurmillaOS (fork of RancherOS) latest beta version supports Raspberry Pi 4 https://github.com/burmilla/os/releases/tag/v2.0.0-beta4