Stability problems since updating to 10 and 10.1 Pi4 8GB NVMe SSD via USB adapter

[zillion42]

Describe the issue you are experiencing

Hi there,

I just want to add to this. I have a:
Pi 4b 8GB
USB Boot to ORICO SSD Portable External 128GB Mini M.2 NVME

I updated from HA OS 9.5 to 10.0, the day it was released and it has been a nightmare since. I read that some people were not even able to boot when they updated with a similar NVME SSD Pi4 hardware configuration.
#2479

Luckily mine did, it just kept crashing every 5 hours or so. I connected the HDMI and saw that it was the SQUASHFS becoming read only and journald errors.

compare:
https://community.home-assistant.io/t/squashfs-error-ext4-fs-error/293167

I since changed the power supply from a 20W 4 Ampere to a macbook usb-c charger and updated to HA OS 10.1 which, brought some stability improvement. But still it crashed, then about every other day.

Today I rolled back to HA OS 9.5:
ha os update --version 9.5
ha core update --version=2023.1.7

and its currently migrating my DB back

Database is about to upgrade from schema version: 41 to: 30

so it's still very busy. It yet remains to be seen whether I get my old regular 1 month or more uptime without crashes. I really hope so.

This is not OKAY!

I suspect it has something to do with the following 'features', from release notes:

• zswap instead of swap in zram is used. This should allow to use Home Assistant OS on systems with lower amounts of RAM with the trade-off of slightly higher storage wear.

What operating system image do you use?

rpi4-64 (Raspberry Pi 4/400 64-bit OS)

What version of Home Assistant Operating System is installed?

10.1

Did you upgrade the Operating System.

Yes

Steps to reproduce the issue

1.Upgrade from 9.5 to 10.0
2.Upgrade from 10.0 to 10.1

Anything in the Supervisor logs that might be useful for us?

can't read relevant logs since I downgraded

Anything in the Host logs that might be useful for us?

can't read relevant logs since I downgraded

System information

System Information

version	core-2023.1.7
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.10.7
os_name	Linux
os_version	5.15.84-v8
arch	aarch64
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.32.1
Stage	running
Available Repositories	1280
Downloaded Repositories	18

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 9.5
update_channel	stable
supervisor_version	supervisor-2023.04.1
agent_version	1.4.1
docker_version	20.10.22
disk_total	116.7 GB
disk_used	33.4 GB
healthy	true
supported	true
board	rpi4-64
supervisor_api	ok
version_api	ok
installed_addons	Samba share (10.0.1), SSH & Web Terminal (13.1.0), Duck DNS (1.15.0), File editor (5.6.0), Mosquitto broker (6.2.1), ESPHome (2023.4.4), SQLite Web (3.7.1)

Dashboards

dashboards	1
resources	11
views	6
mode	storage

Recorder

oldest_recorder_run	7 November 2022 at 21:22
current_recorder_run	5 May 2023 at 00:06
estimated_db_size	8282.91 MiB
database_engine	sqlite
database_version	3.38.5

Additional information

I downgraded to 9.5 today

I also posted this on the forum:
https://community.home-assistant.io/t/home-assistant-os-10-update-has-broken-my-pi-4b-4gb/561918/24

I hope you are aware that many Pi4b users have a very unstable system at the moment.

[Stooovie]

Similar issues, catastrophic issues with OS 10.1 on Rpi4 never encountered before

EDIT: this made me give up on Rpi4, went to HAOS VM on Proxmox, 100% stable. Cheaper, much more robust with builtin NVMe, using the same power.

[starryalley]

Similar issues. I happened to be upgrading from SD card to SSD during this OS 10.0 release and it was a nightmare. I also saw the similar SQUASHFS error. My HA runs smoothly like for 1 years without any issue. It crashes a few times already during the past week. I'm not sure if it is OS 10's issue or is it because of this new SSD hardware. Let's see if I have more information in the coming weeks.

Edit: I'm on OS 10.1 and it still crashes once.

[markusmauch]

Have exactly the same issue and obviously lots of people do. Since 10.0 my HA crashes at least every second day. Before the Update it ran seamlessly on a Raspberry Pi 4b with an external SSD.

[zillion42]

Hi,

I just wanted to update on this issue. I have had no crashes since I downgraded to HA OS 9.5, uptime is now since 9 May 2023 at 18:17, which was a normal host reboot. I also updated to core 2023.5.2 again, that does not seem to cause any problems.

[danir-de]

I'm having similar issues, it's behaving like a system, starting with HomeAssistant OS 10.0, where the hard-drive got removed while running. Things keep running, but the longer they do, the less functions.

The status page on port 4357 isn't available, the dashboard loads, but everything on it fails to display, error while loading setting pages,...

"Error while loading page hardware"

the app failing to connect locally, automations stop running, Addons crashed, you get the picture.

And since there are no logs kept after a restart, it's impossible to get an idea why the system went into this state.

I'm running the system on a Raspberry Pi 4B+, with an external SSD from the supported RPi SSD-Adapter list and I've even reflashed the OS 2 times already, before restoring my backup - it still keeps happening every 48 hours.

I've attached a capture card, to be able to see what the system prints out, when it crashes again.

[mundschenk-at]

I have experienced the same thing with my Yellow/NVMe combo and symptoms persisted after an in-place downgrade to 9.5 until I did a full reinstall of HAOS from scratch (i.e. wiping the NVMe and reinstalling using USB mass storage mode, as well as uploading a known stable firmware).

I have some serial logs showing that journald can't access its log directory, squashfs errors etc., but they don't show the initial stages of the problem. Once the connection to the drive is lost, there appears to be no way to get a root shell or access kernel logs even with a serial connection. A reboot fixes the issue for some time (a few hours to days), but all logs from the beginning of the fault are wiped.

Since this persisted after the in-place downgrade, I think that the latest firmware maybe to blame, but I don't have anything approaching "proof" for this hypothesis.

Hardware:

• Non-POE Yellow
• CM4 8 GB (full)
• Samsung 980 1 TB
• eMMC is empty (i.e. direct NVMe boot)

[danir-de]

It happened again today, but the screen wasn't on, so I couldn't capture any screenshots etc. Will try to see if I can gather logs any other way.

[zillion42]

Since we're all speculating, maybe 10.1 just uses more power. I used to have this power supply:

I now changed to this power supply:

This seems to support this hypothesis: #2513
External usb3 NVMe also uses a lot of power. Maybe you guys can give it a shot with a good power supply and report back.

from here https://community.home-assistant.io/t/installing-home-assistant-on-a-rpi-4b-with-ssd-boot/230948 :

The second most important factor for your success is to use a power supply that is capable of driving your Pi 4 and (!) your SSD. Nothing will give you more headaches than an insufficient power supply. You system will stall when you don’t expect it and you will not understand why!
At least use the original Pi 4 power plug with 3 Ampere. If you can get a good quality supply with 3.5 Amps or more: Use it!
Alternatively you can use a powered USB hub or casing to give your SSD a dedicated power supply. I personally don’t like that idea because a second power plug creates additional energy losses and is another item that can break. But it is still much better than an unstable supply.

EDIT: Still rock solid, no crashes, since I downgraded to HA OS 9.5

[Stooovie]

I did, 3A PS for the Pi 4 PLUS externally powered SSD. Didn't make a difference.

[zillion42]

@danir-de

I've attached a capture card, to be able to see what the system prints out, when it crashes again.

Would be nice to have rsyslog as an addon maybe.
sudo apt install rsyslog

[danir-de]

I'm using a POE adapter, that gives up to 15W in 5V mode (= 5V 3A) and I don't think the upgrade resulted in a higher power draw, especially, since the drive is still signaling some activity even after it crashed.

I haven't researched on how to access the hypervisor directly via SSH without any Add-on, in order to be able to access logs directly, maybe I'll have some time next week to look into it. I've tried to connect via port 22222 before, but the sshd daemon seemed unresponsive, once the system crashes.

[markusmauch]

The first thing I did once the system started to crash was to replace my old 15 W power supply with one that has 20 W. I think this rules out any concerns regarding the power consumption.

I can also confirm that it is no longer possible to SSH into the host (admin via port 22222) once the system becomes unresponsive.

I did, however, notice that recently one of my custom integrations stopped polling. I have a Riemann sum sensor based on one of the integration’s native sensors which then had really strange values. I replaced the custom integration by self-configured REST entities two days ago and since then the system seems to be stable. Not sure though if this is related in any kind…

[mundschenk-at]

I am pretty confident it is not the power supply. After the first incident, I replaced the stock power supply that came with the Yellow with a 36 W battery-buffered Eaton 3S Mini and it still happened again.

[agners]

@mundschenk-at unfortunately several reports of Samsung 980 1TB models not behaving well with Yellow. But since that was also in 9.5, and on Yellow, this is not related to the topic at hand. The symptoms might look similar, but the cause is different: On Yellow the NVMe is directly attached. On RPi 4 an USB to NVMe adapter is being used.

Unfortunately the exact cause for Samsung 980 Pro missbehaving on Yellow is unknown. At this point I can only suggest to use a different NVMe (note also that the Samsung 980 Pro is a bit overkill for Yellow: The CM4/BCM2711 only uses PCIe Gen 2 x1).

See also: #2235 (comment)

[danir-de]

This isn't related to that Samsung SSD though, I'm using an Intenso M.2 SSD TOP 128GB as well as this SSD Adapter from SSK.

Also in addition to the POE power supply I'm using, my Pi also is mounted inside a PI-TOP [4], which provides battery power to the Pi 4, so undervoltage or an unreliable power supply shouldn't be any problem here.

The problem is persisting with the latest version 10.2 btw and I haven't been able to extract any useful logs afterwards.. :/

[agners]

@danir-de right, this Thread is not related to Samsung NVMe SSDs or Yellow.

@mundschenk-at your case is really off-topic here. This thread is about issues with NVMe and USB adapters connected to a Raspberry Pi.

[agners]

The problem is persisting with the latest version 10.2 btw and I haven't been able to extract any useful logs afterwards.. :/

This is most likely related to Raspberry Pi's Linux kernel and/or firmware. There hasn't been an update to them since a while, so this is kinda expected.

Are you using USB boot? Can you try to use SD-card boot along with the data disk feature to see if that works better?

[markusmauch]

After the latest 'incident' I accessed the OS via SSH (admin / port 22222) and did some research in the journalcrl. I found that there are no entries for almost two hours until the time noticed that the system got unresponsive and restarted the hardware:

Jun 06 06:07:52 homeassistant addon_77113f40_powerbox-mqtt[567]: END Reading value of ...
Jun 06 06:07:52 homeassistant addon_45df7312_zigbee2mqtt[567]: Zigbee2MQTT:info 2023-06-06 08:07:52: MQTT ...
Jun 06 06:07:52 homeassistant addon_45df7312_zigbee2mqtt[567]: Zigbee2MQTT:info 2023-06-06 08:07:52: MQTT ...
-- Boot 53ee02a2a235499faf6b7779d455b507 --
Jun 06 08:03:16 homeassistant systemd[1]: Starting HassOS AppArmor...

There is also nothing before that event in the logs that would look suspicious. The system just suddenly hangs and does not even write logs anymore.

I updated to 10.2 a few days ago and this was the first crash since the update.

I'm using this external SSD: https://www.amazon.de/dp/B085TL8W6V?psc=1&ref=ppx_yo2ov_dt_b_product_details

[zillion42]

After the latest 'incident' I accessed the OS via SSH

I can't SSH into my pi anymore after it hangs.
Sending logs to a remote host with Rsyslog would really be helpful.

[markusmauch]

I can't SSH into my pi anymore after it hangs. Sending logs to a remote host with Rsyslog would really be helpful.

Me neither but after a reboot you can still see the old logs using journalctl.

[zillion42]

I know this is not very helpful for solving the problem, but downgrading to 9.5 really helps, just in case you need HA to be running stable again.

ha os update --version 9.5

[mundschenk-at]

@mundschenk-at your case is really off-topic here. This thread is about issues with NVMe and USB adapters connected to a Raspberry Pi.

That was not clear before. I'll create a separate ticket.

[mundschenk-at]

I can't SSH into my pi anymore after it hangs. Sending logs to a remote host with Rsyslog would really be helpful.

Me neither but after a reboot you can still see the old logs using journalctl.

That unfortunately does not help much when the disk with those logs becoming offline is the issue at hand.

[HumanSkunk]

Same issue here when using 10.1. I have a usb adapter to a 2.5 SSD (not an NVMe). I went back to 9.5 a month or so ago and it’s been solid. I came to check the bug reports to see if others had the same issue and it seems so. This reminds me of an issue back in late 2020 early 2021 where it came down to an issue with the Pi firmware after months of people debugging differences.

[markusmauch]

Okay, went back to 9.5 a week ago and it's running stable ever since. Do you see a problem staying there for a longer period of time? Doesn't look like there will be a fix in the near future?

[danir-de]

This still is a problem with the latest 10.3 release.
I've upgraded back to the latest version, after rolling back for a few weeks, since I want to use some of the new features. I'm trying a workaround by restarting the host twice a day via an automation.

[markusmauch]

• I thought maybe one of you might be able to give me a rundown how to enable ssh access on 22222

Just follow the procedure described here:

https://developers.home-assistant.io/docs/operating-system/debugging/

It looks worse than it is and is very useful in many occasions...

[zillion42]

• I formatted a usb stick FAT, created a folder CONFIG <-Match Case,
• inside that folder I created a file called "authorized_keys" no extensions,
• I then pasted my public key inside that file, making sure its ANSI formatted and uses LF instead of CR

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXsruscncaScl6oRh92F0Htrq+1urxUo2ySoHV90zvqu7ZJekgLrKnAVh1A8WupXhOxSIApllUERGtMNfW+F9tfXnsqkwK93AuQ4n6bkb4fWf1hiSfM1jfvGLlwxjmrYcttJSGUxBLPEgmsGGkmmcy6+2S73iuqKecAVuYwZn5aSSaOvJHuAkKBdcyXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXyWQdHDZDCP0qWcsS6ec9HcVyF+sMwTqItqHj8K8rZkTefxwGg5A+OlW3n4tcmteq0bjSqSQrRCNqg3N+xu1JPVW9LOHWHKujVP2Ttyr7+xOwb3V7AlOq7tc155qY2TkSOrP2eLBApJz/zVcr/aCO5aYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

• I inserted the USB stick where my SONOFF CC2652P radio lives, last remaining USB Port meaning NO LIGHTS

• I rebooted the Host OS

• I tried to connect to port 22222

• Guys, I'm pulling my hair out here, of course it didn't work

• Next I'll try to access /dev/sda1/boot/cmdline.txt from another OS

[zillion42]

tobi@introvision2:~$ sudo lsblk
[sudo] password for tobi:
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
...
(blah)
...
sda           8:0    0 119,2G  0 disk
├─sda1        8:1    0    32M  0 part
├─sda2        8:2    0    24M  0 part /media/tobi/disk
├─sda3        8:3    0   256M  0 part /media/tobi/disk3
├─sda4        8:4    0    24M  0 part /media/tobi/disk1
├─sda5        8:5    0   256M  0 part /media/tobi/disk2
├─sda6        8:6    0     8M  0 part
├─sda7        8:7    0    96M  0 part /media/tobi/hassos-overlay
└─sda8        8:8    0 118,6G  0 part /media/tobi/hassos-data
nvme0n1     259:0    0 931,5G  0 disk
├─nvme0n1p1 259:1    0   100M  0 part /boot/efi
├─nvme0n1p2 259:2    0    16M  0 part
├─nvme0n1p3 259:3    0   465G  0 part /media/win10
├─nvme0n1p4 259:4    0   614M  0 part
└─nvme0n1p5 259:5    0 465,8G  0 part /

tobi@introvision2:~$ sudo mount -t vfat /dev/sda1 /mnt

tobi@introvision2:~$ ls -l /mnt/
total 6652
-rwxr-xr-x 1 root root   52656 Aug  1 19:29 bcm2711-rpi-400.dtb
-rwxr-xr-x 1 root root   52524 Aug  1 19:29 bcm2711-rpi-4-b.dtb
-rwxr-xr-x 1 root root   53265 Aug  1 19:29 bcm2711-rpi-cm4.dtb
-rwxr-xr-x 1 root root    2411 Aug  1 19:29 boot.scr
-rwxr-xr-x 1 root root     137 Aug  1 19:29 cmdline.txt
-rwxr-xr-x 1 root root    2160 Aug  1 19:29 config.txt
-rwxr-xr-x 1 root root    3170 Aug  1 19:29 fixup4cd.dat
-rwxr-xr-x 1 root root    5398 Aug  1 19:29 fixup4.dat
-rwxr-xr-x 1 root root    8386 Aug  1 19:29 fixup4x.dat
drwxr-xr-x 2 root root   24576 Aug  1 19:29 overlays
-rwxr-xr-x 1 root root  805436 Aug  1 19:29 start4cd.elf
-rwxr-xr-x 1 root root 2250848 Aug  1 19:29 start4.elf
-rwxr-xr-x 1 root root 2998344 Aug  1 19:29 start4x.elf
-rwxr-xr-x 1 root root  533432 Aug  1 19:29 u-boot.bin

tobi@introvision2:~$ sudo vim /mnt/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 usb-storage.quirks=174c:55aa:u,2109:0715:u,152d:0578:u,152d:0579:u,152d:1561:u,174c:0829:u,14b0:0206:u,0bda:9210:u
:x

tobi@introvision2:~$ sudo umount /dev/sda1

[zillion42]

Success for blacklisting UAS.

I will now update to 10.3 and report stability.

[zillion42]

I'm going to stay, optimistic here.
If I come home, back from work, tomorrow evening, and all lights wont work, again, because my SSD became unresponsive again...

How hard can it be to blacklist: 0bda:9210 Realtek Semiconductor Corp. RTL9210 M.2 NVME Adapter for the @HA_dev_Team?
I've been struggling since May 5th.

EDIT: Pi4 SSD USB Boot support is STILL broken with kernel 6.1
EDIT2: THX @Baxxy13

[zillion42]

@markusmauch
if you really have the same hardware, try:
lsusb
That should say 0bda:9210 somewhere

Add:
,0bda:9210:u to the end of your /mnt/boot/cmdline.txt. Add the comma, add the :u

[markusmauch]

if you really have the same hardware, try:
lsusb
That should say 0bda:9210 somewhere

I can confirm this. I changed the /mnt/boot/cmdline.txt accordingly and updated to 10.3. I'll keep you informed...

[Baxxy13]

because my SSD became unresponsive again...

Sad to hear that. I really hoped blacklisting the uas-mode works.
But ist seems that the active uas-mode is not the problem.

Searching for...

[    2.020055] usb 2-2: Enable of device-initiated U1 failed.
[    2.020983] usb 2-2: Enable of device-initiated U2 failed.

leads to articles about problematic U1/U2 implementations which had to do with LPM (low power mechanism) of USB3-Devices.
e.g. here

Disabling LPM for your USB-Device might be worth a try.
What i have read, this could also be done via quirks in cmdline.txt
See here in the answer
But i never used or tested this and i don't know if it's supported by HA-OS.

[zillion42]

@Baxxy13, I think you got me wrong. I said, "if"

I'm going to stay, optimistic here.
If I come home, back from work, tomorrow evening, and all lights wont work, again, because my SSD became unresponsive again...

All is peachy so far:

[SamuelCarson]

this is really frustrating.. I can't use network storage while on 9.5, I need to be on 10.0+. No one is assigned, I anyone from the development team even looking at this?

[nheuermann]

I had the same problem, wasn't even able to startup HA on Raspi4, first it keeps checking the file system and then SquashFS errors flood the logs.
After taking my Kingston SSD from the ICY BOX IB-AC703-U3 USB adapter and putting it into my older HDD/SSD adapter (one to put them in vertically, Sharkoon QuickPort USB 3.1 Type C) everything worked fine immediately. Latest HA version.
I guess it's a (Linux Kernel?) incompatibility with certain SSD USB bridges.

[zillion42]

I unplugged my ssd and ran my old SD card with Raspian for deconz to check my current firmware Version.
It said this:

pi@phoscon:~ $ sudo vcgencmd version
Dec  1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Maybe I should update the firmware ?

pi@phoscon:~ $ sudo rpi-eeprom-update
*** UPDATE AVAILABLE ***
BOOTLOADER: update available
   CURRENT: Tue 26 Apr 2022 10:24:28 AM UTC (1650968668)
    LATEST: Wed 11 Jan 2023 05:40:52 PM UTC (1673458852)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1

If so:

• Is there anything I need to consider when booting from USB-SSD again?
• Can I just boot from SD update firmware with:

sudo rpi-eeprom-update
sudo apt update
sudo apt upgrade
sudo rpi-eeprom-update -d -a
sudo reboot

Or might I run into similar problems as this guy?
https://community.home-assistant.io/t/raspberry-firmware/466510

Thx for letting me know, I will now boot back into my SSD and await your reply

P.S. Might this eventually help with SSD issues for kernel 6.1 on raspberry pi ?

[markusmauch]

Add ,0bda:9210:u to the end of your /mnt/boot/cmdline.txt.

This did do the trick. My system ran stable on 10.3 for a whole week. I updated to 10.4 yestarday and so far it looks good.

Thanks for the assistance!

[agners]

this is really frustrating..

Tell me about it, it really is! Just search this bug tracker for issues with the label usb-ssd. Just crazy, how many issues piled up until today.

We've been tempted to declare USB SSD unsupported entirely. But on the other hand, there are configurations of USB SSD adapters + disk + Raspberry Pis which do work really reliable. It just seems a huuuge hit and miss. 😢

I can't use network storage while on 9.5, I need to be on 10.0+. No one is assigned, I anyone from the development team even looking at this?

I follow it loosly.

In the end, Raspberry Pi and USB SSD has a painful journey since.. forever, essentially. It was one reason we created Yellow: Proper NVMe SSD using M.2/PCIe did help to alleviate a lot of the problems. This is the technology PCs and Notebooks are using, and sidesteps all the USB powering and USB UMS protocol issues. Granted it seems that the Raspberry Pi SoC also has troubles to talk to some high end NVMe's such as Samsung 980 Pro and WD_BLACK NVMe SSD, unfortunately 😢 , but pretty much every other NVMe really works rock solid.

That said, I'd really hope that USB SSD support gets more stable as time progresses. But we rely on the progress of the Raspberry Pi kernel. I am waiting for a new Raspberry Pi Linux kernel release, but it seems they stopped releasing regularly, their last release was in April, see https://github.com/raspberrypi/linux/tags 😩

[Baxxy13]

Just thinking...
Most Pi4B-USB-Boot problems comes from the uas-mode of the usb driver.
Blacklisting uas-mode for known problematic adapters via /mnt/boot/cmdline.txt works.
But the "Blacklist" isn't frequently updated and even if so, an HA-OS Update doesn't update the /mnt/boot/cmdline.txt as far as i know.
So only fresh installations of HA-OS would benefit from an actual "Blacklist".

Why not disabling usb-uas mode totally for the Pi4B HA-OS?
I never heard of problems with USB-Boot if the usb-driver uses usb-storage instead of uas.
(other issues like insufficent power adapters, bad cables etc. not included)

Ok, performance throughput is a bit lower with usb-storage but this isn't really appreciable within the running HA-OS.
As said earlier, better slightly lower performance and stable, than crashing and unstable system.

Sidenote from my last adapter-testing:
If the same adapter (with the same ssd) is running in usb-storage mode (instead of uas) the whole system consumes slightly lesser power.

[HumanSkunk]

For the sake of reliability, stability and my sanity is there an easy migration path to the other supported solution with SSDs with having HAOS installed on the SD card and HA on the SSD? Would it be a case of creating a backup in my current configuration of it all on an SSD, installing a new instance direct on an SD card, restore the backup to the SD card and then migrate the data disk back to the SSD? Or would it be better to migrate the data disk and then restore the backup? The back up is universally supported as long as its a supervised instance of HA right regardless of what hardware and configuration i stick it on?

[fromNL]

I just installed my new RaspBerry 8GB 4+ and wrote an SSD (a cheap brand, LITEON or something like that, data-disk: USB3.0-Super-Speed-DD564198838B0) with Balena Edger. It would not boot, it got stuck at SquashHfs errors.

Then I wrote an SD card and tested: the desktop runs just fine, so the sbc is okay.

After reading all of the above (to the letter) and rewriting the SSD 3 times, trying and trying while I am reading, in the end I came up with my own solution. I pushed the USB plug not in the USB3 (blue) outlet but in the USB2.0 outlet of my RaspBerry.
Now it boots!
I got passed those SquashHfs problems (they did not show up), and I am now on a prompt
ha >
and I can even open http://homeassistant.local:8123/

I do not think I need to update more here, I did a temporary registration, and I solved the problem for myself.
Note: I did have at hand several other (AT) SSD to USB enclosures/cables lying around for further testing, but I am okay with USB 2.0 (unless it is too slow in the future, after setting up my h.a.).

Note: my H.A. version is 10.4, downloaded and flashed it today.

If changing the USB cable had not worked, I would have gone for the docker solution. I would boot from SD and from there on use docker on the SSD (if that would work, I have not tested it and leave it here for others as an idea).

[zillion42]

Just to report back,

my system runs stable as well, since I blacklisted UAS in /mnt/cmdline.txt
Not my favorite solution, but better than no solution:
@fromNL you can use usb3
@HumanSkunk I don't know how to restore your data
@Baxxy13
^yes, that seems the best solution so far.
Reliability OVER Speed

EDIT:
Does anyone think it would help to update my pi4 firmware?

[DaniEll-AT]

I would like to report my experience:

• I had a perfectly stable Pi4 with USB-attached SSD (no SD-card) on HA OS 9.5
• After updating to HA OS 10.1 the system seemed to work but after some time became unresponsive like described in this issue (this comment)
• I rolled back to HA OS 9.5, all was fine again.
• After updating to HA OS 10.5 the problem is gone without any further changes (stable for a week now)

After i found this issue on GitHub, i checked UAS. With HA OS 9.5 it was using usb-storage, with 10.5 it does so too.
Because this is my live system, i am not going to test 10.1 again.

[ZdenekM]

@DaniEll-AT I'm on 10.5 and it still happens... Maybe it's a different issue but the symptoms seem to be the same.

[ZdenekM]

Downgraded to 9.5, and so far, so good...

[danir-de]

Otherwise (not blacklisted) you can blacklist your adapter by yourself by editing
/mnt/boot/cmdline.txt and adding your adapter's VendorID:ProductID at the end of the line.
(see cmdline.txt)

This also worked for me, I've been running it rock stable for over a month now. Also not my favorite solution, it also seems slower on bigger operations like updating and creating backups but everything else seems to be running just as before.

For everyone still suffering, try this: #2536 (comment)

If it's not helping, maybe you have another problem?

[ZdenekM]

Anyone tried with 11.x?

[TheOriginalMrWolf]

I am running

• Core 2023.11.1
• Supervisor 2023.10.1
• Operating System 11.1
• Frontend 20231030.1
• Samsung 1TB T7 USB SSD (UASP mode, USB3 socket)
  • I only use T5 & T7 SSDs on various projects on RPi and Odroid N2 as I've found them to be extremely reliable & performant
  • Manually enabled FSTRIM
• Raspberry Pi 8GB + original RPi plugpack PSU
• Wired LAN
• ESP8266/NodeMCU running ESPHome also connected via USB 2 (for power & fallback access as this is a remote site)

Survived several core & OS upgrades, working perfectly so far for around 3 months (touch wood 🤞 - hope I haven't cursed myself by mentioning this...).

[markusmauch]

[    2.016595] usb 2-2: New USB device found, idVendor=0bda, idProduct=9210, bcdDevice=30.00
[    2.016638] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.016670] usb 2-2: Product: GV100- 128G
[    2.016697] usb 2-2: Manufacturer: ORICO

Does somewone know if it's still necessary to blacklist the Orico SSD with Home Assistant OS 11.1?

[github-actions]

There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant OS version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[zillion42]

@agners
I'm facing the exact same problem again, only this time it's a pi5 with a pcie ssd hat.

It crashes far to often, only I cant blacklist any usb adapter, since its pci express.

I'm pretty sure I should open a new thread for this, could you point me in the right direction?

# cat /mnt/boot/cmdline.txt

zram.enabled=1 zram.num_devices=3 rootwait cgroup_enable=memory fsck.repair=yes console=tty1 root=PARTUUID=8d3d53e3-6d49-4c38-8349-aff6859e82fd rootfstype=squashfs ro rauc.slot=A systemd.machine_id=3514abb663c6460392093ebcb3449919

# lsusb
Bus 003 Device 001: ID 1d6b:0002
Bus 001 Device 001: ID 1d6b:0002
Bus 004 Device 001: ID 1d6b:0003
Bus 004 Device 002: ID 346d:5678
Bus 002 Device 001: ID 1d6b:0003
# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1  58.6G  0 disk
`-sda1        8:1    1  58.6G  0 part
zram0       254:0    0     0B  0 disk
zram1       254:1    0    32M  0 disk
zram2       254:2    0    16M  0 disk /tmp
nvme0n1     259:0    0 119.2G  0 disk
|-nvme0n1p1 259:1    0    64M  0 part /mnt/boot
|-nvme0n1p2 259:2    0    24M  0 part
|-nvme0n1p3 259:3    0   256M  0 part /
|-nvme0n1p4 259:4    0    24M  0 part
|-nvme0n1p5 259:5    0   256M  0 part
|-nvme0n1p6 259:6    0     8M  0 part
|-nvme0n1p7 259:7    0    96M  0 part /var/lib/systemd
|                                     /var/lib/bluetooth
|                                     /var/lib/NetworkManager
|                                     /etc/systemd/timesyncd.conf
|                                     /etc/hosts
|                                     /etc/hostname
|                                     /etc/NetworkManager/system-connections
|                                     /root/.ssh
|                                     /root/.docker
|                                     /etc/udev/rules.d
|                                     /etc/modules-load.d
|                                     /etc/modprobe.d
|                                     /etc/dropbear
|                                     /mnt/overlay
`-nvme0n1p8 259:8    0 118.5G  0 part /var/log/journal
                                      /var/lib/docker
                                      /mnt/data
#

[zillion42]

#3188