Firmware update procedure requires to completely wipe SD card

[jens-maus]

Currently, upon a new RaspberryMatic release all users still have to completely wipe their SD card, update it with the next image and then restore a backup. This is quite cumbersome and error prone. For proper updates in future we should support normal "firmware updates" like this is currently possible for the old CCU2 hardware. For this, we have to lookup how this is currently done and how we can automatically generate similar "firmware update" archives users can then directly upload via the WebUI.

[j-a-n]

I began to analyze the original firmware update process on the CCU2:

1. firmware update archive (.tgz) is uploaded via /www/config/cp_maintenance.cgi to /tmp/
2. the files update_script, EULA.en, EULA.de and EULA.tr are extracted from the uploaded archive to /var directory
3. update process aborts if update_script is not present in archive
4. firmware update archive is moved from /tmp to /var/new_firmware.tar.gz
5. EULA is displayed and has to be accepted by user
6. If user accepts EULA the system reboots
7. the new_firmware.tar.gz seems to get extracted at boot and the update_script is executed
   original update_script:

#!/bin/sh

STARTPWD=$(pwd)
ROOT_FS=rootfs.ubi
KERNEL_IMAGE=uImage

ROOT_MTDCHAR=/dev/mtd7

KERNEL_MTDCHAR=/dev/mtd5

if [ -f $STARTPWD/$KERNEL_IMAGE ]; then
 	echo "Erasing kernel partition ..."            
 	flash_erase $KERNEL_MTDCHAR 0 0                      
 	echo "Writing kernel to NAND ..."         
  	nandwrite -p $KERNEL_MTDCHAR $KERNEL_IMAGE
fi

if [ -f $STARTPWD/$ROOT_FS ]; then
  	echo "Writing rootfs to NAND ..."
	ubidetach -p $ROOT_MTDCHAR
	ubiformat $ROOT_MTDCHAR -f $ROOT_FS
fi

Changes needed for RaspberryMatic:

1. cp_maintenance.cgi uses tar to extract files from the firmware update archive.
   The tar version included in RaspberryMatic needs different parameter order:
   original (not working):
   exec tar zxvf [lindex $firmware_file 0] update_script EULA.en EULA.de EULA.tr -C /var/
   working command:
   exec tar zxvf [lindex $firmware_file 0] -C /var update_script EULA.en EULA.de EULA.tr

2. There is no mechanism in RaspberryMatic which would process a new_firmware.tar.gz on sys boot.
   Possible (simplest) solution: Start firmware update from running system (cp_maintenance.cgi)

Proof of concept:

This patch for cp_maintenance.cgi corrects the tar command and starts update_script right before system reboot:

--- WebUI/www/config/cp_maintenance.cgi.orig	2017-03-08 23:04:47.141292627 +0100
+++ WebUI/www/config/cp_maintenance.cgi	2017-03-09 23:43:02.329296667 +0100
@@ -753,7 +753,7 @@
     set file_valid 0
     catch {
         #set file_valid [expr [string first "update_script" [exec tar tzf [lindex $firmware_file 0]]] >= 0 ]
-        exec tar zxvf [lindex $firmware_file 0] update_script EULA.en EULA.de EULA.tr -C /var/
+        exec tar zxvf [lindex $firmware_file 0] -C /var update_script EULA.en EULA.de EULA.tr
     }
     set file_valid [file exists "/var/update_script"]
 
@@ -864,6 +864,7 @@
     catch { exec killall hss_lcd }
     catch { exec lcdtool {Saving     Data...    } }
     rega system.Save()
+    catch { exec /bin/sh /var/update_script 2>&1 >/var/log/upgrade.log }
     catch { exec lcdtool {Reboot...             } }
 
     if {[isOldCCU]} {

The update_script uses dd to overwrite the boot and root filesystems on the sdcard (which are mounted while overwriting but since they are mounted readonly this seems to work reliable).

Here is a bash script which creates an update archive from a RaspberryMatic release archive (RaspberryMatic-X.XX.XX.YYYYMMDD.zip).

make_firmware_update_archive.sh

The script relies on losetup, kpartx, mount and dd and has to be executed as root:
sudo bash make_firmware_update_archive.sh RaspberryMatic-2.25.15.20170114.zip

The resulting archive then can be used to update the system in combination with the patched cp_maintenance.cgi via the WebUI.

[jens-maus]

@j-a-n First of all, thanks for this very nice summary (actually, I tried to improve the layout of your comment using GitHub markup functionality - :) and the nice proposal to actually implement this

In fact, I was of course also already thinking about how we could potentially support a direct update via the standard "Firmware Update" mechanism like in a standard CCU2 system. However, reading through your proposal I still feel that while this could actually work there might also be some pitfalls and though that we should discuss first:

1. the main problem I see with your approach is the direct use of dd to overwrite the /boot and /root filesystems. While for the exact same size of the root/boot fs this would perfectly work it will fail with different sizes of the /boot and /root filesystem between the current version of RaspberryMatic and the version to be installed. Thus, using dd to simply overwrite the existing partitions might actually result in accidentally overwriting the /usr/local user partition in case the new root partition is larger than in previous versions.
2. Furthermore, using dd on partitions that are already mounted (even read-only) might result in an unstable system because some other process might actually read new data from these partitions while they are being overwritten and thus might cause a kernel crash or other severe problems.
3. As you correctly stated in your rational, the firmware update file is first being uploaded to /tmp, which actually resists in RAM. While for a pure firmware update the 1GB RAM of the Pi2/3 might be perfectly enough, this poses the same problem like with the current Backup and Addon upload/install routines which might thus cause the same problems like discussed in Backup/Restore and Addon install not working for filled up /usr/local partition #23

As said, I really appreciate your work/analysis of the possibility and I really hope we can find a nice solution together how firmware updates could be performed in future in RaspberryMatic. As such I would like to raise the following proposals:

1. We could actually think about introducing a dedicated "update partition" just for the sake of performing the updates and if a user upload the firmware update, RaspberryMatic will then put the update firmware files to this partition and actually boot from this partition instead to perform the update (overwriting boot and root fs with new data, potentially resizing /usr/local, etc. etc.). And then RaspberryMatic will reboot again into its normal and updated root fs.
2. In theory one could also think about disabling the old firmware update mechanism in the WebUI all together and simply tell all users to always perform an update like it is currently done. Because if you think about all the work that would be required to implement a nice and error safe update mechanism it would actually be probably faster for users to always perform an update by completely wiping the SD card and restoring their backup afterwards. IMHO this would then also force users to perform a nice full backup. And as they had already used the software to actually flash the RaspberryMatic *.img file onto their SD card they should be trained how to do that again. Thus, it actually could also be quite a viable approach to simply say that this is the official way of updating RaspberryMatic because it is quite straight forward IMHO. But for this to be actually the official way of updating RaspberryMatic one would have to first fix the backup/restore mechanism (see Backup/Restore and Addon install not working for filled up /usr/local partition #23) to be able to deal with a /usr/local partition and backup file that might be larger than the /var partition size currently provides.

So what do you think?

[j-a-n]

Thanks for formatting my post ;)
My comment was meant as a starting point for a discussion and not yet a perfect solution.

In my opinion an update has to be as simple as possible to ensure that users actually are upgrading their systems to recent versions which possibly fix security problems.
Your proposal of a dedicated update partition seems to be a very good but also extensive solution.

What is the main reason to put /tmp in RAM?
What about a dedicated partiton for /tmp (and /var) which gets emptied at each boot?
This would also open up the possibility to process a new_firmware.tar.gz file at boot and would fix #23.

I don't think that /usr/local gets accidentally overwritten because dd would just exit with No space left on device.
Here are some ideas to improve the update process described in my previous post:

• Stop all daemons before running dd to prevent read errors while writing the image to the sdcard.
• Increase root and boot partition to reserve space for future upgrades.
• Check sizes of the root and boot partiton before starting dd and cancel upgrade if image is to big.
• Use something like partimage instead of dd.
  This would reduce the images to the actual used filesystem space and improve handling of changing filesystem sizes.

[jens-maus]

Well, the reason why /tmp is in RAM is not only because this automatically wipes all content away but also because this speeds up operation for all applications putting data into /tmp or /var. And this is actually something very common. Furthermore, if you put /tmp and /var back onto the SD card this would put additional load on the SD card and will thus reduce its life time considerably. Thus, simply putting back /tmp and /var on the sd card to also solve #23 isn't really a viable solution IMHO.

Of course /usr/local might not get overwritten if the dd would be only be performed on the partition itself, but as a matter of fact RaspberryMatic comes with root and boot partition sizes as small as possible and thus in case these sizes are increased considerably due to more functionality being put in the image at some later point will render your whole dd driven update mechanism obsolete (because it is really not easy to decide which might be the optimal maximum size of for the root partition).

In addition, I still feel that using dd over a currently actively mounted partition (no matter if read-only mounted or not) isn't really a good idea. This IMHO would definitly cause some strange and unwanted problems and should be avoided.

Thus, I am still not sure if your proposed dd approach would be really the best way of integrating a firmware update mechanism similar to what the CCU2 itself provides. IMHO, compared to that the current approach of forcing users to completely create a new SD card and restoring a previously created backup is probably still more safe and straight forward. And in case of a RaspberryPi (compared to a CCU2) you can easily keep two SD cards ready for being used in case of an update so that you can easily and quickly revert if the update didn't work as expected. So I am really still not convinced that the methods you are proposing will help end users to perform an update more easier and in a more safe way. Any other ideas?

[j-a-n]

Whatever method is used to update the filesystem on the partition, the problem with the partition size will persist.
If you really want to handle the partition size dynamically things are getting complicated.
Possible solutions are LVM or Btrfs. In my opinion this is overkill.
Why not fix the partion size of boot and root partion to "likely big enough" (i.e. 100M and 4G).
This would simplify things.

One more possibility to update the filesystem on the running system could be rsync.
Loop mount the update images, remount the existing filesystems rw and then rsync with delete.

Another approach could be:
2 root partitions and a boot mechanism which is able to boot system 1 or system 2.
Update process would then update the inactive partition and switch boot mechanism over to the new system. The kernel would has to be moved to the root partition and a boot loader like uboot needs to load either the one or the other system.
If the updated system is not working as expected, one could revert easily to the old system.

[j-a-n]

For now I build an addon which implements a simple up/-downgrade mechanism using rsync.
Before the addon can be used, it is needed to increase the filesystems on the SD card because some extra space is needed for rsync temp files and future RaspMatic versions will have to fit into the existing filesystems.
The easiest way is to achieve this is to download the provided RaspMatic image with different partition sizes (100M boot and 1900M root).
The addon can be found here.
Everybody is invited to test this addon.
Please use the HomeMatic-Forum or Github to provide feedback.

[jens-maus]

@j-a-n First of all thank you for your contribution to the RaspberryMatic project. I really appreciate any help in getting RaspberryMatic more and more widely used and on an equal technical level like a CCU2.

However, I still have very deep doubts that your add-on approach is the right way to go. There are some very fundamental doubts I already raised in this thread (e.g. overwriting /root while it is still mounted, etc.). Furthermore, your approach requires different RaspberryMatic images which you alter yourself and which you seem to distribute yourself as well. This will IMHO just confuse end users and might lead to problems where I will e.g. receive bug reports where users are using your images where in this case I will have to reject any bug report because I hope you understand that I cannot support these kind of adapted images. Furthermore, you will have to update your adapted images to every new RaspberryMatic version I will release in future and thus this might not only introduce additional delay for end users but might also result in technical problems.

Thus, I feel that while your rsync-driven approach in providing updates looks like a nice technical approach (apart from the initial problem that overwrites on a root file system while being mounted isn't really something I would suggest), I really deeply feel continuing your work as an Addon isn't really something I would advice. A firmware update ist such a fundamental task that it shouldn't be part of an Addon package. Thus, I would have preferred that we instead would have continued our discussions here and that you might have come up with a bunch of pull requests so that we could integrate it directly into RaspberryMatic (first in terms of an additional branch and then after having sorted out technical things we could have merged it to master).

Furthermore, you have never commented on my initial comments that I currently still feel that having to wipe the whole SD disk isn't something utterly complicated. There are some fundamental differences between a CCU2 hardware and RaspberryPi driven system (e.g. being able to exchange microSD card with root fs) that raises the question if the current update path (backup->SD card wipe->restore) is really soo bad!?! But don't get me wrong, if we can find a nice, smooth solution to provide a similar level of updating raspberryMatic directly via its web interface I am all for it. I just feel that 1. overwriting the root fs while it is currently mounted isn't a wise approach and 2. stuffing all this into a third-party add-on shouldn't be continued. So please consider preparing pull requests for the RaspberryMatic project instead.

[j-a-n]

@jens-maus First of all thanks for all your work, I am using RaspberryMatic for several weeks now and it is working really stable and fast. It is really a big improvement in comparison with the CCU2.

The current update path is not really complicated but I prefer not having to touch the RasPi and just press a button in the webinterface to upgrade the system. This process could also be automated completely.

There is not really a need for different RaspberryMatic images, we just need bigger partitions/filesystems. The addon downloads the original RaspberryMatic images.

Rsync is able to replace in-use binaries. In my opinion the rsync approach is working really nice so far.
I would be pleased if you would try the addon and share your opinion.

[jens-maus]

@j-a-n I will certainly try it soon and see how it works and test how stable the rsync approach really is. However, I really still feel instead of distributing this solution solely as an add-on we should continue and provide a solution which is directly integrated in RaspberryMatic instead. And increasing the root and boot partition to larger fixed sizes shouldn't be a problem directly in RaspberryMatic. So what do you think about transforming your rsync add-on approach over into something more integrated directly in RaspberryMatic? I would really appreciate it if you could provide pull requests instead of continuing your add-on approach.

[j-a-n]

If you think the rsync approach is worth it, I also think it would be great to integrate it into RaspberryMatic.

[jens-maus]

I would say, go ahead and prepare a pull request for integration of your rsync approach and then we can evaluate how well it actually works as soon as it is integrated.

[renne]

I suggest to use GIT. This allows to update RaspberryMatic from the Github repository without the need of running additional servers (e.g. rsyncd).

[chrostek]

Version 1.0 from j-a-n's addon uses two partitions. this solution is a very good way in my opinion. it would be great if you could integrate this in the core system.

[jens-maus]

Well, I told @j-a-n already that instead of working on an add-on doing such an integral job he should work/concentrate on working on a pull request to potentially integrate his changes in a future version of RaspberryMatic.

[j-a-n]

@jens-maus I would need your support to start the integration.
If you could adopt the needed partition layout (I really do not have any idea how this is done with buildroot), I could start to submit the first pull requests.
Currently the adapted image uses the following layout:

partition 1: size=100M, label="bootfs"
partition 2: size=1000M, label="rootfs1"
partition 3: size=1000M, label="rootfs2"
partition 4: label="userfs"

[jens-maus]

@j-a-n I would love to help you or even explain you how to work with build root. Looking at you project I think it should be quite easy for you to get started with build root once you are starting to work with it.

Regarding integration I think we first need to talk over some things:

1. One thing that came immediately into my mind is the doubled size your partitioning scheme is causing. I think this is quite critical because I really want to keep the img size as small as possible. One approach would be to just deliver RaspberryMatic with a fake rootfs2 partition as small as the delivered userfs and then resize / repartition it upon the first boot of RaspberryMatic.
2. How exactly are you extracting a new rootfs from a new img file once a user uploaded it via WebUI?

[j-a-n]

1. What are your exact concerns about the image size? The duration of the write process? If parted is able to move a ext4 partition, the following should be possible:

• Move user partition and resize
• Move root2 partition and resize
• Resize root1 partition

2. Current update process in short:

• zip file is automatically downloaded from github (to userfs)
• img file is extracted from zip (to userfs)
• some checks are performed (sufficient partition sizes, etc.)
• loop mount new rootfs from img
• mount unused rootfs1 or rootfs2 and rsync from img
• loop mount new bootfs from img
• remount bootfs rw and rsync from img
• patch cmdline.txt to use updated root partition

[jens-maus]

Please note that I have created a new "webui-upgrade-feature" branch and started integrating your proposed functionality in using two swappable rootfs partitions. Now, upon its very first boot raspberryMatic should create a rootfs2 and userfs partition on the fly so that we don't have to supply it via the downloadable *.img file.

Next will be to reenable the webui firmware upgrade components and then we would have to integrate your proposed loop mount and rsync mechanisms to actually perform the web-based upgrade.

So please checkout this new branch and test it as I will add new functionality to it. And of course, feel free to submit pull requests against it.

As soon as this branch stabilized we can then consider merging it to master to have this feature integrated in one of the next RaspberryMatic versions.