use bmap files to skip flashing unused blocks #394

dnbazhenov · 2023-10-07T20:09:51Z

Use .bmap files created by WIC tool the same way as bmaptool does, i.e. skip flashing irrelevant blocks.
This is a -raw2sparce extension.

By default, a file with the same base name but with "bmap" extension would be searched to get block map from.
An option '-no-bmap' would inhibit the search.
Also, a specific bmap file can be specified in the flash command via '-bmap' option.

libuuu/CMakeLists.txt

libuuu/bmap.h

libuuu/bmap.cpp

libuuu/sparse.cpp

nxpfrankli · 2023-10-19T15:36:12Z

Can you fixed CI build error?

/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp: In function ‘bool parse_image_size(bmap_t&, const tinyxml2::XMLElement*)’:
/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp:42:24: error: ‘const class tinyxml2::XMLElement’ has no member named ‘Unsigned64Text’; did you mean ‘UnsignedText’?
42 | auto img_size = elem->Unsigned64Text();
| ^~~~~~~~~~~~~~
| UnsignedText
/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp: In function ‘bool parse_block_size(bmap_t&, const tinyxml2::XMLElement*)’:
/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp:53:24: error: ‘const class tinyxml2::XMLElement’ has no member named ‘Unsigned64Text’; did you mean ‘UnsignedText’?
53 | auto blk_size = elem->Unsigned64Text();
| ^~~~~~~~~~~~~~
| UnsignedText
/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp: In function ‘bool parse_blocks_count(bmap_t&, const tinyxml2::XMLElement*)’:
/home/runner/work/mfgtools/mfgtools/libuuu/bmap.cpp:64:25: error: ‘const class tinyxml2::XMLElement’ has no member named ‘Unsigned64Text’; did you mean ‘UnsignedText’?
64 | auto blk_count = elem->Unsigned64Text();

ondrejhennel · 2023-10-23T09:40:47Z

Hi. I tried to use this PR as we have a very large image with a lot of empty space.

I built it on linux (wsl) first. At first it did not work with either the builtin scripts or a custom one. This is because the version number generated from tags was incorrect. After removing the tags "uuu_db_bmap" and db_bmap", it worked correctly and sped up the flash significantly.

Then, I tried to build for Windows. Same issue with tags. It also couldn't find tinyxml2.h even with the submodules checked out and I had to install tinyxml2 using vcpkg. After this, the build suceeded, but the binary did not work correctly. The flashing finished after a few seconds (way too fast) and the image in flash was corrupt and could not boot.

There seems to be an issue that occurs only on Windows and not on Linux.

nxpfrankli · 2023-10-23T15:01:07Z

@ondrejhennel Thank you for your testing this PR.

mothacehe · 2023-10-31T14:39:50Z

On Linux, with the bmap file available, with the same name as the image, it works well and the flashing time is considerably reduced (thanks!).

However, if modify the flashing script this way:

# @_image   [_flash.bin] [_bmap] | wic image burn to emmc.
...
FB: flash -raw2sparse -bmap _bmap all _image

I see:

3:644>Start Cmd:FB: flash -raw2sparse -bmap /home/mathieu/tmp/s0-image-imx93-var-som.wic.bmap all /home/mathieu/tmp/s0-image-imx93-var-som.wic.zst/*

which is expected, but then it fails:

3:644 4/ 6 [partition does not exist ] FB: flash -raw2sparse -bmap /home/mathieu/tmp/s0-image-imx93-var-som.wic.bmap all /home/mathieu/tmp/s0-image-imx93-var-som.wic.zst/*

dnbazhenov · 2023-11-05T07:20:11Z

@nxpfrankli @ondrejhennel I can't deduce how the tags thing should work. The build is not working without a tag.

ondrejhennel · 2023-11-05T09:25:34Z

@nxpfrankli @ondrejhennel I can't deduce how the tags thing should work. The build is not working without a tag.

During the build, git describe is used to generate a version number from the latest tag. The build will be successful if there is any tag. However, the version number is used to check compatibility with the scripts. If there is a tag like uuu_db_bmap, uuu will think it is incompatible with the builtin scripts and fail to run.

I think the easiest solution to make the build work is to just use the tags that are in the original repo and not add any others.

The Unsigned64Text call is a recent change in the TinyXML while the previous versions of the library don't implement it. For backward compatibility with TinyXML v2.7 which is only available on Ubuntu 18 and other OS, replace the call to Int64Text.

* restored the original flash -raw2sparce behavior. .bmap files will be ignored if not explicitly specified in script commands * added -no-bmap global options to ignore .bmap files even if those are specified in script commands * added -bmap global option to try using .bmap files even if thise are not specified in script commands

dnbazhenov · 2023-11-11T13:57:07Z

@nxpfrankli I have fixed CI issues
@ondrejhennel @mothacehe I fixed the found issues. Please, give it another try. Also, note that I restored the default behavior in order to not surprise badly existing automation. Use either -bmap global option or -bmap binding to the flash command.

ondrejhennel · 2023-11-13T11:15:53Z

The flashing with bmap now works correctly on Windows. However, there is some strange behavior with the command-line switches.

Here are some examples:

> uuu -b emmc_all .\imx-boot .\image.wic
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.5.125-15-g5283d01

Wait for Known USB Device Appear...
Error: Missed file name

-----------------------------------------------------------------------------------------------

> uuu -no-bmap -b emmc_all .\imx-boot .\image.wic
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.5.125-15-g5283d01

Wait for Known USB Device Appear...
Error: Missed file name

-----------------------------------------------------------------------------------------------

> uuu -bmap .\image.bmap -b emmc_all .\imx-boot .\image.wic
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.5.125-15-g5283d01

Success 0    Failure 0


2:3      1/ 0 [                                      ]
# gets stuck like this forever

The flags only work correctly in a custom script in a command such as FB: flash -raw2sparse -bmap image.bmap all image.wic or FB: flash -raw2sparse -no-bmap all image.wic. When no flag is specified (FB: flash -raw2sparse all image.wic), Error: Missed file name occurs.

dnbazhenov · 2023-11-16T07:52:42Z

@ondrejhennel Thanks for reporting. I found where I screwed with the command line parsing. Could you try with the fix? I locally verified so it should work as expected.

ondrejhennel · 2023-11-20T11:11:49Z

When using the -bmap command line flag, it still gets stuck like this.

> uuu -bmap .\image.bmap -b emmc_all .\imx-boot .\image.wic
uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.5.125-15-g5283d01

Success 0    Failure 0


2:3      1/ 0 [                                      ]
# gets stuck like this forever

Other than that, it works correctly now.

dnbazhenov · 2023-12-09T06:22:57Z

When using the -bmap command line flag, it still gets stuck like this.
> uuu -bmap .\image.bmap -b emmc_all .\imx-boot .\image.wic
# gets stuck like this forever

This is an incorrect usage. The global -bmap option doesn't take an an argument. I'm not sure thath the stuck is related to the command handler itself.

marcusfolkesson · 2023-12-12T11:26:38Z

I've tested this PR locally with success.

dnbazhenov changed the title ~~use bmap files to skip unused blocks~~ use bmap files to skip flashing unused blocks Oct 7, 2023

adapt bmap files to skip unused blocks

bc4bb98

nxpfrankli reviewed Oct 7, 2023

View reviewed changes

libuuu/CMakeLists.txt Show resolved Hide resolved

dnbazhenov requested a review from nxpfrankli October 8, 2023 14:50

fix CI build broken by newly added bmap support

8328ed6

nxpfrankli reviewed Oct 10, 2023

View reviewed changes

libuuu/bmap.h Show resolved Hide resolved

nxpfrankli reviewed Oct 10, 2023

View reviewed changes

libuuu/bmap.cpp Show resolved Hide resolved

nxpfrankli reviewed Oct 10, 2023

View reviewed changes

libuuu/bmap.cpp Show resolved Hide resolved

nxpfrankli reviewed Oct 10, 2023

View reviewed changes

libuuu/sparse.cpp Outdated Show resolved Hide resolved

dnbazhenov requested a review from nxpfrankli October 14, 2023 05:47

addressed review comments

8e7588b

dnbazhenov mentioned this pull request Oct 14, 2023

uuu FB: flash -raw2sparse never works efficiently #361

Open

dnbazhenov added 7 commits November 11, 2023 13:35

Replce Unsigned64Text with Int64Text.

d6d05c8

The Unsigned64Text call is a recent change in the TinyXML while the previous versions of the library don't implement it. For backward compatibility with TinyXML v2.7 which is only available on Ubuntu 18 and other OS, replace the call to Int64Text.

fixed incorrect bmap range calculation.

bdc1050

added tinyxml2 to macOS.yaml

9f0768c

fixed missing tinyxml2 header path dependency for Win32

428004c

fixed missing libtinyxml2.lib dependency for Win32

7165329

fixed libtinyxml2 output path for Win32

5283d01

fixed flash command line parsing in some cases

b992a96

nxpfrankli merged commit 41e04c7 into nxp-imx:master Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use bmap files to skip flashing unused blocks #394

use bmap files to skip flashing unused blocks #394

dnbazhenov commented Oct 7, 2023 •

edited

Loading

nxpfrankli commented Oct 19, 2023

ondrejhennel commented Oct 23, 2023

nxpfrankli commented Oct 23, 2023

mothacehe commented Oct 31, 2023 •

edited

Loading

dnbazhenov commented Nov 5, 2023

ondrejhennel commented Nov 5, 2023

dnbazhenov commented Nov 11, 2023

ondrejhennel commented Nov 13, 2023

dnbazhenov commented Nov 16, 2023

ondrejhennel commented Nov 20, 2023

dnbazhenov commented Dec 9, 2023

marcusfolkesson commented Dec 12, 2023

use bmap files to skip flashing unused blocks #394

use bmap files to skip flashing unused blocks #394

Conversation

dnbazhenov commented Oct 7, 2023 • edited Loading

nxpfrankli commented Oct 19, 2023

ondrejhennel commented Oct 23, 2023

nxpfrankli commented Oct 23, 2023

mothacehe commented Oct 31, 2023 • edited Loading

dnbazhenov commented Nov 5, 2023

ondrejhennel commented Nov 5, 2023

dnbazhenov commented Nov 11, 2023

ondrejhennel commented Nov 13, 2023

dnbazhenov commented Nov 16, 2023

ondrejhennel commented Nov 20, 2023

dnbazhenov commented Dec 9, 2023

marcusfolkesson commented Dec 12, 2023

dnbazhenov commented Oct 7, 2023 •

edited

Loading

mothacehe commented Oct 31, 2023 •

edited

Loading