[BUG] Signature issue on TGL UP Extreme i11 - openssl3 #99
Comments
|
@plbossart can you attach your FW. Would be nice to diff it's headers against v2.1. |
|
One more thing, is your rimage submodule up to date ? (and in $PATH) |
|
git log --oneline cd rimage/ rimage is not in $PATH. It was not needed before and must be set by scripts. |
|
compilation log firmware |
|
This reminds me this bug fix: 95d887251ee40397300 |
|
Interesting, I compared with mine
|
I implemented Also, please share the manifest differences as shown by BTW |
|
Don't know the root cause, but from the build log, Source content hash is different. This is mine, This is Pierre's, |
|
@keqiaozhang reported a similar issue worked-around with the "scripts/xtensa-build-all.sh -d tgl" option. In my case the debug option doesn't solve anything. |
This is the very first difference and the one we must focus on first. All other differences could be impacted by this. The source hash is not affected by any of these:
So there is really, absolutely no reason for @plbossart's source hash to be different. I have the same |
if |
|
@plbossart can you please first make sure that wc build_tgl_?cc/source_hash/*
1408 1408 57728 build_tgl_gcc/source_hash/tracked_file_hash_list
1408 1408 61688 build_tgl_gcc/source_hash/tracked_file_list
2816 2816 119416 total
md5sum build_tgl_?cc/source_hash/*
0c24b8d8a641391d053cba84ec86f20f build_tgl_gcc/source_hash/tracked_file_hash_list
72028df5e905505089a10e92e4d94781 build_tgl_gcc/source_hash/tracked_file_listgit ls-files src/ scripts/ | md5sum
72028df5e905505089a10e92e4d94781 - |
|
@marc-hb, I built without -d, I have 100% same with yours above |
|
@marc-hb I removed all ignored files and rebuilt, same SHA1 |
|
FWIW, I cloned a clean sof and same results, this difference in source hash is not due to my local setup or files that might have side effects. |
|
Fascinating, so you have the same list of files and same git commit but First, please run If it says Roughly how many files have a different hash? If a small number then which ones?
None of this depends on the toolchain, it's pure source and git. |
diff -u ~/Downloads/5917_source_hashes.txt plb_hash.txt
--- /home/pbossart/Downloads/5917_source_hashes.txt 2022-06-14 16:04:19.667677436 -0500
+++ plb_hash.txt 2022-06-14 16:05:22.211575742 -0500
@@ -134,7 +134,7 @@
cc203436ad67d5fc42f6205a8815f55c7d061418 src/arch/xtensa/hal/mp_asm.S
bacbfc6ff0c71b0aba8bb31c16ed345feefdfacc src/arch/xtensa/hal/mpu.c
a2a544bd354d6cf4c30c8bb326ec1173694bc39c src/arch/xtensa/hal/mpu_asm.S
-b1b53ed4ab216f6a0c8e7c628d93de627ac370b1 src/arch/xtensa/hal/set_region_translate.c
+27ed6b80a50b1b89f7f8b7653355f25ad0cb9932 src/arch/xtensa/hal/set_region_translate.c
316ddb4e829827a7b1637415030a1c2c37121e07 src/arch/xtensa/hal/state.c
108986228584696b4c6a235fecd0760f8b4c2ca7 src/arch/xtensa/hal/state_asm.S
0716ddca17ff2586d31b94c3ab1d5bca14377355 src/arch/xtensa/hal/syscache_asm.S
@@ -158,7 +158,7 @@
44874cd946df8d92f28241df54854a73ecdec15c src/arch/xtensa/include/arch/spinlock.h
1172cb488b88d4966b5f8a4b57d9deb9c3bcfddc src/arch/xtensa/include/arch/string.h
c6b04a250575a0f9616d4c59385db05311b35163 src/arch/xtensa/include/xtensa/board.h
-4b17987ea95c462625f792507f624e241776fa0e src/arch/xtensa/include/xtensa/c6x-compat.h
+ca91bd7183971221923b6e882796e88d0acfb3cd src/arch/xtensa/include/xtensa/c6x-compat.h
9cb2c8fcc6b85f6d21d78ad0755285a8fe5d27f7 src/arch/xtensa/include/xtensa/cacheasm.h
211803aedbf39318f912fcc291efafad970f78ef src/arch/xtensa/include/xtensa/cacheattrasm.h
f5bb44faf2ab30bdb3de131f86cdb420b244cec6 src/arch/xtensa/include/xtensa/config/core.hAbsolutely no idea what this is. |
|
one possibility is that I don't have gcc installed for TGL. I never use GCC anyways even for older hardware. |
|
git --version |
|
Only 2 files are different, all others the same? Can you please run these: git hash-object src/arch/xtensa/hal/set_region_translate.c
b1b53ed4ab216f6a0c8e7c628d93de627ac370b1
md5sum src/arch/xtensa/hal/set_region_translate.c
36cae0b29a2c1b3a65f1e9dbb3bb829b src/arch/xtensa/hal/set_region_translate.c
git cat-file -p 27ed6b80a50b1b89f7f8b7653355f25ad0cb9932 | md5sum
git cat-file -p b1b53ed4ab216f6a0c8e7c628d93de627ac370b1 | md5sum
36cae0b29a2c1b3a65f1e9dbb3bb829b -
wget https://raw.githubusercontent.com/thesofproject/sof/b4886bebbe49454850d59f1a49a0460e590db71c/src/arch/xtensa/hal/set_region_translate.c
diff -u set_region_translate.c src/arch/xtensa/hal/set_region_translate.c |
|
|
looks like git hash-object provides a different value for the same file? |
|
I don't know how this would impact the signature though? The sha1 used for the signature should only work with the binary itself. |
|
Bingo! |
Agreed, this should not affect the rest of the build. It's still a serious bug though because: 1. it breaks the logger dictionary checksum; 2. it makes troubleshooting other issues much more complicated. |
|
after breaking audio since 1997, I just started a new career with crypto. Bitcoin, here I come :-) |
|
OK, these two files and only these two have Windows end of lines:
Some repos have |
|
There is an assertion failure in the rimage. sof_ri_info for none -d built sof.richeck rimage signing process with valgrind |
|
Thanks @aiChaoSONG , does this |
|
Below is the I ran the same command on @fredoh9's build and only his signature differs with mine. @plbossart can you please run this If there is some memory corruption with rimage+openssl3 then all bets are off, anything can happen. --- mine 2022-06-15 10:44:00.308980601 -0700
+++ chao 2022-06-15 10:44:16.514391243 -0700
@@ -1,47 +1,47 @@
-SOF Binary build_tgl_xcc/sof.ri size 0x81300
+SOF Binary build_tgl_xcc/sof.ri size 0x82300
Extended Manifest ver 1.0.0 length 768
CSE Manifest ver 0x102 checksum 0x0 partition name ADSP
ADSP.man (CSS Manifest) type 0x4 file offset 0x35c hdr_len 900 ver 0x21000 date 2022/06/15
Rsvd0 0x0
Modulus size (dwords) 96
6b 75 ed 58 20 08 85 95 ... 55 d1 7d c6 0d 79 12 a9 (Community 3k key)
Exponent size (dwords) 1
01 00 01 00
Signature (file offset 0x560, length 0x180)
- 8c 42 36 21 c1 5c 5a e6 ... 02 83 81 7f 4c af 01 5a
+ bc 82 30 c5 09 45 2d 3a ... 6e 20 78 e8 7e 30 1a 5e
Plat Fw Auth Extension type 0xf file offset 0x6e0 length 0x78
name ADSP vcn 0x0 bitmap 00 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00 svn 0x0
Other Extension type 0x16 file offset 0x758 length 0x68
cavs0015.met (ADSP Metadata File Extension) type 0x11 file offset 0x7c0 length 0x70
- ver 0x0 base offset 0xfe15179d limit offset 0x5b667f21
+ ver 0x0 base offset 0x30f7628d limit offset 0xf5f479cf
IMR type 0x3
Attributes
- 9c 06 84 92 54 50 c5 49 00 20 00 00 c0 2a 08 00
+ d6 6c 05 2d d1 76 5c d0 00 20 00 00 c0 3a 08 00
cavs0015
cavs0015 (ADSP Manifest) file offset 0x2300 name ADSPFW build ver 2.0.0.1 feature mask 0xffff image flags 0x0
HW buffers base address 0x0 length 0x0
Load offset 0x30000
BRNGUP 2b79e4f3-4675-f649-89df-3bc194a91aeb
entry point 0xb0038000 type 0x21 ( loadable LL )
cfg offset 0 count 0 affinity 0x3 instance max count 1 stack size 0x1
.text 0xb0038000 file offset 0x8000 flags 0x1001f ( contents alloc load readonly code type=0 pages=1 )
.rodata 0xb0039000 file offset 0x9000 flags 0x1012f ( contents alloc load readonly data type=1 pages=1 )
.bss 0x0 file offset 0x0 flags 0xf00 ( type=15 pages=0 )
BASEFW 0e398c32-5ade-ba4b-93b1-c50432280ee4
entry point 0xbe02c400 type 0x21 ( loadable LL )
cfg offset 0 count 0 affinity 0x3 instance max count 1 stack size 0x1
- .text 0xbe02c000 file offset 0xa000 flags 0x2e001f ( contents alloc load readonly code type=0 pages=46 )
- .rodata 0xbe05a000 file offset 0x38000 flags 0x49012f ( contents alloc load readonly data type=1 pages=73 )
- .bss 0xbe0a3000 file offset 0x0 flags 0x23d0202 ( alloc type=2 pages=573 )
+ .text 0xbe02c000 file offset 0xa000 flags 0x2f001f ( contents alloc load readonly code type=0 pages=47 )
+ .rodata 0xbe05b000 file offset 0x39000 flags 0x49012f ( contents alloc load readonly data type=1 pages=73 )
+ .bss 0xbe0a4000 file offset 0x0 flags 0x23c0202 ( alloc type=2 pages=572 )
Memory layout undefined
|
|
@plbossart please also try this --- a/src/arch/xtensa/CMakeLists.txt
+++ b/src/arch/xtensa/CMakeLists.txt
@@ -127,7 +127,7 @@ separate_arguments(EXTRA_CFLAGS_AS_LIST NATIVE_COMMAND ${EXTRA_CFLAGS})
# de-duplication "feature"
target_compile_options(sof_options INTERFACE
$<$<COMPILE_LANGUAGE:C>:
- -${optimization_flag} -g
+ -${optimization_flag} -g0
-Wall -Werror
-Wl,-EL
-Wmissing-prototypes
@@ -449,11 +449,11 @@ if(MEU_PATH OR DEFINED MEU_NO_SIGN) # Don't sign with rimage
# Passing -s ${MEU_OFFSET} disables rimage signing and produces
# one .uns file and one .met file instead of a .ri file.
add_custom_target(
run_rimage
- COMMAND ${PROJECT_BINARY_DIR}/rimage_ep/build/rimage
+ COMMAND valgrind ${PROJECT_BINARY_DIR}/rimage_ep/build/rimage
-o sof-${fw_name}.ri
-c "${PROJECT_SOURCE_DIR}/rimage/config/${fw_name}.toml"
-s ${MEU_OFFSET}
-k ${RIMAGE_PRIVATE_KEY}
-i ${RIMAGE_IMR_TYPE}
@@ -491,11 +491,11 @@
)
endif()
else() # sign with rimage
add_custom_target(
run_rimage
- COMMAND ${PROJECT_BINARY_DIR}/rimage_ep/build/rimage
+ COMMAND valgrind ${PROJECT_BINARY_DIR}/rimage_ep/build/rimage
-o sof-${fw_name}.ri
-c "${PROJECT_SOURCE_DIR}/rimage/config/${fw_name}.toml"
-k ${RIMAGE_PRIVATE_KEY}
-i ${RIMAGE_IMR_TYPE}
-f ${SOF_MAJOR}.${SOF_MINOR}.${SOF_MICRO}EDIT: not using valgrind is a waste of time.
Bonus feature: adding valgrind is the easiest way to print the rimage command line whereas a verbose build is crazy verbose. |
|
@aiChaoSONG your last parameter to rimage should be: it is now wrong and you are bailing out even before signing... with last parameter fixed I can sign with openssl3 and valgrind looks clean (I think @marc-hb also found that). so we really need to look at the signing differences in the image. |
@marc-hb I am afraid I have another signature, not the same as @aiChaoSONG sof_ri_info--- chao.log 2022-06-15 13:51:02.061698055 -0500
+++ plb.log 2022-06-15 13:51:41.166699970 -0500
@@ -4,32 +4,32 @@
CSE Manifest ver 0x102 checksum 0x0 partition name ADSP
ADSP.man (CSS Manifest) type 0x4 file offset 0x35c hdr_len 900 ver 0x21000 date 2022/06/15
Rsvd0 0x0
Modulus size (dwords) 96
6b 75 ed 58 20 08 85 95 ... 55 d1 7d c6 0d 79 12 a9 (Community 3k key)
Exponent size (dwords) 1
01 00 01 00
Signature (file offset 0x560, length 0x180)
- bc 82 30 c5 09 45 2d 3a ... 6e 20 78 e8 7e 30 1a 5e
+ bc 18 72 5e 87 53 70 06 ... 2c 6c 20 cd c9 43 ab 72
Plat Fw Auth Extension type 0xf file offset 0x6e0 length 0x78
name ADSP vcn 0x0 bitmap 00 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00 svn 0x0
Other Extension type 0x16 file offset 0x758 length 0x68
cavs0015.met (ADSP Metadata File Extension) type 0x11 file offset 0x7c0 length 0x70
- ver 0x0 base offset 0x30f7628d limit offset 0xf5f479cf
+ ver 0x0 base offset 0x97e32029 limit offset 0xc60fedb4
IMR type 0x3
Attributes
- d6 6c 05 2d d1 76 5c d0 00 20 00 00 c0 3a 08 00
+ 5a 9c b4 4a 6b e7 62 03 00 20 00 00 c0 3a 08 00
cavs0015
cavs0015 (ADSP Manifest) file offset 0x2300 name ADSPFW build ver 2.0.0.1 feature mask 0xffff image flags 0x0
HW buffers base address 0x0 length 0x0
Load offset 0x30000
BRNGUP 2b79e4f3-4675-f649-89df-3bc194a91aeb
entry point 0xb0038000 type 0x21 ( loadable LL )
cfg offset 0 count 0 affinity 0x3 instance max count 1 stack size 0x1
|
|
FWIW on my device I have this: openssl version Edit: |
|
Thanks, can you make sure you have the same reproducible.ri sha256sum as everyone else? (after disabling If they're the same then we are 100% sure this is a pure signing or manifest issue. |
|
As a workaround please try the Docker build (without XCC) docker pull thesofproject/sof # get coffee
./scripts/docker-run.sh openssl version
# OpenSSL 1.1.1f 31 Mar 2020
./scripts/docker-run.sh ./scripts/xtensa-build-all.sh tgl
|
|
yep, same reproducible -> pure signature issue. |
instant coffee with Google Fiber :-) works fine, the firmware boots on Up Extreme11. |
@marc-hb I think I signed the wrong binary (build_tgl_xcc/sof), that's why I have assertion failure. if I use the correct binary (build_tgl_xcc/src/arch/xtensa/sof-tgl), no failure at all. thanks Jaska to correct me. |
|
@aiChaoSONG & @plbossart can you try: #97 |
|
Manually tried #97, firmware can boot on UpExtreme, file a SOF test PR to test the rimage PR on other platforms except ACE: thesofproject/sof#5928 |
|
Confirmed fix, works for me as well. Thanks @juimonen |
|
@plbossart , @aiChaoSONG and anyone with openssl3, could you run For |
Local filters in ~/gitconfig, such as [core] autocrlf = input can impact the result of git hash-object. Make sure no filters are used so that the hash value remains unmodified across user setups. BugLink: https://github.com/thesofproject/sof/issues/5917 Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
|
SOF commit: 1083bc2a39fbfcf4056ece5983ff00532e370179 Without #97, rimage commit: 9d45332xcc_ri_info_with_ssl3.txt With #97xcc_ri_info_with_ssl3_and_rimage_fix.txt |
marc-hb
changed the title
|
The difference between
So maybe everything is OK? I wish I knew or remembered why there is so much --- chao gcc_ri_info_with_ssl1.txt 2022-06-17 17:51:58.976044719 +0000
+++ chao gcc_ri_info_with_ssl3_and_rimage_fix.txt 2022-06-17 17:51:58.976044719 +0000
@@ -19,10 +19,10 @@
Other Extension type 0x16 file offset 0x758 length 0x68
cavs0015.met (ADSP Metadata File Extension) type 0x11 file offset 0x7c0 length 0x70
- ver 0x0 base offset 0x261b8ed7 limit offset 0xf6ab7adf
+ ver 0x0 base offset 0xedd9b8ac limit offset 0x38c0e79e
IMR type 0x3
Attributes
- da e4 ae 75 c7 c7 10 be 00 20 00 00 c0 ca 06 00
+ fa db 8c d0 e8 bd 2d c3 00 20 00 00 c0 ca 06 00
cavs0015--- chao gcc_ri_info_with_ssl1.txt
+++ marc_gcc_ri_info_with_ssl1.txt
@@ -19,10 +19,10 @@
Other Extension type 0x16 file offset 0x758 length 0x68
cavs0015.met (ADSP Metadata File Extension) type 0x11 file offset 0x7c0 length 0x70
- ver 0x0 base offset 0x261b8ed7 limit offset 0xf6ab7adf
+ ver 0x0 base offset 0x8e159352 limit offset 0xcbaeed49
IMR type 0x3
Attributes
- da e4 ae 75 c7 c7 10 be 00 20 00 00 c0 ca 06 00
+ 35 d2 98 c9 6a 0a c3 18 00 20 00 00 c0 ca 06 00
cavs0015
|
|
@plbossart rimage fix now merged. Can you try and we can close if good ? |
|
#97 fix confirmed |
I'm reverting the |
and others
Describe the bug
The firmware generated locally on my NUC does not boot on UpExtreme 11. The daily-build firmware and the 2.1 version boot fine, so it's not a hardware/CSME issue, rather an undocumented or missing environment for the signature to work.
Standard Ubuntu 22.04 on my side. Same scripts work fine on Up Extreme (WHL)
To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)
scripts/xtensa-build-all.sh -d tgl (debug version)
scripts/xtensa-build-all.sh tgl
Reproduction Rate
100%
Expected behavior
no errors on boot
Impact
showstopper
Environment
Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
Kernel: 9eaf67f8c40cc (sof/topic/sof-dev) soundwire: intel: set dev_num_ida_min
SOF: e9442e04c (HEAD -> main, origin/main) module_adpater: change the buffer size of waves_codec
Name of the topology file
Name of the platform(s) on which the bug is observed.
Screenshots or console output
The text was updated successfully, but these errors were encountered: