cfi tests failing on 32-bit arm linux

[rovka]

Bugzilla Link	44157
Version	unspecified
OS	Linux
Blocks	#51489
Attachments	Output of cfi tests failures
CC	@zatrazz,@zmodem,@slacka,@tstellar

Extended Description

We get the following failures:
cfi-devirt-lld-armhf :: cross-dso/simple-fail.cpp
cfi-devirt-lld-armhf :: cross-dso/simple-pass.cpp
cfi-devirt-lld-armhf :: cross-dso/stats.cpp
cfi-devirt-lld-thinlto-armhf :: cross-dso/simple-fail.cpp
cfi-devirt-lld-thinlto-armhf :: cross-dso/simple-pass.cpp
cfi-devirt-lld-thinlto-armhf :: cross-dso/stats.cpp
cfi-standalone-lld-armhf :: cross-dso/simple-fail.cpp
cfi-standalone-lld-armhf :: cross-dso/simple-pass.cpp
cfi-standalone-lld-armhf :: cross-dso/stats.cpp
cfi-standalone-lld-thinlto-armhf :: cross-dso/simple-fail.cpp
cfi-standalone-lld-thinlto-armhf :: cross-dso/simple-pass.cpp
cfi-standalone-lld-thinlto-armhf :: cross-dso/stats.cpp
More details in the attached log.

[rovka]

The issue reproduces when running test-release.sh on master. I've been looking at stats.cpp.

The issue that we're hitting is an invalid instruction while running __cfi_check. This looks like a Thumb function (it's disassembled that way by gdb, and it's also after a $t symbol in an objdump of the executable). However, we're hitting it in ARM mode ('p $cpsr & 0x20' returns 0). So we fetch 4 bytes instead of 2 for the first instruction (which is just a 'push {r7, lr}') and then things go boom as we try to fetch from the middle of the following 4-byte instruction.

I've been stepping through to see how we end up here, and it seems __cfi_check is called from __cfi_slowpath, which is also a Thumb function according to the objdump. We're running that one in ARM mode too, although it seems to have only 4-byte instructions so I'm guessing it's just a coincidence that it doesn't blow up.

__cfi_slowpath is called from _dl_runtime_resolve by doing a 'bx r12', where r12 contains the value returned by _dl_fixup: 0x2a01eff0. I think _dl_fixup should have returned an address with the 0 bit set to 1, since as I mentioned earlier this is a Thumb function according to objdump. I'm not sure how or why this is happening though.

[rovka]

Symbols for stats.cpp.tmp (objdump --syms --special-syms)

[zatrazz]

Hi Diana,

The glibc _dl_fixup handle thumb symbols similar than arm ones, it is up to static linker to setup both symbol symbol value and DT_SYMTAB base to the expected values.

On _dl_fixup, global symbols address on PLT calls will be set as:

67 const ElfW(Sym) *const symtab
68 = (const void *) D_PTR (l, l_info[DT_SYMTAB]);
[...]
73 const ElfW(Sym) *sym = &symtab[ELFW(R_SYM) (reloc->r_info)];
[...]
112 result = _dl_lookup_symbol_x (strtab + sym->st_name, l, &sym, l->l_scope,
113 version, ELF_RTYPE_CLASS_PLT, flags, NULL);
[...]
126 value = DL_FIXUP_MAKE_VALUE (result,
127 SYMBOL_ADDRESS (result, sym, false));
[...]
138 value = elf_machine_plt_value (l, reloc, value);
[...]
148 return elf_machine_fixup_plt (l, result, refsym, sym, reloc, rel_addr, value);

(assuming that the symbol is not an IFUNC and LD_BIND_NOW is not set).

The arm 'elf_machine_plt_value' is a nop operation (it just returns the 'value' input) and 'elf_machine_fixup_plt' just write down the 'value' on 'rel_addr':

275 static inline Elf32_Addr
276 elf_machine_fixup_plt (struct link_map *map, lookup_t t,
277 const ElfW(Sym) *refsym, const ElfW(Sym) *sym,
278 const Elf32_Rel *reloc,
279 Elf32_Addr *reloc_addr, Elf32_Addr value)
280 {
281 return reloc_addr = value;
282 }
283
284 / Return the final value of a plt relocation. */
285 static inline Elf32_Addr
286 elf_machine_plt_value (struct link_map *map, const Elf32_Rel *reloc,
287 Elf32_Addr value)
288 {
289 return value;
290 }

Also arm uses the generic DL_FIXUP_MAKE_VALUE definition, which does:

23 #define DL_FIXUP_MAKE_VALUE(map, addr) (addr)

So the _dl_fixup returned value for this case will be essentially:

const ElfW(Sym) *const symtab = (const void *) D_PTR (l, l_info[DT_SYMTAB]);
const ElfW(Sym) *sym = &symtab[ELFW(R_SYM) (reloc->r_info)];
result = _dl_lookup_symbol_x (strtab + sym->st_name, l, &sym, l->l_scope,
version, ELF_RTYPE_CLASS_PLT, flags, NULL);
return result->l_addr + sym->st_value;

So, for instance with a simple example:

--
#include

class Foo
{
public:
void foo (void)
{
std::cout << func << std::endl;
}
};

extern "C" {

void foo_c (void)
{
class Foo foo;
foo.foo ();
}

}

$ c++ -O0 -g -fno-stack-protector -mthumb -fpic -c -o libfoo-thumb.o libfoo.cc
$ c++ -mthumb -shared -Wl,-soname,libfoo-thumb.so -o libfoo-thumb.so libfoo-thumb.o
$ readelf -a
[...]
20: 00000779 22 FUNC GLOBAL DEFAULT 12 foo_c
[...]

This should be the value of 'sym->st_value' on _dl_fixup which will be added on the mmap memory region of the PT_LOAD that represent the text segment of the symbol obtained from _dl_lookup_symbol_x.

What I can't answer is why 'readelf -s' and 'objdump --sym' is showing different results for the same shared library:

--
$ readelf -s libfoo-thumb.so | grep foo_c
20: 00000779 22 FUNC GLOBAL DEFAULT 12 foo_c
72: 00000779 22 FUNC GLOBAL DEFAULT 12 foo_c
$ objdump --syms --special-syms libfoo-thumb.so | grep foo_c
00000778 g F .text 00000016 foo_c

Could you check if readelf shows the same st_value output for __cfi_slowpath symbol?

[rovka]

Oh, interesting, it seems readelf and objdump show different things. I think objdump always shows the "real" address, without the mode bit set, but readelf shows the mode bit as well. With readelf you can see that __cfi_check has the thumb bit set, but cfi_slowpath doesn't.

readelf -a stats.cpp.tmp | grep cfi_slowpath

0x1eff0 <__cfi_slowpath>: 0x8001848f
69: 0001eff 308 FUNC GLOBAL DEFAULT 15 __cfi_slowpath
2371: 0001eff 308 FUNC GLOBAL DEFAULT 15 __cfi_slowpath

objdump --syms stats.cpp.tmp | grep cfi_slowpath
0001eff g F .text 00000134 __cfi_slowpath

---

readelf -a stats.cpp.tmp | grep "cfi_check"
0x24000 <__cfi_check>: 0x808408b0
79: 00024001 204 FUNC GLOBAL DEFAULT 15 __cfi_check
2388: 00024001 204 FUNC GLOBAL DEFAULT 15 __cfi_check

objdump --syms stats.cpp.tmp | grep cfi_check
00024000 g F .text 000000cc __cfi_check

[rovka]

Also, here are the commands for building stats.cpp.tmp:

clang -fuse-ld=lld -flto -fsanitize=cfi -fwhole-program-vtables --driver-mode=g++ -fsanitize-cfi-cross-dso -fvisibility=default -DSHARED_LIB -fPIC -g -fsanitize-stats -shared -o stats.cpp.tmp.so llvm-project/compiler-rt/test/cfi/cross-dso/stats.cpp

clang -fuse-ld=lld -flto -fsanitize=cfi -fwhole-program-vtables --driver-mode=g++ -fsanitize-cfi-cross-dso -fvisibility=default -g -fsanitize-stats -o stats.cpp.tmp llvm-project/compiler-rt/test/cfi/cross-dso/stats.cpp stats.cpp.tmp.so

So there's no explicit -mthumb there. I'm not sure whether the cfi functions are expected to be thumb or arm in this case.

[zmodem]

Is this a regression from 9 -> 10, and if so would it be possible to bisect it?

Otherwise we probably shouldn't block on this.

[rovka]

IIRC this wasn't in 9.0.0, but it showed up in 9.0.1, so that's probably the branch that we should bisect. I'd need to check again with 9.0.0 though since I don't remember whether we had any infrastructure changes between 9.0.0 and 9.0.1. Unfortunately I haven't had much time to look into it. In any case, we can unblock.

[zmodem]

IIRC this wasn't in 9.0.0, but it showed up in 9.0.1, so that's probably the
branch that we should bisect. I'd need to check again with 9.0.0 though
since I don't remember whether we had any infrastructure changes between
9.0.0 and 9.0.1. Unfortunately I haven't had much time to look into it. In
any case, we can unblock.

Okay, unblocking.

[rovka]

Seeing a new cfi failure in 11.0.0-rc2 : cfi-devirt-lld-armhf :: bad-cast.cpp.

[rovka]

bad-cast.cpp is not failing anymore, but as of 13.0.0-rc3 we're having a failure in multiple-inheritance.cpp

[tstellar]

mentioned in issue #51489

[tstellar]

The deadline for requesting fixes for the release has passed. This bug is being removed from the LLVM 13.0.1 release milestone. If you have a fix or think this bug is important enough to block the release, please explain why in a comment and add the bug back to the LLVM 13.0.1 release milestone.