Tools for dumping information about .rlibs

[novocaine]

It would be great if there was a set of tools for dumping information about a given .rlib. This could be used to debug linkage and export issues.

Something like nm and ldd.

I note that nm does appear to partly work on a .rlib, but the output is a bit raw.

Where is the best place to start for understanding the .rlib format as it stands? Is it a good idea to build this on rustc::metadata?

See also http://stackoverflow.com/questions/24603672/rust-library-for-inspecting-rlib-binaries

[lifthrasiir]

cc me

For inspecting metadata this tool might be a good starting point. I have a version of that for post-reform metadata, I'll update the script soon.

[novocaine]

👍

[alexcrichton]

All rlibs are actually just static archives (e.g. *.a files) so tools like nm and ar will work on them naturally assuming they ignore the non-object files located within (e.g. metadata and bytecode). As @lifthrasiir mentioned, however, I'd love to have a tool to inspect the metadata in more detail to, for example, dump a module structure, type signatures, etc. The metadata format is a huge black box and is indeed all centralized around rustc::metadata. Note that a tool which does lots and lots with metadata will likely want to be tightly integrated to rustdoc as well as rustdoc already has tons of code to inspect metadata and reconstruct ASTs and/or module hierarchies.

[novocaine]

Would you expect ldd to work?

Otool on os x does not

[alexcrichton]

I think that ldd and otool only work on actual elf files like dynamic libraries and executables, so in that sense no I wouldn't expect them to work.

[novocaine]

(sorry about the 20 questions, but I can't find any resources on this stuff)

How do .rlibs link against 'normal' dynamic libraries provided to rustc via -l ? Is there an ELF image inside the ar somewhere that's dynamically linked against them? Or does rust embed dynamic loader code e.g dlopen?

[alexcrichton]

but I can't find any resources on this stuff

That... may be because none exist :(. I'm more than willing to answer questions though!

How do .rlibs link against 'normal' dynamic libraries provided to rustc via -l ?

No actual linkage occurs, the compiler emits information into the rlib metadata saying what dynamic native libraries should be linked in later, but they're not actually linked until the rlib is linked into a dylib or executable. If the rlib is later used to assemble a staticlib the compiler just prints out what native libraries are needed to link everything.

Is there an ELF image inside the ar somewhere that's dynamically linked against them?

Nah the only ELF file inside rlibs is the object file that the compiler generates (and possibly the object files of native static libraries linked against).

Or does rust embed dynamic loader code e.g dlopen?

No the compiler doesn't embed any code like this in this respect.

[novocaine]

Thanks, Alex :)

[m4b]

What is the status of this? I wouldn't mind working on something like this, I think I could do it fairly easy, as long as someone pointed me at the format of metadata.

Also, while I'm here, what precisely is bytecode.deflate in .rlibs; is it compressed llvm bitcode?

[m4b]

@alexcrichton sorry I always forget to ping and not sure if notified since this is very old thread

[alexcrichton]

AFAIK no changes. You'd basically need to read the compiler source to learn what these are, rlibs are full of internal implementation details.

[m4b]

I see. Is this something that's still actively desired, or not really? And what did you mean above about tightly integrated with rustdoc? The tool would use it as a lib to pretty print the metadata, or something else?

[alexcrichton]

With not really much activity for >2 years I wouldn't necessarily classify this as "actively desired". The effort required to stabilize something like this seems quite high, but adding debugging utilities on nightly perhaps may not be so hard.

[m4b]

Ok.

While you're here, do you know offhand what the contents of bytecode.deflate are? I've asked on the channel but no one seems to know, and I can't seem to find it documented anywhere.

[alexcrichton]

Yes it's just compressed LLVM bytecode. The actual format is a bit wonky with some headers and maybe a footer, you'd have to look around in trans for the exact format.

[m4b]

Cool, thanks so much for your time and answers :)

Let me know if there's something similar to this w.r.t. binary formats, etc., that might need some help/tooling, would be interested in helping.

[ruuda]

I am trying to obtain the stdlib bitcode, so I can use llvm-link to produce a self-contained bitcode program. Being able to extract it from the rlibs would be nice, because those are bundled with Rust, and I wouldn’t have to compile everything from source. However, I have been unable to decompress the bytecode.deflate file, which makes sense if it contains Rust-specific headers. (It definitely starts with RUST_OBJECT.) Is there a simple way to strip these headers so I can inflate the data?

[steveklabnik]

Triage; not aware of anyone having implemented this, which, given the fact that all of this is unstable, isn't that surprising. It would be nice to have, but it's unclear that it's a good idea to keep this issue open.

[Mart-Bogdan]

This os really needed feature, as currently IDEs (Idea and rls/rust-analyzer) have to expand all macros and analyze source code of dependecies.

It would be nice if they would be able to extract info about types and signarures from rlib in sone json format.

AFAIK Currently there are no way to provide autocompletion for proc-macro generated code. Only for regular macros, as IDEs are expanding them by themselves, but it looks like overkill for already compiled libraries.

[bjorn3]

Current rustc has -Zls which dumps a tiny bit of information:

$ rustc -Zls $(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-*.rlib
Crate info:
name std-977ece543e1e3d2f
hash 493ec6fd8120daa9 stable_crate_id StableCrateId(15100501329461213700)
proc_macro false
=External Dependencies=
1 core-0e3656b1fda5fd7b hash bf602c611c93048d host_hash None kind Explicit
2 compiler_builtins-16d69221f10b0282 hash 98516d7f9ee4dc1f host_hash None kind Explicit
3 rustc_std_workspace_core-2a6a2797f7a73818 hash 7669287c749f7e23 host_hash None kind Explicit
4 alloc-89782a6344bc3ddf hash 1998205beffe24d7 host_hash None kind Explicit
5 libc-e534461cfee9dab8 hash a1e6883f7e6924a3 host_hash None kind Explicit
6 unwind-cb0068802dd8f031 hash 9e9ab65c1f206b57 host_hash None kind Explicit
7 cfg_if-aa03de290f9594ce hash b325cc00df81012f host_hash None kind Explicit
8 miniz_oxide-1c5c08d77aa4ee1f hash ab361b9594d8cdef host_hash None kind Explicit
9 rustc_std_workspace_alloc-3ad551729ddf5bdf hash d33ff096cbf61200 host_hash None kind Explicit
10 adler-43c97e136c6f66b3 hash 43903e2b4d3800a9 host_hash None kind Explicit
11 hashbrown-626bd4749ba5679b hash 2af9a104d611783a host_hash None kind Explicit
12 std_detect-9961a029e2fb69de hash 48c6d697101a743a host_hash None kind Explicit
13 rustc_demangle-5543e955d2b2e107 hash 3db3e4e8af49176c host_hash None kind Explicit
14 addr2line-2cd7f06709609788 hash 119ed746aeeccd16 host_hash None kind Explicit
15 gimli-05bd833c6cc845b5 hash 835369855f9bdf9b host_hash None kind Explicit
16 object-93491bde8b3642ba hash 59cb584eda23ec10 host_hash None kind Explicit
17 memchr-d338f5690e3fda2f hash 84134cd8ef0ca3e9 host_hash None kind Explicit
18 panic_unwind-1141b5caaf7b2195 hash ec463f279262c141 host_hash None kind Implicit

I also wrote https://github.com/bjorn3/rust_read_rlib at some point, but it would need to be updated to latest rustc.

[SamB]

It seriously vexes me that things like this can't be allowed on stable because somebody would inevitably come to rely on the exact format of the output. :-(

Specifically, the part where we can't inspect the rust-specific metadata.

The fact that such functionality would be nightly-only probably isn't helping to motivate anyone to implement it.