Include jars as bytes

[vitorenesduarte]

Closes #21

[vitorenesduarte]

@romac do you have any ideas on this? I'm not sure if this is the best approach, or if we should have a build.rs.
Maybe the only downside here is the increased memory usage.

[romac]

I am not too worried about the memory usage, as all the JARs combined weigh <100MB and as far as I can tell, the OS will only page in the section of the executable where the data is stored on demand (provided you include it as a static instead of a const). This means that we will only consume those 100MB of memory when the files haven't been written to disk yet.

use std::{
    fs::File,
    io::{self, Write},
};

static DATA: &[u8] = include_bytes!("../data.jar"); // ~62MB of data

fn main() -> io::Result<()> {
    if std::env::args().len() > 1 {
        let mut file = File::create("/tmp/data.jar")?;
        file.write_all(DATA)?;
        println!("data");
    } else {
        println!("no data");
    }

    Ok(())
}

❯ time cargo build --release
   Compiling load-static v0.1.0 (/Users/coromac/Code/load-static)
    Finished release [optimized] target(s) in 5.43s
cargo build --release  4.43s user 0.80s system 94% cpu 5.549 total

❯ /usr/bin/time -l target/release/load-static
no data
        0.00 real         0.00 user         0.00 sys
              819200  maximum resident set size              <------------ few KB
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
                 214  page reclaims
                   0  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                   1  involuntary context switches
             5337410  instructions retired
             4917034  cycles elapsed
              315392  peak memory footprint

❯ /usr/bin/time -l target/release/load-static foobar
data
        0.05 real         0.00 user         0.04 sys
            65683456  maximum resident set size            <------------ ~62MB
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               16050  page reclaims
                   0  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   1  voluntary context switches
                  11  involuntary context switches
            96121624  instructions retired
           133812503  cycles elapsed
              335872  peak memory footprint

Compile times aren't great though (4.5s on my machine for 62MB of data).

---

Another solution would be to append the JAR files + a footer at the end of the executable, and then load the data from the executable itself by mapping the executable (std::env::exe_path), reading the footer at the end containing the offsets to the various files, and then dumping the various files to disk. As the total size of the data isn't too big, that seems a bit overkill to me at the moment. See https://fasterthanli.me/series/making-our-own-executable-packer/part-16 for more details.

[vitorenesduarte]

Awesome, thanks for this great analysis @romac!

I'm observing exactly what you described: memory consumption only increases if the jars haven't been written to disk yet. I didn't have to change the bytes from const to static, but maybe the only way to ensure that those bytes won't be loaded unless needed is to use static.

[romac]

Nevermind what I said on const vs static, they're both equivalent in this case, but static might compile a little bit faster (cf. rust-lang/rust#65818 (comment)).

[vitorenesduarte]

Got it, thanks again! 👍

[vitorenesduarte]

This should be good to go. The only downside seems to be compile times but overall they have decreased since we removed a few dependencies:

• for a fresh release build:
  • before: 36s
  • after: 26s
• for incremental builds, I see no substantial difference

[andrey-kuprianov]

Sounds good to me! Also happy we have removed a few dependencies.

May be in the future we should consider packing the JARs not as part of the executable, but to ship them alongside with it; the details are to be figured out.

The reason we might want to do this is because probably we'll have to support multiple versions of JARs soon, for backward compatibility with previously produced tests.

No objections against merging from my side:)

[romac]

@vitorenesduarte Happy to hear the incremental debug builds are fine but if those get bad or you often have to do a clean debug build, you can try https://docs.rs/lazy-static-include/3.0.5/lazy_static_include/ to only include the bytes in the executable when doing release builds.