not download and build artifacts again, example: arch linux, cosmic desktop

[soloturn]

Problem

on arch linux cosmic desktop is now split into 10 or more components. all of the components have similar dependencies. building takes forever, as the dependency git repositories are downloaded again, built again, crates are downloaded again, vendored again, built again. this ticket is to get ans answer to properly configure the environment so all of this happens only one time.

the links of the build scripts:

• https://aur.archlinux.org/packages/cosmic-session-git
• https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=cosmic-session-git
• https://github.com/pop-os/cosmic-session/blob/master/Justfile

  export CARGO_HOME="$srcdir/cargo-home"
  export RUSTUP_TOOLCHAIN=stable
  cargo fetch --target "$CARCH-unknown-linux-gnu"
  export CARGO_HOME="$srcdir/cargo-home"
  export RUSTUP_TOOLCHAIN=stable
  cargo fetch --target "$CARCH-unknown-linux-gnu"
  RUSTFLAGS="-C link-arg=-fuse-ld=mold"
  nice just all

$ GITFLAGS="--filter=tree:0"  CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse CARGO_BUILD_JOBS=2 paru -S cosmic-session-git
...
==> Starting prepare()...
    Updating crates.io index
    Updating git repository `https://github.com/pop-os/cosmic-protocols`
    Updating git repository `https://github.com/pop-os/libcosmic`
    Updating git submodule `https://github.com/pop-os/iced.git`
    Updating git repository `https://github.com/pop-os/dbus-settings-bindings`
    Updating git repository `https://github.com/pop-os/cosmic-settings-subscriptions`
    Updating git repository `https://github.com/pop-os/cosmic-time`
    Updating git repository `https://github.com/pop-os/cosmic-comp.git`
    Updating git repository `https://github.com/pop-os/cosmic-notifications`
    Updating git repository `https://github.com/pop-os/cosmic-panel`
    Updating git repository `https://github.com/DioxusLabs/taffy`
    Updating git repository `https://github.com/jackpot51/rust-atomicwrites`
    Updating git repository `https://github.com/smithay/client-toolkit`
    Updating git repository `https://github.com/pop-os/xdg-shell-wrapper`
    Updating git repository `https://github.com/pop-os/window_clipboard.git`
    Updating git repository `https://github.com/pop-os/cosmic-text.git`
    Updating git repository `https://github.com/pop-os/softbuffer`
    Updating git repository `https://github.com/pop-os/smithay-clipboard`
    Updating git repository `https://github.com/wash2/accesskit.git`
    Updating git repository `https://github.com/pop-os/glyphon.git`
    Updating git repository `https://github.com/gfx-rs/wgpu`
...
==> Starting build()...
rm -rf vendor
tar pxf vendor.tar
cargo build --release --frozen --offline 
...
 Downloaded image v0.24.9
  Downloaded 615 crates (114.6 MB) in 33.25s (largest was `windows` at 11.8 MB)
   Vendoring accesskit v0.12.2 (https://github.com/wash2/accesskit.git?branch=winit-0.29#26f72916) (/home/st/.cache/paru/clone/cosmic-applets-git/src/cargo-home/git/checkouts/accesskit-7ed9810dd0d642c4/26f7291/common) to vendor/accesskit
...
   Compiling proc-macro2 v1.0.85
...

for every component this is done in the components directory, and thus downloaded and ocmpiled 10 times minimum.

[soloturn]

@saethlin you are on arch, and know rustup / cargo best, you d have an idea what build variables we should put into the cosmic readme to ease the compile a little?

[epage]

This seems more Arch specific than Cargo.

For example

as the dependency git repositories are downloaded again

is not because of anything Cargo does but because the PKGBUILD's are setup to counter Cargo's caching features, e.g.

export CARGO_HOME="$srcdir/cargo-home"

We're looking at adding more caching (#5931) but that would also be ignored by Arch.

At this is more of an Arch thing and not Cargo, I am inclined to close this. Open to input on what is being asked of Cargo here, feature or bug fix. For user support, we generally recommend https://users.rust-lang.org/.

[arlosi]

Agreed with @epage. Since these are individual Arch packages, and Arch's PKGBUILD is isolating builds from each other, Cargo can't re-use the build artifacts between packages.

If there are specific ideas about what Cargo could do to help for this type of situation, please open a new issue.

[soloturn]

PKGBUILD can use environment variables, and it just calls cargo. so what cargo does, arch does, @arlosi @epage .

[epage]

@soloturn the PKGBUILD is controlling cargo, telling it to not cache things.

[soloturn]

i am controlling PKGBUILD, what parameter you suggest to add so it caches things @epage ?

[epage]

Remove

export CARGO_HOME="$srcdir/cargo-home"

that will help with download, extraction, and storage of dependencies.

This won't help with caching intermediate build artifacts. #5931 would be the main way to do that.

[soloturn]

ok, will try. @epage i read the per user cache dir proposal and got a chicken skin about the complexities mentioned there :) if i could specify a directory where all build files end up this would be perfect, and just ignoring all edge cases and complexities mentioned in #5931. kind of a "build-dir" ? if the build gives problems i can drop this directory any time and we have status quo, build from scratch. no necessity for cargo to have some complicated logic or am i seeing this too simplistic?

[soloturn]

@epage the arch linux packages have default CARGO_HOME now, and download is reused, which is a pleasant speedup already. thanks for the hint, and thanks @yochananmarqos for putting that in everywhere.

but - the build still is VERY slow on older hardware, as download is not done any more, but still for every COSMIC compnent it builds 600 or so crates, times 20. mostly the same. what paramteer to specify to not recompile?

[arlosi]

The work to track re-using compiled artifacts between multiple compilations is covered by #5931.

[epage]

@soloturn as we've said, #5931 is the real solution. A "hack" some people do is to share a CARGO_TARGET_DIR between projects but this has problems with where they can overwrite each other when it comes to final artifacts (at least). However, I assume your "projects" are all coming from the same workspace, so during development they are built with the same CARGO_TARGET_DIR. I would not extend it beyond projects that come from the same workspace. Maybe you could hash the repository/source field in the PKGBUILD which has a high correlation with "same workspace" to help avoid unintentional issues.

[soloturn]

@epage @arlosi am using sccache now, and when building the SAME component again, the cache works. but when a crate is used for 2 components, it seems to be cache misses only. why is this so?

[epage]

I don't use sccache and can't speak to its behavior and how it interacts with cargo.

[weihanglo]

@soloturn Please don't post the same comment twice in different issues. I'll hide the other one. Also, while epage and arlosi are so nice to respond to your questions in a timely manner, but they also need time for rest (it's Friday).

At this is more of an Arch thing and not Cargo, I am inclined to close this. Open to input on what is being asked of Cargo here, feature or bug fix. For user support, we generally recommend users.rust-lang.org.

Please use the forum or https://rust-lang.zulipchat.com if you want to reach out to a wider audience, or discuss specific issues in cosmic.

[soloturn]

@weihanglo thank you, the cargo poeople indeed are very nice :)

let me open a bug for sccache in cargo: #14313 , as RUSTC_WRAPPER=sccache is a CARGO flag