-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to verify individual crates build (and ideally pass tests) for distro packaging #3732
Comments
|
I think that |
@alexcrichton we are trying to verify that crate works itself (for libs) and to build real thing (for bins). While it can be partially lying on first category, second one is definitely the case. |
@ignatenkobrain can you give me some more information? What does it mean "verify crate works itself"? Why/where are you validating libraries? How is this specific to fedora? etc... |
/usr/bin/mkdir -p .cargo
cat > .cargo/config << EOF
[build]
rustc = "/usr/bin/rustc"
rustflags = ["-Copt-level=3", "-g", "-Clink-arg=-Wl,-z,relro,-z,now", ]
[term]
verbose = true
[source]
[source.local-registry]
directory = "/usr/src/rust"
[source.crates-io]
registry = "https://crates.io"
replace-with = "local-registry"
EOF
/usr/bin/cargo build --release -j8
if /usr/bin/cargo-inspector --target-kinds Cargo.toml | grep -q -F "$(printf 'lib\nproc-macro')"; then
CRATE_NAME=$(/usr/bin/cargo-inspector --name Cargo.toml)
CRATE_VERSION=$(/usr/bin/cargo-inspector --version Cargo.toml)
REG_DIR=/home/brain/rpmbuild/BUILDROOT/aho-corasick-0.6.2-1.fc26.x86_64/usr/src/rust/$CRATE_NAME-$CRATE_VERSION
/usr/bin/mkdir -p $REG_DIR
/usr/bin/cargo package -l | xargs cp --parents -p -t $REG_DIR
/usr/bin/install -p Cargo.toml.orig $REG_DIR/Cargo.toml
echo '{"files":{},"package":""}' > $REG_DIR/.cargo-checksum.json
fi
if /usr/bin/cargo-inspector --target-kinds Cargo.toml | grep -q -F bin; then
/usr/bin/cargo install -j8 --path . --root /home/brain/rpmbuild/BUILDROOT/aho-corasick-0.6.2-1.fc26.x86_64/usr
/usr/bin/rm /home/brain/rpmbuild/BUILDROOT/aho-corasick-0.6.2-1.fc26.x86_64/usr/.crates.toml
fi
/usr/bin/cargo test --release -j8 That's what we do.
Why do we "verify crate works itself"? Because we don't want to package "cat in box", if crate can't do build either it's nightly or it's broken. That's the reason why we also run While running test suite sometimes broken (#3642), at least |
P.S. |
What is our workaround now:
Basically if %check is disabled in RPM, then we remove |
@ignatenkobrain ok thanks for the info! The tl;dr; I believe is that unfortunately Cargo just doesn't support this use case today. I don't currently know the best way to doing so as well. |
With a broader view, I think the basic concept of what we'd want is already there in the "ephemeral" flags in cargo. That is, when dealing with packaging, we want to manage a single standalone crate much the same way that Maybe something like an #3369 is related too, I think. |
Lending my experiences in case it's useful for drafting/prioritizing features. We import third-party packages as source to our internal repository for various security, legal, and operational reasons (patching, license compliance, etc.). Our import tool finds a specified package on crates.io, recursively determines what dependencies need to be imported at the same time, and generates configuration to glue it to our in-house build system. When source code is updated (or initially imported) we automatically build it and store the artifacts corresponding to that version/target/etc. This currently breaks on certain crates like libc which defines workspace packages that aren't included in the crate itself. It seems like our options are:
Rust tooling at my company is community maintained and some of it is old so I'm open to hearing alternative approaches to this problem. As interest increases, removing these hurdles will go a long way towards encouraging more adoption in real projects. P.S. This doesn't reflect the views of my employer. |
@cuviper @awalcutt I think the patch in #3369 may solve your problem, along with one other change. For Debian packaging, we construct a directory registry, which includes the sources of all dependencies (recursively), including the current crate. We then run That should help you solve the bootstrapping problem. Once you've bootstrapped, you may be able to opt into running |
Yes, we are making a system-wide directory registry in Fedora. As I understand Debian's scheme, you have the system-wide location too, which you then symlink into the builddir along with the current crate. But even if we follow the same scheme, how can you build/test a library-only crate within a local directory registry? We'd like to have some measure of confidence that the crate is working before we get all the way to a leaf crate to The fact that |
No context here, but I see that people are discussing how to make published Cargo.toml's buildable. FWIW here is how I modify toml files in cargobomb. |
@cuviper At the moment, we don't. I'd like to run not just I'd welcome an option that made a build from a local directory equivalent to one from the registry, but even with that, |
@brson It's cool that you've automated this frobbing for cargobomb, but I think we'll all be better served if cargo gains a direct option to ignore those things. @joshtriplett I do think it's still worth doing build-only testing, because said distro developers may still make mistakes, e.g. forgetting to declare a system dependency in the spec. And of course the crates.io author may be building on a totally different platform. |
I believe a number of issues here were fixed with #4030, but are there more remaining? |
All background written in upstream cargo GitHub issue[0]. In short, cargo build/install enforces us to have all dev-dependencies even they are not used for building/installed. [0] rust-lang/cargo#3732 Signed-off-by: Igor Gnatenko <ignatenkobrain@fedoraproject.org>
Optimistically close, and reopen if need be? |
Sure! |
In Fedora we're packaging each crate and there are definitely circular dependencies in dev-dependencies (memchr -> quickcheck(dev) -> regex -> memchr), so we first need to build memchr, then regex, then quickcheck and then we can rebuild memchr with enabled cargo-test.
We override crates.io registry, but cargo-build still tries to find quickcheck:
But build doesn't really need that dependency to be present...
The text was updated successfully, but these errors were encountered: