-
Notifications
You must be signed in to change notification settings - Fork 12
Rust should integrate easily into large build systems #12
Comments
I started a repo not long ago to collect anecdotes about people integrating Rust into existing projects. I only have a few examples in there (and I keep seeing more crop up all the time). It'd be great to at least do a survey of all the examples we can find of people attacking this problem to see what the major issues were. From the Firefox perspective, we have a pretty custom build frontend that generates Makefiles in the backend, so we would have had to write the custom integration bits regardless. We did find that life got a lot better once we started invoking cargo instead of rustc directly. I'm sure there are projects that would get value out of having individual Rust source files in their codebase, but it feels like a lot of the value in the Rust ecosystem comes from using cargo and leveraging crates.io (I doubt this is contentious). |
I've worked on build systems to compile Rust components for Erlang applications, so I'm in a good position talk about a few issues. An overview of the building that takes place can be found in this README. Erlang can make use of bins, dylibs, and cdylibs.
|
This is great. Integrating Rust/Cargo with Bazel would be the first major hurdle to us using more Rust in our codebase at work, which is a pretty large mass of C++ code with some Python sprinkled throughout. Since both Bazel and Cargo are fairly opinionated about how builds and package management should work, and since I am an expert in neither, it's not immediately clear to me which build system should be doing what or if we should just try to integrate |
I thought Bazel already had basic support for rustc? Maybe my info is
outdated though.
…On Mon, Feb 6, 2017 at 5:51 PM, Bob Somers ***@***.***> wrote:
This is great. Integrating Rust/Cargo with Bazel <https://bazel.build/>
would be the first major hurdle to us using more Rust in our codebase at
work, which is a pretty large mass of C++ code with some Python sprinkled
throughout.
Since both Bazel and Cargo are fairly opinionated about how builds and
package management should work, and since I am an expert in neither, it's
not immediately clear to me which build system should be doing what or if
we should just try to integrate rustc into Bazel without Cargo at all.
Using strictly Cargo is (unfortunately) probably out of the question since
most of the C++ and Python packages are dependency tracked with Bazel BUILD
files, and any serious integration with the C++ code would require our Rust
libraries and binaries to be dependency tracked by Bazel as well.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#12 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABsipZo1JTHfFy5idL4sB97BOsZPUT1ks5rZ6P3gaJpZM4Ly9Nx>
.
|
Bazel has support for local sources, which is to say it does not support crates.io. There's an issue open for it, but the project is relatively inactive bazelbuild/rules_rust#2. One would need to keep a local repository clone of a crate and all its dependencies to make bazel happy without modifications. Bazel's rust targets also can't be depended on by c/c++ at the moment. The kythe project has wrappers for bazel to call cargo, but I it's not the most robust approach to integration. For reference, kythe:tools/build_rules/rust. disclaimer: I don't know too much about bazel, happened to learn the above looking into it last week. |
Bazel's been mentioned several times this has come up. I believe Dropbox needs to integrate its Rust into a Bazel build process, and I think I might have been told Facebook uses it as well (but it might also have been that Facebook has an internal tool is similar to Bazel). It seems like a promising tool to look into for this issue. |
Bazel Rust Rules author here. Apologies for the inactivity on the rules_rust project; I have been busy with other projects on my plate. I am planning to implement the workspace rules for pulling crates from Cargo (bazelbuild/rules_rust#2) by the end of Q1 with a stretch goal of also implementing tooling for automatically generating Bazel BUILD files from Cargo.toml (bazelbuild/rules_rust#3), though the latter will likely extend into Q2. The Kythe project has rules that shell out to Cargo directly as quick stop-gap measure, but those rules are meant to be used internally in Kythe for now (since they are not hermetic for instance) and the plan is to replace them with the rules in rules_rust once features such as pulling from Cargo are supported. Of course, if anyone is interested in helping with improving the Bazel rules for Rust, contributions are certainly welcome. :) +cc @damienmg |
Kythe contributor here. We'd definitely like to see better integration with Cargo from the upstream Bazel Rust rules. Our extant integration was very much a hack to allow our intern to make progress on the Rust indexer itself, rather than getting bogged down with Bazel integration. |
Facebook uses Buck, and there is some early Rust support: https://buckbuild.com/rule/rust_binary.html Facebook vendors their dependencies in-tree. |
I've spent quite a bit of time on that over the last few months, and it's getting pretty solid now. It's well integrated with the overall build/test system and (most recently) can also interop with cxx rules. |
I'm the main author of Meson build system that is being used by GStreamer and a bunch of other projects. We also have Rust support, which is a bit rudimentary but can be used to build stuff like a Python extension module that uses C, C++, Rust and Fortran in a single target. We aim to improve Rust support. This is especially important for mixed-language projects, since Cargo is nice for plain Rust projects but I'm fairly certain that Cargo developers do not want to add first class multiplatform C/C++ build support to Cargo. |
A key part of doing this well will involve handling updates to the dependency graph in a deliberate and piecemeal way, particularly in scenarios where upstream "master" has moved to a new version (e.g., as in rust-lang/cargo#2649). We (@nox, @SimonSapin, and I) have had many conversations with @alexcrichton and @wycats on this front, and I believe the leading contender from their point of view is for some extensions to fix up |
Nix (which has a similar pure view of build systems as Bazel) uses a trick that involves cloning a well known version of the crates index git repository, see https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/rust |
Sadly the nixpkgs crates index is not hermetically sealed, it gave us many problems. So we implemented a crate index nixifier, which reads crates.io-index and spits out a nixfied crates index. This allows one to use nix to completely manage transitive crate dependencies of a project without needing cargo. The repo: https://github.com/fractalide/nixcrates |
I've taken a look at teaching bazel how to digest Cargo's toml files to pull down third party crates.io dependencies automatically, and there don't seem to be very many sticking points. A couple of rough patches I've seen though:
|
Just commenting here because I believe there is a lot of opportunity to appeal to the JVM eco-system, and incrementally replacing java/scala/etc code with rust if we are able to make it trivial to incorporate rust into ant/maven/etc builds, as well as go the other way and be able to add a jar to a Cargo.toml or build.rs, and have automatic rust bindings generated, maybe using a combination of https://github.com/kud1ing/rucaja/ and https://github.com/kenpratt/jvm-assembler as starting points. This could all happen outside core, obviously, but rust adoption by the very large, and as-of-yet untapped-by-rust, ecosystem of jvm code and developers would be quite beneficial. |
I've proposed haskell/cabal#3882 for Cabal. The same thing can work for Cargo, and would solve this problem for everyone. |
@raphlinus and I were discussing some related issues on IRC not long ago. One of the ideas that he floated was a way to make cargo simply output the commands that it would run to do the build, so that we could leave parsing Cargo.toml to cargo, but allow other build systems like bazel to run the build like they expect. |
Thanks for the comments everyone! I and a few others have thought a lot about this in the past as well, and I wanted to jot down some notes and conclusions that we've reached historically. First and foremost we've historically concluded that build system integration is not finished until you've got access to crates.io crates. The standard library is purposefully conservative and small in size with the explicit intent of having rich functionality in the ecosystem on crates.io. If an integration doesn't allow easy access to crates.io, then there's more work yet to be done! Today, of course, Cargo is the primary gateway into the crates.io ecosystem. Cargo is also the primary build tool for Rust, but there's typically perennial questions about how to integrate Cargo into existing build tools. Many issues have been solved over time in this vein, such as vendoring dependencies, workspaces, etc. Cargo also has the benefit of being friendly and familiar to existing Rust programmers with a shared workflow across the ecosystem. Something else that we've concluded, however, is that preserving Cargo workflows should not necessarily be a hard constraint for build system integration. Existing projects already have a workflow associated with them, and Rust code should integrate as it is fit instead of imposing restrictions on how it works. Of course though preserving a Cargo-based workflow for the Rust-specific portions is a nice-to-have! And finally one last thing we've talked about is compilation units. For example C/C++ have files as compilation units and that's typically what build systems for C/C++ are normally architected around. Rust, however, doesn't have this granularity of compilation unit. Fundamentally the compiler supports crates as the compilation unit. Moving up the stack to Cargo it ends up generally being the case that the entire crate graph is Cargo's compilation unit (one command outputs the entire crate graph). The question is then how does this integrate into an existing build system? Is the crate graph sufficient? Does the granularity need to be finer, such as crates? (I'm not sure!) One part to consider about compilation units is that they typically heavily affect caching in build systems. For example distributed caches may cache entire compilation units but nothing more granular. This means that a DAG-as-a-unit would be probably too coarse for caching. On the other hand, though, crate-as-a-unit is not clear how to integrate between build systems and Cargo today. So with all that I think we're faced with a few problems that may be thorny to solve:
Unfortunately I don't have a whole lot of solutions just yet, I'm personally still at least trying to grapple with the problem space. @luser does the above sound accurate though for Gecko's Rust integrate at least on a high level? @jsgf could you detail some of the work you've done at a high level for Buck for Rust support? I've found that each build system tends to have its own unique set of constraints for integration, but the more we know the easier we can accommodate everyone! |
Oh one point I should also mention is that I personally think that it's at least relatively important to try to lean on Cargo as much as possible with build system integration. Cargo is the bread and butter of building Rust, and avoiding Cargo leaves build systems as massive number of features to reimplement. I'd much rather pursue avenues to add features and/or make Cargo more flexible to interoperate with existing build systems. For example I could imagine Cargo generating build files or working in a much more granular fashion assuming another process manages inputs/outputs. |
The things that make Cargo awkward to use in our environment are:
What I think you're saying is that going directly to Perhaps there's some scope for a mid-level tool that provides Right now I'm handling all this by using |
@alexcrichton DAG-as-a-unit will always be too course grained. That is basically what we have now, and it is not good enough. For crate-as-a-unit though, I'd think So yeah, what needs to be done at a minimum is making two new mode of operation for Cargo:
A cool follow-up to this would be writing a dependency-management library that both allows serializing plans such that external tools can drive the build, or executing the plan in the current process. This would avoid any code duplication in Cargo and, I'd guess, be useful for rustbuild, too. See haskell/cabal#4174 for analogous effort with Cabal and Shake (though, unfortunately, Shake does not allow exporting static dependency graphs like this). |
I also echo @jsgf's point about For example, the Bazel rust rules supports interop with C/C++ code, meaning that you can have the following: cc_library(
name = "foo",
srcs = ["foo.cc"],
hdrs = ["foo.h"],
)
rust_library(
name = "bar",
srcs = ["src/lib.rs"],
deps = [":foo"],
) Also, of note, while one mode of using Bazel is to vendor all dependencies (which is the practice followed by Google internally), Bazel also supports fetching external dependencies (which are all done prior to build time) and provides an simple API for writing repository rules. As mentioned above, a rule that fetches crates from Crates.io is in the pipeline for users who do not prefer to vendor all dependencies. |
Thanks for typing that up @jsgf!
To clarify, I'm under the impression that this is a solve problem today. With multiple vendoring options available that was at least the intention! Did you find, though, that the vendoring support wasn't suitable for Buck's use case?
To clarify, this is from a build system perspective, not a language perspective, right? If possible I'd like to focus this thread at least on just the build system aspect and we can perhaps continue the language discussion over at #14 :) It definitely makes sense to me that it's difficult to depend on C++ code in a build-system sense. Some of this I think is the granularity builds today (DAG vs crate) but in general I think it's just flat out unergonomic and difficult to plug in preexisting artifacts into a Cargo build.
Definitely makes sense! I don't think it's out of the question though for Cargo to support custom caching though. In fact, with sccache we may get exactly this! In general I'd like to keep an open mind to Cargo's current implementation today, and we can basically extend it in any way we see fit. For example Cargo's already got enough information to create a unique hash key for a crate and we could restructure it with custom caching to pull in artifacts on demand (or assume they're at a predetermined location) or something like that. Not saying this is a silver bullet of course, but our options are still open!
I'm not sure I quite understand this constraint, so I wonder if we could dig in a bit? I definitely agree that a workspace may not scale to thousands of crates and projects, but the idea of a I guess I'm not fully understanding what's not scaling here. Are you thinking this is a fundamental compiler limitation? Or just something that needs working around in Cargo today? As with above, I'd like to keep in mind the possibility of changes to Cargo to make it more amenable to situations like this rather than assuming the functionality of today is impossible to change!
Yeah I can definitely understand how this may be nonstandard. I don't think this is something that can be sidestepped for too long, though, as a concept. Custom derive (macros 1.1) was stable in Rust 1.15, and that requires compiling a plugin a build-time to then run inside the compiler. I would draw such a practice as very similar to build.rs (in principle at least), and I'd expect that ergonomically using Rust will basically require using Serde in the near future (especially for communicating services). In that sense, is it literally the
I definitely agree! I do think that Cargo's too high level for Buck's use case today, and I personally feel that rustc will almost always be too "low level" to get real benefit. As we continue to add features to Rust, the compiler, and Cargo, the idiomatic and most ergonomic way to consume these features will be through Cargo. For example macros 1.1 might be an absolute nightmare if you had to manage all the builds yourself, especially when cross compiling. I personally think that @davidzchen thanks for the input about Bazel! I'm curious on your thoughts about my comments above related to build scripts as well. Do you agree that compiler plugins (e.g. macros 1.1) are along the same vein as build scripts, or is one much easier to support in Bazel than the other? Or are they both very difficult to support? |
@alexcrichton Regarding compiler plugins and build scripts, the way I see it, the key difference between the two is that people write compiler plugins when they have a need to reuse functionality in the Rust compiler whereas people write One analogy that I can draw is that In any case, neither of these are difficult to support in Bazel:
A main concern that I have with relying too heavily on |
Mechanically it's solved because
Yes. We have C++ libraries that have their own complex dependency graphs managed by Buck that I'd like to add a Rust FFI bindings for, and make sure that everything gets rebuild properly. I'd also like to be able to expose Rust libraries to C/C++ code (mostly for things like python extensions), and again, make sure the build system knows all the deps. Trying to manage dependencies across build systems seems like it could be awkward.
Do you mean Cargo might be able to make use of Buck's cache? That poses lots of problems, not least because its unclear how Cargo would be able to compute the correct key. Buck's cache is indexed by both the immediate dependency (the source file contents), but also the keys of all its dependencies, with the goal of being able to skip as much of the dependency graph as possible. Cargo wouldn't have access to the information needed to either lookup or insert blobs into the cache. Effectively Buck treats the compiler as a pure function of inputs -> output, and memoizes the result. If the build tool is more complex than that, then Buck can't cache its state well, and it complicates the interface to the build tool if its doing its own caching/memoization. I haven't looked a sccache in detail, but this is quite different from how something like ccache works; ccache caches the results of individual compiles, but doesn't take the dependency graph into account.
Yeah, I was being pretty unclear. Without workspaces, every binary cargo package has a dependency graph on other packages with library crates, and building that executable builds them all as needed. If multiple binaries share some or all of the same crates, then they all get rebuilt regardless. You can use workspaces to effectively share the dependencies between multiple binary crates so that the library crates only get built once. But to achieve this, all the crates - binary and library - must be in a single workspace. There's a few issues with this:
I know there's been some discussion about loosening the constraints on workspaces to either allow nesting or have other relationships (esp with path dependencies), but its not clear to me they're the right way of modelling a complex dependency dag in a way to minimize building. (Also I haven't really looked at workspaces in a while, so perhaps I'm completely out of date here, or just wrong.)
I haven't looked at macros 1.1 yet, but are you saying that every crate that uses - say - serde will need to also build a compiler component, or is it just when serde itself is built? If its the latter then I can handle that when I pre-build the crates.io crates, and is basically no more difficult contraint than object files are compiler-version dependent. If they need to be rebuilt for every user, then yeah, that's trickier. The more general problem with Of the use cases listed in the docs, "Building a bundled C library", "Finding a C library on the host system" and "Performing any platform-specific configuration needed for the crate" are all pretty horrifying from a build integrity/reproducability perspective - they are strong antipatterns. The only one that makes any sense is "Generating a Rust module from a specification" (ie, generated source), but that could be done with a much more constrained interface (and perhaps macros 1.1 is that interface). There's also the general security problem of just running random binaries on a build host that can do arbitrary things. It can be managed, but the less it happens the better.
rustc is about the right level for Buck, since its similar to gcc/javac/etc; certainly integrating at the rustc level (while not trivial) was more conceptually straightforward than trying to work out a conceptual mapping between buck and cargo. What might be useful is:
Buck and Bazel are extremely similar in a lot of ways, so I expect that a solution that works for one will likely help with the other. As @davidzchen mentioned, the concept of a compiler plugin is very important for Java, so the Buck can also deal with it (since Java/Android is one of its primary use-cases); extending the concept to Rust is reasonably straightforward. |
This thread is delight to my eyes. |
In Gecko we are effectively limiting things to a single crate per output binary in the build system. We haven't crossed the "Spidermonkey requires Rust" bridge yet, but when we do we will probably just have it behind the existing "building the JS engine standalone" flag, and otherwise have that code pulled in via the crate that gets linked into libxul. We've discussed this in other forums when we first ran into this issue, I know. The core problem was that when outputting something other than an rlib rustc includes support code such as jemalloc, and you can only link one copy of that into a binary. |
We've discussed this before, but no, there's not. I have a cargo issue open for making our Gecko use case nicer. For Gecko we currently vendor all our crates.io dependencies with
This isn't implemented yet (I'm working on it this quarter), but we're planning on making sccache able to cache rust compilations at the crate level. There's a good writeup of the plan here. This is different from ccache, which operates at the object file level, but Rust compilation is fundamentally different from C compilation. |
Ok I had some discussion with @aturon the other day and we felt that the discussion here has led to at least two concrete courses of action to start tackling this issue. To that end I've opened up two new issues on the Cargo issue tracker: Our thinking is that specific technical discussion about those two issues should move towards those issues specifically. Otherwise more general discussion should of course continue here! |
One concern I have is that the "build plan", as described, sounds like a "concrete build plan" - that is, one in which things like Cargo features have been bound, rather than remaining free. This is not compatible with some distros. In particular, on Exherbo, it's desirable to surface the Cargo features of crates as Exheres options on the corresponding packages. As a result, generating Exheres from crates requires something closer to Haskell's This corresponds to a somewhat different set of phases - the ones in the prior listing have been italicized:
The new phase is performed by the system package manager, and this results in a very different workflow. The system PM thus asks Cargo to extract a data structure that has some number of free parameters that, when all are bound, can be queried for a build plan. In this system, Cargo is used to extract this data structure. This data structure is then converted to a system package, which then has those free parameters bound by the system package manager (based on end user configuration). The system package manager then constructs a download plan (of crate sources, though the system PM would much prefer if there was a stable ABI, and it could thus construct the inter-crate build plan entirely by itself), and then invokes Cargo to plan and execute the build. The download plan and the Cargo build plan being mismatched is an error. As a result, unlike the above buildsystem use cases (which basically call in to Cargo), Exherbo's use case basically wants to bookend its own model with exporting information from Cargo, and then reinjecting it to Cargo. |
Big fan of official active and continued support for rust in bazel! This would be very timely for my company. I'm working making the switch from sbt to bazel for one our most active repos and at the same time introduce curious engineers to rust. For engineers being onboarded to a new language within an organization it helps to use a familiar toolchain. That said cargo is fantastic tool and does many things better for certain tasks. This is probably mentioned above but cargo is more than a build tool, it's also a package manager. Bazel is just a build tool and it really excels and just doing that. Losing the package manager aspect is a real drag after having used it. It would be great if cargo could split out the dependency management aspect into a standalone tool which the bazel rules for rust could use |
@sholsapp, there certainly is, would love to try it out. It will be useful when building apps that have modules written in various languages (e.g., Java + Rust communicating over JNI or IPC). Stephen, could you please let me know if you still plan to open-source your Gradle plugin? |
Hi there. I would like to comment on this, as I have been working to integrate Rust into the existing 389 Directory Server project. We use gnu autotools as our build system, and subsequently have a strong requirement to operate with rpm and across a number of other platforms. I think the summary of my position is that cargo is too "opinionated" about builds and projects. Cargo wants to control your source code, output targets and dependcies, and that conflicts most build systems. Cargo will often attempt to download online (even with --frozen - this ruined some live demos for me actually). It also has strict ideas about output of artifacts, especially .so or binaries. A simple task like "cargo, build this bin, and output to this location and name" is difficult to achieve. Cargo is a great tool, and awesome for green field applications, but it's suitability to integrate to other tools is just not there. To make cargo work as a child process to autotools or others, would require stripping cargo down significantly or adding so many switches and extra complexities to it's build process, that it may in turn become too fragile or hard to use. For example, a simple requirement to "cargo install" an object or dylib to a location, is not possible today, because cargo's focus is 100% around rlibs or the final resultant executable. For example, we build a number of dylibs that are then installed and dlopened at run time. We can't use cargo to build these .so. as a result, so we produce .a and .o, then use autotools to finish them and install. This process is already pretty fiddly, and often causes problems. For example, when you edit a source file, autotools doesn't know to re-make it (so you have to mark all targets as PHONY). Another subtle issue is that when we make a -release build we need debug symbols to remain so that rpm can extract them for debuginfo tools, but cargo is "-release" or "-debug". Again, this is because cargo has opinions of how it should be used. As another example, cargo does duplicate work. Consider I have 3 plugins:
Were I to build all of these in my application, they would all pull their rlib deps individually, and would not share the output or work. So my other options is make cargo at the root, and have various specifications of libs and build targets, (which comes with it's own complexity). This then goes back to cargos "opinion". It wants to control the whole build, rather than allowing smaller translation units. In the end, I have chosen to stop using cargo, and likely will not utilise it in projects. I have solved this through the use of git submodules to check out crate dependencies, and careful use of rustc. You can see my example of this as:
I still have the issue with make not knowing when to recompile, but this makes a much clearer and cleaner build for the project. With minimal effort I can now build many .so, they can share rlibs, I do not have to define a make target per object. As well, I gain the ability that autotools has to install shared objects to defined locations. I hope this helps a bit, but I think that rather than investing in cargo to build rust ubiquitously, why not make cargo the build system for greenfields, and "improve" rustc to work for integration to build systems? This would give you the ability to balance the two, where you can have a feature rich system in Cargo, but the required integration components in rustc that external build tools need. Thanks, |
@Firstyear I think some of your issues can already be solved (most of this is in Cargo's docs):
|
I'll answer these points
My points about the opinionated nature of cargo still stand however, it feels like I'm fighting cargo to make it do something it doesn't want to - integration with rustc standalone was much easier for us with our existing sources. Let me put the response a different way: why do we insist on building rust with cargo? why not make rustc a tier 1 build tool of it's own right, and help to improve the rustc build story alongside cargo? Thanks, |
Hm? How so? My first instinct would be to put the following in LIBRARIES := libfoo.so
PROGRAMS := hello
libfoo.so: rust-foo/src/*.rs rust-foo/Cargo.toml rust-foo/Cargo.lock rust-foo/build.rs
cd rust-foo && cargo build
cp -a SO_LOCATION libfoo.so
hello_LDADD := libfoo.so
hello_DEPENDENCIES := libfoo.so As noted in the manual:
As a result, you have the full power of make available to you for producing |
My 2c, having spent the last very long time fighting with what you're fighting with @Firstyear, but for a different build tool.... It actually gets worse -- many crates are super coupled to Cargo. Once you solve the vendoring problem, and the Makefile code generation problem for normal crates, you also will get the fun of handling
I consider the openssl crate my final boss, as it uses all of these features to tremendous effect. Take a look at that and see how you might try compiling it without Cargo. |
Here are my thoughts:
|
Cargo can already create .a and .o, the issue is that for autotools projects we need .la and .lo. Without these, we get a lot of complaints. As well, because autotools doesn't know "what changed", it doesn't know when to trigger the build (arguably, you can make the target .PHONY and let cargo handle it.). Cargo-vendor may alleviate the dist pain issues though. |
@eternaleye That looks like a nice solution but it doesn't work: autotools doesn't like the raw .so, and doesn't know how to feed them to the linker correctly, so it fails to build :( need to produce .a and .o still sadly. One of the biggest challenges is getting cargo to know when to rebuild source. autotools only knows that there is a ".a" or ".o", but not what constitutes them. So as a result, you have to make clean to trigger cargo to actually rebuild. You can't add the target to .PHONY because then you end up building the targets multiple times pointlessly (I saw 8 builds in a complex environment here). A subtle gripe is the output: warning: due to multiple output types requested, the explicitly specified output file name will be adapted for each output type Which no matter what options or changes I make (even just a single -o target) I can never manage to stop it. Similar, when you use staticlib, you get a very very verbose warning about links and no apparent way to quiet it. |
There really should only be one build system that invokes compilers such as |
You're free to re-implement cargo in your own build system, of course. I tried that for a bit, but decided it was too much work. But those are the two choices really, calling cargo or re-implementing it, unless you want to maintain a custom build description for all of your rust-language dependencies. In Firefox we do use the A no-op
I believe you can get rid of this warning by specifying all the output filenames directly. E.g. drop the You can parse the dependent library output for a staticlib and feed it back into your eventual link line. This needs to be dynamic since it can be affected by conditional compilation. It would be nice if the compiler could write that to a file instead. |
Being quite new to rust I may be missing many nuances in this discussion; but coming from python it strikes me that it is the conda (binary) package management that is sorely lacking here. Not that conda is not python-specific but language-agnostic, and also used for quite some C++ projects nowadays. The main 'problem' as it strikes me is that it has quite a bit of overlap with the functionality of cargo; but is there a way for these two tools to put their egos aside and combine elegantly in some manner? |
Is it just the distribution of binary packages (as well as source, like cargo does) that you're interested in, e.g. to save waiting on build times? Or is there something else conda offers which cargo doesn't? I've never used conda, so it would help be understand what you mean if you could offer more details. It would certainly be possible to add a binary caching layer to cargo or crates.io. Possibly by adding some kind of sccache integration to cargo. I think it hasn't been a priority so far because every rust release has an ABI change, but as applications written in rust get more complex and popular being able to install without the compile step gets more interesting. I believe there's also a few more steps to get to reproducible builds, which are important for verifying distributed binaries. |
Build times are one thing; the last python project I worked on had 2gb of binary dependencies; building all of that from source would have been no fun whatsoever; perhaps bordering on practically infeasible. But its mostly the fact that I end up getting dragged into more internal compilation details than I was used to with conda; for instance, the need to have a fortran compiler installed and configured and whatnot. I wasnt planning on getting involved into any of that; I just wanted to use netlib as a dependency. That kind of 'leaking' of concerns across package boundaries isnt going to scale to large and complex dependency graphs as well as conda did. So its not so much the 'distribution' of the binaries that I think is the crucial part (though a considerable convenience); but rather it is the pushing of encapsulation of nontrivial concerns as far as possible which I think makes a large difference for the scalability of an ecosystem. |
I see. It's more about needing a way for crates with |
An eRFC on this topic is now up! |
Okay, so since then I want to refine this: I have some success with .so for binaries, but trying to link two libraries is an issue. We have a plugable system which generates .so which we dlopen, and those libraries link to other libraries. For example: libuiduniq.so -> libslapd.so. Now the issue I face is that we have a library for datastructures which is librsds.so - the library I have a rust poc in. The issue is linking libuiduniq.la to this can not be done. It's quite challenging to make this configuration play nicely. Second - .so versioning. If we use cargo to make the .so, we lose the ability to provide these versions in our .so files, so having cargo output resources that can be used by autotools for .la, and allowing libtool to create the .so will really ease the process of integration. |
Overview
When working with larger organizations interested in using Rust, one of the first hurdles we tend to run into is fitting into an existing build system. We've been exploring a number of different approaches, each of which ends up using Cargo (and sometimes
rustc
) in different ways, with different stories about how to incorporate crates from the broader crates.io ecosystem. Part of the issue seems to be a perceived overlap between functionality in Cargo (and its notion of compilation unit) and in ambient build systems, but we have yet to truly get to the bottom of the issues—and it may be that the problem is one of communication, rather than of some technical gap.By the end of 2017, this kind of integration should be easy: as a community, we should have a strong understanding of best practices, and potentially build tooling in support of those practices. And of course, we want to approach this goal with Rust's values in mind, ensuring that first-class access to the crates.io ecosystem is a cornerstone of our eventual story.
Projects
At this point, we are still trying to assess the problems people face in this area. If you have experience here, please leave a comment with your thoughts!
The text was updated successfully, but these errors were encountered: