-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make some aspects of check/build available as an API. #3427
Conversation
requires rust-lang/cargo#3427 closes #82 cc #90
@@ -289,7 +325,7 @@ fn rustc(cx: &mut Context, unit: &Unit) -> CargoResult<Work> { | |||
} | |||
|
|||
state.running(&rustc); | |||
if json_messages { | |||
let cont = if json_messages { | |||
rustc.exec_with_streaming( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that this part bypasses the Mutltishell
and uses standard streams directly (machine_message::emit
and #3410). I don't think you'll hit this when using Cargo from RLS, but it's something to keep in mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I'm slightly worried about this in terms of only bits and pieces of Cargo's API will work with these custom arguments. @nrc is this just avoided by the rls?
☔ The latest upstream changes (presumably #3413) made this pull request unmergeable. Please resolve the merge conflicts. |
use util::{CliResult, Config}; | ||
|
||
#[derive(RustcDecodable)] | ||
pub struct Options { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These structures typically are command-specific, so I'd prefer to leave this in src/bin/check.rs
. The functionality though should all be configurable, I believe, through the options generated later. Was something missing though perhaps?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was trying to keep calls via the API as close to calls via the CLI. I'll take another look and see what I can figure out
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yeah the intention was that all the CLI options have their own suite of flags and such but everything "compiles down" to CompileOptions
where that's the common functionality amongst everything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea!
Could you elaborate more to the purpose of ContinueBuild
? It looks like it may not cancel the entire build but just the copy of artifacts right after the compile? If we do support that I'd expect the job graph to stop executing as soon as possible once it sees a "stop"
|
||
pub fn compile_with_exec<'a, E: Executor>(ws: &Workspace<'a>, | ||
options: &CompileOptions<'a>, | ||
exec: &mut E) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this take &mut Executor
(trait object) to cut down on monomorphization?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm so Executor: Clone
could throw a wrench into this, but perhaps that could be jerry-rigged with something like:
fn clone(&self) -> Box<Exectuor>;
(or something else manual like that)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is monomorphisation a problem? I'd have thought using generics is more efficient here (the common case is to just have a single Executor) and adding an extra param doesn't seem too much of a burden.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The type parameter ends up getting plumbed all the day down throughout cargo_rustc
, which is the majority of the code in Cargo. I'd prefer to keep Cargo's own compile times down and there's not really a need for speed here, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This shouldn't affect Cargo's compile times at all - there is only one actual type which implements Executor, so unless you're using Cargo as a lib, exactly the same amount of code is processed as before. The only downside is we add a generic parameter in a few places, which is a bit ugly, but is also kind of best practice (IMO). Speed is important here, but you are right in that the speed improvement (vs trait object) is unlikely to be important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm I'd still very much prefer to have this be a trait object, can we work out how to make the trait object safe?
re ContinueBuild, the idea is that it applies only to building a single Unit, so if you return Stop, you stop the build process for that unit, but continue to build others. This is useful because Cargo assumes it has created some artefacts and wants to do stuff with them, but if you've done things differently, you might not create those artefacts and thus Cargo errors out. |
Updated with check's Options back in bin/check.rs |
ping @alexcrichton for re-review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh oops sorry! I think this fell out of my inbox by accident..
With ContinueBuild
I was under the impression that Cargo already handled commands that didn't actually generate any output? I know that cargo rustc
, for example, isn't guaranteed to produce output and we should handle that.
I'm just slightly wary of adding that feature which others may expect to fully stop the build on Stop
whereas this just stops that step, and likely causes failures down the road for other dependent steps, right?
|
||
pub fn compile_with_exec<'a, E: Executor>(ws: &Workspace<'a>, | ||
options: &CompileOptions<'a>, | ||
exec: &mut E) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The type parameter ends up getting plumbed all the day down throughout cargo_rustc
, which is the majority of the code in Cargo. I'd prefer to keep Cargo's own compile times down and there's not really a need for speed here, right?
} | ||
|
||
impl Options { | ||
pub fn default() -> Options { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is in theory only ever used in this file (not exported), so I'm curious why this was needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, it's probably a remnant of when I had this as API, it can probably be removed now.
where F: FnOnce(&Workspace) -> CliResult<Option<()>> | ||
{ | ||
let root = find_root_manifest_for_wd(flag_manifest_path, config.cwd())?; | ||
let ws = Workspace::new(&root, config)?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like not a lot of code to inline locally perhaps? If we keep this function, though, could it just return a Workspace
instead of taking a closure?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there was a borrow check issue with doing that - ws
has to have a shorter lifetime than root
. I agree it is kind of a small function, but there is already a little bit of code that needs to be inlined (to make the config) and it seemed like voodoo from the client's perspective - there seems no reason to allow the flexibility - so I thought it better to present a simpler API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm just always hesitant about closure-taking functions and such. Cargo's API already isn't really the most ergonomic and I just prefer to keep most operations as only having one consistent method of being done. For example this PR doesn't update all subcommands to use this method, so it's possible for them to start to diverge in behavior from cargo check
sorta...
I'm fine making these sorts of API patterns aliased with something but I'd prefer to avoid the closure protocol wherever possible (e.g. by changing the API of Workspace
or something like that) and to ensure that Cargo uses the pattern consistently internally if it does so.
@@ -62,6 +62,29 @@ pub struct CompileOptions<'a> { | |||
pub target_rustc_args: Option<&'a [String]>, | |||
} | |||
|
|||
impl<'a> CompileOptions<'a> { | |||
pub fn with_default<F, T>(config: &Config, mode: CompileMode, f: F) -> T |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this just be a derive(Default)
basically?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's kind of annoying with the borrows here - there needs to be a lifetime for 'a
(which is why this takes a closure, rather than returning a CompileOptions
- I think the only way to do this would be to take a bunch of empty slices as args, which is pretty annoying. i don't think we can derive here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have Default for &[]
though?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried again, and I don't think we can implement Default, because we want to take the config and mode as arguments, but we can just add a default
method, we can get rid of the taking a closure thing.
if let Ok(meta) = fs::metadata(path) { | ||
let mtime = FileTime::from_last_modification_time(&meta); | ||
*slot.0.lock().unwrap() = Some(mtime); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come this was changed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think because with the RLS, we don't actually produce the file at path
, and so we would get an error (even though we return Stop
, there was no way to skip this code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm I'd prefer to not change this in exactly this manner. This seems like it's just waiting to accidentally ignore errors and then not cause rebuilds when they should happen otherwise. Could such an error like this be caught elsewhere and handled?
For example I could imagine something higher up catching this error and leaving behind a tombstone for "this must be recompiled", or something like that.
@@ -289,7 +325,7 @@ fn rustc(cx: &mut Context, unit: &Unit) -> CargoResult<Work> { | |||
} | |||
|
|||
state.running(&rustc); | |||
if json_messages { | |||
let cont = if json_messages { | |||
rustc.exec_with_streaming( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I'm slightly worried about this in terms of only bits and pieces of Cargo's API will work with these custom arguments. @nrc is this just avoided by the rls?
I am in general worried about the idea of linking Cargo into the rls... I can't put my finger on exactly what I don't like here though. I feel that the project description produced by Of course you'll need cargo to build compiler plugins and execute build scripts, but this can be done outside of rls I think? If user adds a new plugin, or changes Intercepting compiler calls seems too crude. I don't see how it can be extended to leverage the crucial optimization of the IDEs: analyzing only the files currently opened in the editor and ignoring most of the code. |
@matklad the way the RLS uses Cargo is only on initial build in order to build dependencies and find out how the project ought to be built. Once we've got that information, we use the compiler directly. I.e., we only do the compiler call intercept once, and then do our own thing. |
@nrc I wonder if |
@alexcrichton we'd still need to have some API for avoiding the build of the last crate. If we're doing that, it seems we may as well get the data we need directly rather than by parsing JSON. We might also want to further customise in the future. |
@nrc true yeah, I was just thinking that downside may not be worth linking to all of Cargo and bringing it in, but I'm fine either way. |
This is not true, at least not in the way in which we are doing things. I believe the difference is that in this case Cargo expects rustc to make an output and we are signalling to Cargo that the build was successful, and in that case, if there isn't an output, Cargo panicked. We could probably fix this by making Cargo less picky - e.g., check for files rather than assuming they are there, and if not silently skipping steps. My assumption was that we'd like to keep this strictness, but if you prefer we could be lax instead. Alternatively, we could try and be more explicit that |
Yeah, the RLS doesn't ask for JSON messages (we ignore any messages from any crates we compile with Cargo) so this is fine. It is kind of ad hoc, but I think the only way to make this nicer is to complicate the API. I'll take another look and see what I can do though. |
So, I fixed the json messages stuff by adding an |
And what exactly is "the last crate"? What if you are compiling a workspace, or have path dependencies? That way, several crates may be in "modified" state (their files are changed in the editor, but are not save to disk yet), and they can depend on each other. |
Honestly, I think we have to leave a lot of these issues till later. The current assumptions are that you'll edit a single crate and save files when changing crates. Clearly this isn't good enough for the long term, but I don't think there is anything fundamental in our approach which prevents us solving this in the future. The API here allows us to check any crate we like, not just the last one, so it should be extensible in the right direction. |
@@ -62,6 +62,18 @@ impl MultiShell { | |||
MultiShell { out: out, err: err, verbosity: verbosity } | |||
} | |||
|
|||
// Create a quiet, basic shell from supplied writers. | |||
pub fn from_write(out: Box<Write + Send>, err: Box<Write + Send>) -> MultiShell { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is necessary, right? You should be able to create a shell via Shell::create
and then use MultiShell::new
above, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could, it felt like it was exposing a lot of unnecessary details to clients though (need to know about Shell
and ShellConfig
and Verbosity
, where the abstraction here is "send everything to me instead of the console and let me deal with it").
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True yeah. Ok seems fine
@nrc I'm ok not solving everything up front, but this currently definitely has a "grafted on at the last minute" feeling which'll be difficult to support in the future as we continue to change Cargo and adapt it to the RLS's needs. If taking the route of |
@alexcrichton my comment was in reply to @matklad, not to any of your comments. |
@alexcrichton I must admit I don't have much idea of how Cargo will evolve, so I've not been able to take that into account here. The trouble I think is that what the RLS wants to do is run as much like a normal Cargo run as possible, but tweak one little thing. The trouble is that that one little thing is deep inside Cargo. Therefore it seems very difficult to design an API where the client can just call various functions to get the behaviour it wants (without the client having to recreate a bunch of Cargo's behaviour, which is obviously fragile). An alternative approach might be to have either a flag for 'skip building the last/primary crate' or to take a list of crates not to build. However, these feel overly specific and thus hacky. |
You can specify the set of packages to build with |
Hm yeah what @matklad makes sense to me, and going further it seems like you could do something like:
That way when your own binary is run with |
@alexcrichton this doesn't quite work - we don't want to use a rustc binary, we want to call bits of rustc as a library. This is also what we did before - we had a bogus argument which caused compilation to fail and parsed the command line - we agreed that that was a pretty gross hack. It seems rather a lot of work to get all the packages, parse that info, cut something out of the list, pass it back to Cargo, etc. That seems both fragile and hacky to me (plus kind of a lot of work on the RLS side). Furthermore, I would expect the RLS to work more closely with Cargo in the future, not less, so using an API will make more and more sense. At this stage the API we're adding feels pretty minimal - the only real change is letting clients to, effectively, |
What exactly are the arguments you are interesting in? I think there's no a single set of arguments to build the crate. What I mean is that Cargo can pass different arguments depending on the circumstances (profiles in |
We want to get exactly the arguments that rustc would get if it were called by Cargo. I.e., taking into account the Cargo.toml and any arguments to Cargo etc., then any processing that Cargo does. I.e., exactly what rustc would be passed, which is what we get if we intercept the call to rustc as I'm doing here. |
@alexcrichton so, if I understand you, you're main concern is that we might be making life in the RLS harder for us, rather than that we are doing bad things to Cargo? Given that for the RLS using this API is easier than not, and the code is already written (in both the RLS and Cargo), it seems conservative to take this route and only do the extra work in the RLS if we need to (i.e., if it turns out to be a problem linking Cargo). |
@nrc to be clear I still don't mind changing Cargo, I just figured it's probably best to work out the best solution regardless of changes necessary first. I am indeed worried that this approach in this PR may be more painful for the RLS in the long term, yeah. The downsides in this PR I believe are:
My thinking with the cargo-as-command-line approach is that the RLS does: let mut cmd = Command::new("cargo");
cmd.arg("rustc");
add_target_flags(&mut cmd); // e.g. --lib, --bin foo, etc
cmd.env("RUSTC", env::current_exe());
cmd.env("IS_RUSTC_SHIM", "1");
cmd.env("MY_ARGS", &tmp_file_path);
cmd.arg("--");
cmd.arg("--my-dummy-argument");
run(cmd);
let rustc_args = File::open(&tmp_file_path().read().split(' '); then later, inside the RLS's fn main() {
if env::var("IS_RUSTC_SHIM").is_ok() {
rustc_shim();
} else {
// ...
}
}
fn rustc_shim() {
let args = env::args().skip(1).collect::<Vec<_>>();
if args.contains("--my-dummy-argument") {
File::create(env::var("MY_ARGS")).write(args.join(" "));
return
}
let mut rustc = Command::new("rustc");
rustc.args(&args);
rustc.exec();
} That is, I don't think this requires IPC, platform-specific code, etc. There's an unambiguous synchronization point of when Cargo ends, and if the Does that make sense? Or does that snippet seem like it won't work for the RLS's purposes? |
This is a valid concern. I'm not sure how much weight it is worth putting on an extra 50MB download, but it seems non-trivial at least. I suppose we could dynamic link to get around this, if it is an issue?
I feel this is not as a big an issue you make it out to be - in either case we have to consider the RLS - under your proposal we have to be certain we are not changing the semantics of cargo commands which are used in a way which regular users would never do, or change the performance of critical bits, etc. In the API approach, we are using Cargo very much like it would be used from the CLI, I believe that only major changes would affect the RLS and in cases such as moving to async IO, I think considering the RLS's API would be very minor compared to the rest of the work.
I think if we are using Rustup for distribution, this is unlikely to happen. The code you present is simpler than I expected, using a file is much simpler than the IPC I was expecting. However, I still feel kind of bad about spawning an extra process and writing to a file, seems very inefficient compared to a function call. It doesn't quite do what we want though - I want to use |
There are two key parts to this commit: * let API clients run `cargo check` with minimal fuss (ops/cargo_check.rs), * let API clients intercept and customise Cargo's calls to rustc (all the Executor stuff).
Mostly focussing on the ergonomics of the API - removes with_* methods with closures, and replaces generics with trait objects.
Remove the ContinueBuild concept, be lenient about missing files in all cases, and undo the fingerprint changes (not an issue now we're not stopping the build early).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Just two minor nits
@@ -47,7 +47,7 @@ impl Config { | |||
rustdoc: LazyCell::new(), | |||
extra_verbose: Cell::new(false), | |||
frozen: Cell::new(false), | |||
locked: Cell::new(false), | |||
locked: Cell::new(false), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trailing whitespace
// if let Ok(meta) = fs::metadata(path) { | ||
// let mtime = FileTime::from_last_modification_time(&meta); | ||
// *slot.0.lock().unwrap() = Some(mtime); | ||
// } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
commented out?
@alexcrichton nits addressed |
@bors: r+ |
📌 Commit 17a0fdc has been approved by |
⌛ Testing commit 17a0fdc with merge 97b698f... |
💔 Test failed - status-travis |
Updated to fix tidy errors. @alexcrichton please to re-summon bors |
@bors: retry |
@bors r=alexcrichton |
I apparently don't have sufficient magic powers to resummon the bors :) |
@bors: r+ oh oops all that and I should have r+'d instead of retry'd |
📌 Commit e50dc1a has been approved by |
Make some aspects of check/build available as an API. There are two key parts to this commit: * let API clients run `cargo check` with minimal fuss (ops/cargo_check.rs), * let API clients intercept and customise Cargo's calls to rustc (all the Executor stuff). r? @alexcrichton
☀️ Test successful - status-appveyor, status-travis |
There are two key parts to this commit:
cargo check
with minimal fuss (ops/cargo_check.rs),r? @alexcrichton