make it work with current rust

[fneddy]

needing to recompile a custom version of rustc is very discouraging and inconvenient.

Is it possible to make this work on stable/beta/nightly?
is it possible to install the custom rustc via rustup and switch to it only when working on this?

[nbp]

As of today it is not possible to use a pre-compiled version of rustc.

I do not think the team behind rust would be willing to ship a custom rustc and I am not the right person to ask.

@fneddy ,
Would making a Nix expression to provide a proper environment solves your issue?

[jonas-schievink]

@nbp Since HolyJIT (at the moment) only needs a single patch to work (nbp/rust@9e0d95e), it doesn't seem like it would require too much maintenance effort if it were upstreamed - have to tried to talk to Rust people about that?
Of course, if you know you'll need more rustc patches in the future, this might not be worth it just yet.

[mateon1]

Unfortunately, even the patched rustc tree linked in the readme fails to compile (And issues on that repo are disabled, probably because it's a fork.)
I am on the patched commit, nbp/rust@9e0d95e, however ./x.py build fails to update the Cargo registry.
Rust seems to fail when doing vendored builds, due to incorrectly detecting sudo. This is fixed by unsetting all environment variables containing SUDO in the name.
Unfortunately, the example fails to build due to differences in compiler internals regardless if I use rustc from commit 9e0d95, or if I apply that patch to the current rust master branch.

Errors with nbp/rust@9e0d95e [dynasm fails to build]

error[E0559]: variant `syntax::ext::base::SyntaxExtension::NormalTT` has no field named `expander`
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/lib.rs:45:39
   |
45 |                                       expander: Box::new(dynasm),
   |                                       ^^^^^^^^^ `syntax::ext::base::SyntaxExtension::NormalTT` does not have this field

error[E0559]: variant `syntax::ext::base::SyntaxExtension::NormalTT` has no field named `def_info`
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/lib.rs:46:39
   |
46 |                                       def_info: None,
   |                                       ^^^^^^^^^ `syntax::ext::base::SyntaxExtension::NormalTT` does not have this field

error[E0559]: variant `syntax::ext::base::SyntaxExtension::NormalTT` has no field named `allow_internal_unstable`
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/lib.rs:47:39
   |
47 |                                       allow_internal_unstable: false,
   |                                       ^^^^^^^^^^^^^^^^^^^^^^^^ `syntax::ext::base::SyntaxExtension::NormalTT` does not have this field

error[E0559]: variant `syntax::ext::base::SyntaxExtension::NormalTT` has no field named `allow_internal_unsafe`
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/lib.rs:48:39

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:343:26
    |
343 |     let span = startspan.with_hi(parser.prev_span.hi());
    |                          ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:343:51
    |
343 |     let span = startspan.with_hi(parser.prev_span.hi());
    |                                                   ^^ field, not a method
    |
    = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:431:44
    |
431 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                            ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:431:69
    |
431 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                                                     ^^ field, not a method
    |
    = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:438:44
    |
438 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                            ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:438:69
    |
438 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                                                     ^^ field, not a method
    |
    = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:445:44
    |
445 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                            ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:445:69
    |
445 |             Ident {node: name, span: start.with_hi(parser.prev_span.hi()) }
    |                                                                     ^^ field, not a method
    |
    = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error[E0599]: no method named `with_lo` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:461:30
    |
461 |         let span = expr.span.with_lo(span.lo());
    |                              ^^^^^^^

error[E0599]: no method named `lo` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:461:43
    |
461 |         let span = expr.span.with_lo(span.lo());
    |                                           ^^ field, not a method
    |
    = help: did you mean to write `span.lo` instead of `span.lo(...)`?

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:511:29
    |
511 |                 span: start.with_hi(parser.prev_span.hi()),
    |                             ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
   --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/arch/x64/parser.rs:511:54
    |
511 |                 span: start.with_hi(parser.prev_span.hi()),
    |                                                      ^^ field, not a method
    |
    = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error[E0599]: no method named `with_hi` found for type `syntax::codemap::Span` in the current scope
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/directive.rs:67:44
   |
67 |                         ecx.span_err(start.with_hi(parser.prev_span.hi()),
   |                                            ^^^^^^^

error[E0599]: no method named `hi` found for type `syntax::codemap::Span` in the current scope
  --> /home/mateon/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/dynasm-0.1.3/src/directive.rs:67:69
   |
67 |                         ecx.span_err(start.with_hi(parser.prev_span.hi()),
   |                                                                     ^^ field, not a method
   |
   = help: did you mean to write `parser.prev_span.hi` instead of `parser.prev_span.hi(...)`?

error: aborting due to 18 previous errors

error: Could not compile `dynasm`.

Errors with patch applied on top of rust-lang/rust@7e70546 [holyjit_plugin fails to build]

   Compiling holyjit_plugin v0.0.0 (file:///shared/dev/rust/holyjit/plugin)
error[E0599]: no method named `as_str` found for type `syntax_pos::symbol::InternedString` in the current scope
   --> plugin/src/lib.rs:109:57
    |
109 |                 let wrpr = wrpr.get_opt_name().unwrap().as_str();
    |                                                         ^^^^^^
    |
    = help: did you mean `as_ptr`?

error[E0308]: mismatched types
   --> plugin/src/lib.rs:170:42
    |
170 |             mir::Literal::Value { value: ConstVal::Function(did, _) } => did,
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found enum `rustc::middle::const_val::ConstVal`
    |
    = note: expected type `&rustc::ty::Const<'_>`
               found type `rustc::middle::const_val::ConstVal<'_>`

error[E0277]: the trait bound `&rustc::ty::Const<'_>: std::cmp::PartialEq<{integer}>` is not satisfied
   --> plugin/src/lib.rs:222:34
    |
222 |                             if l != 1 {
    |                                  ^^ can't compare `&rustc::ty::Const<'_>` with `{integer}`
    |
    = help: the trait `std::cmp::PartialEq<{integer}>` is not implemented for `&rustc::ty::Const<'_>`

error[E0308]: mismatched types
   --> plugin/src/lib.rs:225:71
    |
225 |                             let arr_ty = ty::TypeVariants::TyArray(t, bytes.len());
    |                                                                       ^^^^^^^^^^^ expected reference, found usize
    |
    = note: expected type `&rustc::ty::Const<'_>`
               found type `usize`

error[E0277]: the trait bound `&rustc::ty::Const<'_>: std::cmp::PartialEq<{integer}>` is not satisfied
   --> plugin/src/lib.rs:235:26
    |
235 |                     if l != 1 {
    |                          ^^ can't compare `&rustc::ty::Const<'_>` with `{integer}`
    |
    = help: the trait `std::cmp::PartialEq<{integer}>` is not implemented for `&rustc::ty::Const<'_>`

error[E0308]: mismatched types
   --> plugin/src/lib.rs:242:63
    |
242 |                     let arr_ty = ty::TypeVariants::TyArray(t, bytes.len());
    |                                                               ^^^^^^^^^^^ expected reference, found usize
    |
    = note: expected type `&rustc::ty::Const<'_>`
               found type `usize`

error[E0308]: mismatched types
   --> plugin/src/lib.rs:259:58
    |
259 |         let mk_literal = |i| mir::Literal::Value{ value: ConstVal::Integral(ConstInt::U8(i)) };
    |                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found enum `rustc::middle::const_val::ConstVal`
    |
    = note: expected type `&rustc::ty::Const<'_>`
               found type `rustc::middle::const_val::ConstVal<'_>`

error[E0308]: mismatched types
   --> plugin/src/lib.rs:476:25
    |
476 |                         mir::Lvalue::Local(idx.clone())
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `rustc::mir::Local`, found enum `rustc::mir::Lvalue`
    |
    = note: expected type `rustc::mir::Local`
               found type `rustc::mir::Lvalue<'_>`

error[E0308]: mismatched types
   --> plugin/src/lib.rs:500:25
    |
500 |                         mir::Lvalue::Local(idx.clone())
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `rustc::mir::Local`, found enum `rustc::mir::Lvalue`
    |
    = note: expected type `rustc::mir::Local`
               found type `rustc::mir::Lvalue<'_>`

error[E0308]: mismatched types
   --> plugin/src/trans.rs:589:34
    |
589 |                     self.operand(operand).or_else(
    |                                  ^^^^^^^ expected enum `rustc::mir::Operand`, found struct `rustc::mir::Local`
    |
    = note: expected type `&rustc::mir::Operand<'_>`
               found type `&rustc::mir::Local`

error[E0308]: mismatched types
   --> plugin/src/trans.rs:837:48
    |
837 |                         (self.tcx.mk_array(ty, operands.len()), 0),
    |                                                ^^^^^^^^^^^^^^ expected u64, found usize

error[E0223]: ambiguous associated type
   --> plugin/src/trans.rs:912:21
    |
912 |                     mir::Literal::Item { .. /*def_id, substs*/ } => {
    |                     ^^^^^^^^^^^^^^^^^^ ambiguous associated type
    |
    = note: specify the type using the syntax `<rustc::mir::Literal<'_> as Trait>::Item`

error[E0308]: mismatched types
   --> plugin/src/trans.rs:915:83
    |
915 |                     mir::Literal::Value { ref value } if self.constval_use_static(value) => {
    |                                                                                   ^^^^^ expected enum `rustc::middle::const_val::ConstVal`, found reference
    |
    = note: expected type `&rustc::middle::const_val::ConstVal<'_>`
               found type `&&rustc::ty::Const<'_>`

error[E0308]: mismatched types
   --> plugin/src/trans.rs:951:42
    |
951 |                         Ok(self.constval(value, constant.ty).or_else(
    |                                          ^^^^^ expected enum `rustc::middle::const_val::ConstVal`, found reference
    |
    = note: expected type `rustc::middle::const_val::ConstVal<'_>`
               found type `&rustc::ty::Const<'_>`

error: aborting due to 14 previous errors

error: Could not compile `holyjit_plugin`.

[nbp]

The way I am working today is using the binary which is in build/x86_64-unknown-linux-gnu/stage2/bin/.
I am using the rustc nightly from 2017-05-30 to build it.

[jonas-schievink]

@mateon1 The correct way is to build the forked repo (nbp/rust).

Your dynasm build failure seems to happen because 5 days ago, @CensoredUsername released dynasm 0.1.3 on crates.io (but it seems that the repo wasn't updated, might want to fix that if you read this), which broke something. Since dynasm is itself a compiler plugin, it frequently needs to be updated to an up-to-date nightly. Fixing its version in the Cargo.toml might be required.

[jonas-schievink]

@mateon1 Try building #17 with the first rustc you've built (nbp/rust@9e0d95e)

[nbp]

@jonas-schievink Thanks for investigating this issue. I remove my Cargo.lock and was able to reproduce the compilation issues.

I just pushed a patch to downgrade dynasm to 0.1.1 and serde_derive to 1.0.12.
You should be able to build against the old version of rustc now.

[nbp]

@jonas-schievink I discussed multiple times on the rustc irc channel, and the plugin approach is (was?) a no-go for the compiler team, as it exposes internals of the compiler which would never be stabilized.

I do not intent to add any additional patches to rustc at the moment, and I do not think this would be necessary.

This patch is a mutated version of a feature which used to be in the compiler, and got remove earlier this year by @nagisa and @nikomatsakis in rust-lang/rust#40239 after I mentioned that I might rely on this interface for making this prototype. One of the reason of the removal of the feature, was to be able to break the API, and avoid stabilizing the MIR. Thus, unless the publication of this project can changed the whole story, I think I will have to find a different way of doing so.

I intent to discuss this topic in-person mid-December with @nikomatsakis and also some persons responsible of Rust integration in Firefox such as @bholley and @glandium.

[jonas-schievink]

Ah, that makes sense, I vaguely remember that PR.

[CensoredUsername]

@jonas-schievink whoops, thanks for the reminder. Dynasm 0.1.3 should be pushed now.

[bjorn3]

rust-lang/rust#45916

[nbp]

This issue is mostly fixed by bfb32d6 which converts HolyJit to a rustc-driver instead of a compiler plugin, which implies that we no longer depend on a patched version of rustc.

As of this commit, HolyJit is now based on a non-patched version of rustc (the nightly version from 2018-02-19), which should make this project more accessible for everybody.

The version of rustc on which HolyJit is based can be found in the release.nix file at the top-level. This script is using the rust overlay from nixpkgs-mozilla. I personally use it as follow:

$ nix-shell ~/rust/holyjit/release.nix -A holyjit
nix-shell$ cargo build --all
nix-shell$ cargo build --example brainfuck
nix-shell$ cargo run --example brainfuck