Experimental JIT compiler

[z0w0]

This is my 3rd attempt to get this working over around 4 months. It's working pretty well compared to last attempts.

I've found that it runs perfectly (slow parsing/compiling times, fast executing times) with most tests running fine. Most benchmarks actually run faster than statically compiled executables! Of course, it's not perfect yet and there are some issues:

• No crates are explicitly linked in yet - this is easy to do code-wise, the issue is llvm::sys::DynamicLibrary::LoadLibraryPermanently is segfaulting when I try and load crates. The code to do this (pretty simple) is commented out and ready to be fixed (I imagine the fix is something simple)
• std and core are the only crates available because of above - they're only available because the JIT compiler uses dlsym to resolve symbols from the main program and any loaded crates - hence it loads the std and core crates used by rustc. This is not a good idea, as users may want to use specific versions of the core libraries, so it has to be fixed with above
• Running some code/tests or using --test results in terminate called after throwing an instance of 'rust_task*' - I do not know enough to understand what's causing that
• Minor/easy thing: You can't pass along arguments to the jitted program yet

These issues can easily be fixed in patches (I'll investigate the first two issues) but I'm not sure about the 3rd one. #rust were pretty keen to see this pulled upstream and I think it's ready for that.

Also important to mention is I based the memory manager source code on LLVM's lli tool (most of the original comments remain) because it worked well with what was needed and it had fixed issues that I knew were going to be hit (also it was the only good example of JIT memory managers that I could find - MCJIT managers require them in order to run). Not sure what sort of things need to be done there in terms of licensing / credits / etc.

[brson]

This looks great. Linking rustllvm to morestack turns out to be problematic because morestack expects two symbols from the Rust runtime to exist. Apparently this isn't a problem on linux, but on mac it doesn't link.

See the errors on mac here: http://bot.rust-lang.org/logs/2012/07/28/2012-08-28T18:37:31Z-f5a3db41-389e-47f0-ab45-e35d9374bac0.html

We can possibly work around this further by linking rustllvm to rustrt.

[brson]

For the licensing issue, probably what we should do is add a comment near the code saying where it is copied from, then in LICENSE.txt add some text like 'Portions of the code for integrating LLVM, under src/rustllvm, is derived from the LLVM codebase, under the following license.' then list the whole license.

[brson]

Here's a patch that makes the address of __morestack available through the 'morestack_addr' intrinsic. It looks to me like it works correctly. You could use from within rustc to grab the correct address and pass to rustllvm.

brson@b74e625

[brson]

This commit adds JIT testing support to the main test harness:

brson@64dc43c

Use it with make check-stage2-rpass TESTARGS=--jit. These are my results for the run-pass tests: '785 passed; 58 failed; 60 ignored'. Pretty amazing!

[brson]

OK, this commit runs the run-fail tests through the JIT as well, and they all fail with the 'terminate called after throwing ...' error.

brson@25a0f74

[brson]

We're going to need to think a bit about what it is that the JIT should actually do. What we're doing here effictively starts an entirely new runtime by calling "main". This will have some subtle consequences, partly because the Rust runtime isn't entirely threadsafe when there are multiple instances, and also because the TLS usage of the second runtime will, in some cases, end up clobbering the TLS of the original runtime.

More importantly though, what we really want is to run the JIT, then get the address of the Rust main function, then begin executing that function from within the Rust stack.

This is all for later though.

[z0w0]

I understand, I didn't think about how there would be two instances of the runtime. Do you think starting two runtimes could have something to do with the terminate error? WRT spawning it inside the compiler's runtime, I don't know how to do that, so if you want to do it / point me to the correct way I'll be happy to. Also can you explain the significance of the intrinsic commit? I don't know what it does or has to be done to make the symbol available.

EDIT: I think I understand. The intrinsic is a custom ABI that I can use to pass the address to rustllvm.

[brson]

"rust-intrinsic" is a special ABI for native functions that refers to a few built-in functions that the compiler can emit on demand. It's mostly for very low-level stuff like getting the alignment of types. The 'morestack_addr' intrinsic tells rustc to create a function that just returns the address of __morestack. Intrinsic extern modules may be declared anywhere. The attached test case illustrates how to use it though it may need to get reinterpret_casted to some other pointer type. Fixing this is the minimum we need to do to get this building everywhere and merged.

I do think that the extra runtime could be related to the terminate error, though I haven't imagined exactly how.

It looks to me that the Rust ABI main function is called "_rust_main", so instead of asking for the address of "main", then executing it inside of rustllvm, we should be able to ask for "_rust_main", return it to rustc, then rustc can use it to synthesize a closure without an environment and call it.

This commit has an example of synthesizing a closure: 6e20ffe

[z0w0]

Seems to be working fine! If someone can test this fixed the build issues on OSX, that would be great. I'm trying to get crate loading working and then everything should be good.

EDIT: The error message is /usr/local/lib/rustc/i686-unknown-linux-gnu/lib/libcore-c3ca5d77d81b46c1-0.3.so: cannot read file data when I try to load the crate. However, when I try to do this in a plain C program it runs fine. It's very strange. I've tried Googling around, really can't find anything on what that error message is implying.

[brson]

I ran your latest revision through the try branch on the bots and they were mostly happy. The windows bot failed but I suspect it's unrelated to your changes. Unfortunately I've been busy with other things and won't be able to merge this until tomorrow.

[brson]

Pushed to incoming.