Riscv64 support?

[lachlansneff]

Apparently llvm has Risc-V64 support now, so is there a timeline on supporting a riscv64 toolchain for rust?

[japaric]

cc @rust-embedded/riscv

What kind of target do you have in mind? Bare metal or Linux?

A bare metal target like riscv32imc-unknown-none-elf needs less things from LLVM. For example, it's OK if the LLVM backend doesn't fully support atomics.

A Linux target needs, off the top of my head, fully working support for atomics and ELF TLS in the LLVM backend. Also, the standard libraries contains pieces of C code that need to be cross compiled to riscv64 using gcc: maybe unwinding support, backtrace support, jemalloc, etc. some you may be able to omit depending on which libc you use (gnu or musl).

[lachlansneff]

I'm looking into porting nebulet to risvc64, which would require riscv64imac-unknown-none-elf.

[japaric]

In that case the required work would involve:

• Upgrading the LLVM submodule if the current version rustc is using doesn't have the riscv64 support you need. This is the hardest part.

• Adding a built-in target. See this commit.

• Enabling builds of the rust-std component (pre-compiled core and compiler-builtins) for the new target iff the target is stable enough (no LLVM errors when compiling core and other crates). See this commit.

If someone volunteers to do the job I can review a PR that does steps 2 and 3. I can't, however, help with step 1; you are best asking the compiler team for help on that one.

[xxuejie]

Hey I dig into this problem a little bit, and after some trials I got a working prototype at here: https://github.com/xxuejie/rust/tree/riscv64

The first issue I encountered, is that Rust's current LLVM version has too strict checking: https://github.com/rust-lang/llvm/blob/caddcd9b9dc9479a20908d93c3e47c49b021379e/lib/Target/RISCV/RISCVISelLowering.cpp#L301. Luckily, this problem has been resolved in upstream LLVM: llvm-mirror/llvm@22abcbc#diff-e08aaff1bcaf5b99c78106d5d7d2f2e5. So updating LLVM submodule version used in Rust could solve this. In my prototype, I just cherry-picked the needed commits.

After that, another problem is that there're still instructions unimplemented even in upstream LLVM: https://github.com/llvm-mirror/llvm/blob/master/lib/Target/RISCV/RISCVInstrInfo.td#L477-L481. It seems the relevant part in this patch: https://github.com/lowRISC/riscv-llvm/blob/master/0044-RISCV-Add-initial-RV64I-codegen-support.patch is left out when merging to upstream. I guess maybe we will need a little help from @asb on the current status. In the meantime for my prototype I just copy over the code from the patch: xxuejie/llvm@8f01ea4, and the result is a Rust version that can compile to RISCV64.

So the above are 2 issues we need to solve to make Rust work. If you guys think this is a legit path I can start working on getting those changes into proper PRs for upstreams.

[asb]

Thanks for tagging me - I'm the code owner and primary author of the upstream RISC-V LLVM backend.

I've been working on getting RV64 support upstream. I recently authored an RFC to get feedback on the implementation approach: http://lists.llvm.org/pipermail/llvm-dev/2018-October/126690.html

I've followed that up with a stream of patches currently under review on Phabricator. These implement RV64IMAFD (and we get 'C' support for free based on the current upstream code). Note that in these in-flight patches, codegen for atomic cmpxchg isn't implemented on RV64, as I'm blocked on final review of my RV32A cmpxchg patch (see my RFC on safe atomics lowering for more background if interested).

Codegen quality is much better than the proof of concept patch previously shared.

See:

• Refactoring in preparation for immediate lowering https://reviews.llvm.org/D52961
• Immediate lowering https://reviews.llvm.org/D52962
• RV64I https://reviews.llvm.org/D53233
• New target hook to improve codegen https://reviews.llvm.org/D52978
• RV64M https://reviews.llvm.org/D53230
• RV64A (minus cmpxchg) https://reviews.llvm.org/D53233
• RV64F https://reviews.llvm.org/D53235
• RV64D https://reviews.llvm.org/D53237

I just updated a bunch of these after getting back from the LLVM Dev Meeting last week, so hopefully the review process will start to move forwards and we can get these merged.

I'm really excited to see interest in RV64 support in Rust and I'm keen to do what I can to support you in this. The immediate priority is pretty clear - get these patches merged upstream. Beyond that, please keep tagging me in issues that I can help with and of course file bug reports on bugs.llvm.org when appropriate.

[xxuejie]

Thanks @asb for your reply and awesome work! It's great to hear that RV64 support is in progress. We are also interested in contributing as much as we can to support RV64 port in Rust(and maybe also help test LLVM code for RV64 :P). In the meantime I guess we will wait for the new patches to land in upstream, and see how we can work from there.

[jethrogb]

• RV64I https://reviews.llvm.org/D53233

Probably meant https://reviews.llvm.org/D52977

[tarcieri]

Great news @asb!

What kind of target do you have in mind? Bare metal or Linux?

For me, initially a riscv64-unknown-linux-gnu or thereabouts target would be useful to me as I'm targeting the HiFive Unleashed, which comes preinstalled with Linux.

Eventually I'll be interested in non-Linux targets.

[memoryruins]

Status of patches as of January 26, 2019 🎉

title	status
Refactoring in preparation for immediate lowering	merged
Immediate lowering	merged
RV64I	merged
New target hook to improve codegen	merged
RV64M	merged
RV64A (minus cmpxchg)	merged
RV64F	needs review
RV64D	accepted

[fintelia]

If I understand correctly this means that rustc could now support a riscv64imac-unknown-none-elftarget?

[xxuejie]

I just did a check, it seems Rust team has just recently updated LLVM with changes till Jan 16. That means all the merged PRs above has landed in Rust's LLVM except RV64A, which was merged on Jan 17.

I think that means we can at least prepare a riscv64imc-unknown-none-elf rustc target now.

[asb]

The RV64A patch was actually updated to include cmpxchg prior to merging (the only reason it didn't have it originally was that RV32 cmpxchg / the new target hooks for late cmpxchg expansion were still under review).

For RV64I, you might want to cherry-pick the simple fix r351806 or the better fix r352169.

[fintelia]

@xxuejie What needs to be done to prepare that target?

[dvc94ch]

Find the 4 patches that enable riscv32. replace 32 with 64 apply them, build rust and test that it works.

…

On Wed, Jan 30, 2019, 21:07 Jonathan Behrens ***@***.*** wrote: @xxuejie <https://github.com/xxuejie> What needs to be done to prepare that target? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#218 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtRr1hbEvVfK7QhNSOoHOWN3J7H1BFSks5vIfuQgaJpZM4W893u> .

[dvc94ch]

Ah sry it looks like you were asking about the Linux target now that atomics are available. In addition to doing the stuff required for a bare metal target there are a bunch of constants in the rust source for syscall numbers and unwinding code. Start by copying the mips constants so that it builds and then keep changing them until it runs. I was stuck because atomics weren't implemented in llvm and I don't have access to hardware.

…

On Wed, Jan 30, 2019, 21:11 David Craven ***@***.*** wrote: Find the 4 patches that enable riscv32. replace 32 with 64 apply them, build rust and test that it works. On Wed, Jan 30, 2019, 21:07 Jonathan Behrens ***@***.*** wrote: > @xxuejie <https://github.com/xxuejie> What needs to be done to prepare > that target? > > — > You are receiving this because you are on a team that was mentioned. > Reply to this email directly, view it on GitHub > <#218 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAtRr1hbEvVfK7QhNSOoHOWN3J7H1BFSks5vIfuQgaJpZM4W893u> > . >

[Disasm]

@fintelia, @dvc94ch I can test riscv64 on Unleashed board.

[fintelia]

I did mean the riscv64imc-unknown-none-elf target. I'm currently interested in targeting qemu emulated "bare metal", so that target (or better yet riscv64imac) working with qemu would be all I need.

[xxuejie]

@fintelia Sorry I forgot to post an update here, I did play with nightly Rust a bit. While adding a rv64imc target is trivial, compiling Rust for this target would just hang there forever when compiling core for this target.

My guess is this has to do with the latest patches @asb mentioned above. I tried cherry-picking the exact commit but more compiling errors occur and I was running out of time trying this.

It's also possible that this is all due to some silly configuration error I made. I see @fintelia already submitted a patch. Cheers!

[Disasm]

As far as I can see, all patches (RV64A + RV64F + RV64D) are merged into LLVM. Thanks @asb!
Now we need to update Rust's fork of LLVM. What can I do to update it?

[asb]

Find the 4 patches that enable riscv32. replace 32 with 64 apply them, build rust and test that it works.
…
On Wed, Jan 30, 2019, 21:07 Jonathan Behrens @.*** wrote: @xxuejie https://github.com/xxuejie What needs to be done to prepare that target? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#218 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtRr1hbEvVfK7QhNSOoHOWN3J7H1BFSks5vIfuQgaJpZM4W893u .

There'll be at least a little more to it for ABI lowering. You want to mark any i32 as signext for instance if passed in a register, even if it was a uint32_t (as specified in the ABI). Do you implement the same ABI lowering logic in rustc as I do in clang?

[dvc94ch]

The abi lowering is parameterized over xlen, so if I implemented it correctly it should just work. I'm still not sure if it is though, since what clang does seems to be much more complicated.

[dvc94ch]

If you'd like to scrutenize it you can find it at https://github.com/rust-lang/rust/blob/master/src/librustc_target/abi/call/riscv.rs, but please keep in mind that I'm not a compiler expert so use small words =P

[Disasm]

Now we have riscv64imac-unknown-none-elf and riscv64gc-unknown-none-elf targets in nightly!
However, with these targets you cannot use addresses that are 0x8000_0000 or higher due to the missing support for relocatable code (I hope, it will be added one day). So, for example, for HiFive Unleashed board you cannot place your code in RAM, but you can still place it in L2 LIM, that starts at 0x0800_0000. For other targets fix your linking scripts accordingly.

[fintelia]

That limitation is rather unfortunate, particularly for targets that have all of their memory above 0x8000_0000. It makes it very hard to write for qemu's virt machine type, or any application that needs to place code at specific virtual addresses. Do you know which component lacks support for relocations? Is it something Rust related or a more fundamental problem with LLVM?

Edit: I wonder if position independent code could help. It isn't really a matter of needing more than 2GB of code+data but rather that the address range has to start at address zero.

[Disasm]

@fintelia,

It makes it very hard to write for qemu's virt machine type, or any application that needs to place code at specific virtual addresses.

What's wrong with qemu? I thought that you can set any base address for RAM in your machine definition. Other workarounds may involve using virtual memory region [-2GB;+2GB) for your application.

Do you know which component lacks support for relocations? Is it something Rust related or a more fundamental problem with LLVM?

It's not a fundamental problem, but you need LLVM support to generate relocatable code with pc-based addressing. Aside from this, Rust uses static relocation model for many if not all embedded targets. I don't know what is the reason, but it can be a problem because you need to set the relocation model to pic in order to enable relocatable code generation in LLVM.

[fintelia]

What's wrong with qemu? I thought that you can set any base address for RAM in your machine definition. Other workarounds may involve using virtual memory region [-2GB;+2GB) for your application.

This is possible in spike, but I'm not sure of any way to configure this in qemu. It is hard coded here and I'm not aware of any way to change that without patching/recompiling qemu. My application is also a bit tricky because I'm trying to build a classic hypervisor. That means that even once early boot is over and I've installed page tables (negating issues with the physical location of DRAM) I still ideally need to pick virtual address for code+data that the guest kernel/userspace isn't using.

It's not a fundamental problem, but you need LLVM support to generate relocatable code with pc-based addressing. Aside from this, Rust uses static relocation model for many if not all embedded targets.

I'd be interested in digging into this more. Long term I think there is a push to make riscv more than just an embedded target and use it for larger systems (this is my interest) which would make the need for using larger virtual addresses and relocatable code more pressing. @asb do you know if LLVM has any targets or timelines for these features?

[Disasm]

@fintelia Yes, you have to recompile qemu with another address, however it's not difficult. If you are trying to implement a hypervisor, you need to execute (at least some of) your code in M mode without virtual addresses. You can write this M-mode part in assembly with relocatable code.

[fintelia]

It seems that LLVM now has support for mcmodel=medium (which is equivalent to GCC's mcmodel=medany). It should now be possible to cherry pick the relevant commits and expose support from Rust.

[Disasm]

@fintelia Wow! Thank you for the information. I'm not sure I'll be able to cherry-pick commits this time, maybe we need some help from the Rust team. I tried to cherry-pick useful commits two weeks ago and failed: I couldn't even figure out "maximum" commit set.

[memoryruins]

riscv64imac-unknown-none-elf and riscv64gc-unknown-none-elf targets are now on stable rustc 1.34.0 :) https://github.com/rust-lang/rust/blob/master/RELEASES.md#version-1340-2019-04-11

[sequencer]

#218 (comment)
May I ask when will mcmodel become ready in rust?
RISC-V is deprecating bbl, using opensbi as a replacement, which won't set paging from bootloader, so if _start don't start from 0x8000_000 will lead to pure rust can't write a bare metal entry.

[Disasm]

@sequencer Unfortunately, I don't have an answer to this question. Someone has to merge corresponding patches from upstream LLVM to Rust LLVM tree. I'm busy at the moment, but I hope I will be able to start dealing with this soon.

[sequencer]

Oh thank you!
A demo kernel by @wangrunji0408 give a solution of sv39 paging in _start written by Rust.

[Disasm]

Good news! This patch arrived 12 days ago: llvm/llvm-project@a524036

Now it's possible to build C programs like this one:

unsigned int a;
void *f() {
    return &a;
}

f:                                      # @f
.LBB0_1:                                # Label of block must be emitted
        auipc   a0, %pcrel_hi(a)
        addi    a0, a0, %pcrel_lo(.LBB0_1)
        ret

UPD: rust-lang/rust#62281

[Disasm]

Closing this because the major addressing problem is resolved.