Support for RISC-V #1702
Comments
|
@rushi47 awesome idea to port CRIU in RISC-V. Please, tell me do you have RISC-V hardware, or do you want to port and test CRIU on the virtualized RISC-V environment? First of all, I think you need to play with Feel free to reach us. Regards, |
|
The 2nd step will be |
|
@mihalicyn Hello Alex :) To answer your question - Yes as of now i will test it on virtualised environment. But I do have provision to get FPGA with RISCV so we can test there as well once, it runs in virtual environment. & Thanks for this tip let me try to play with examples in compel :). Yesterday night I was looking at the flow but I think this makes more sense to make test examples run first. I will update you, once I try to run this examples with findings I get. |
|
And would love to know if any other things I should be trying or reading organically with understanding to get this thing running on RISCV as soon as we can. |
|
you are welcome :) offtopic: Ah, I'm also thought about the RISC-V motherboard but it's sooo expensive (and mostly just not in stock) right now... :( |
|
I would like to join the effort too, I have a sifive unmatched board for testing if it may help. I also quite newbie to system programming, but i will try to do my best. Opened a MR for traceability and reviewing : #1713 |
|
@nirousseau Hey thanks for joining the effort. |
|
@mihalicyn Hello Alex, So i was trying to run compel as you suggested before and did some changes. While compiling am facing the below issue (Have attached detailed logs below) After reading the log with my knowledge, one strange thing am noticing is, when function : fini_sigreturn is called as per the logs : It's supposed to go to ARCH specific : ARCH_RT_SIGRETURN definition, for this case from my assumption it should call this macro : https://github.com/rushi47/criu/blob/mod_criu/compel/arch/riscv64/src/lib/include/uapi/asm/sigframe.h#L36-L43 But from log it looks to call : ARCH_RT_SIGRETURN_NATIVE & ARCH_RT_SIGRETURN_COMPAT . After searching, i found that it only exists in x86 arch (https://github.com/rushi47/criu/blob/mod_criu/compel/arch/x86/src/lib/include/uapi/asm/sigframe.h#L180-L212). So am not sure why, it is executing code related to x86 arch. I am not sure if am doing anything wrong while compiling, your help will be much appreciated. Below are the logs : |
@rushi47 you need to install |
|
@rst0git Thank you for writing it out :) , i thought its unrelated but i did install it as you suggested. I am trying to figure out, why it's going on the same unrelated method. |
|
@rushi47 x86 and RISC-V use different assembly registers. |
|
@rst0git yes, that's correct. My doubt is I have copied code of aarch64 as riscv - in compel/arch & I am try to compile it for RISCV, from my understanding ideally it should go to riscv and should break there, am wondering why it's going to ARCH_RT_SIGRETURN_ COMPAT & NATIVE as this definition only exists in x86 arch like i pointed before, when am compiling for RISCV. |
|
I think am getting there, it hit the right spot now. |
|
[Update] Reached till parasite code, trying to figure it out - |
|
@mihalicyn hey alex will you be able to help me, with assembly for parasite code injection. I am bit confused with, how it's done for riscv |
|
A friendly reminder that this issue had no activity for 30 days. |
|
I think am able to compile compel part it looks to be successful [i think at least syntactically]. |
|
@adrianreber @avagin will you be able to guys take a look ? |
|
Hi @rushi47 feel free to reach me on the Gitter [https://gitter.im/save-restore/CRIU]
if it's actual - ping me. I will try to help you. |
|
@mihalicyn I am running it on RISC-V machine like native code. Breaking here now, when am trying to run test: Ahh ok let me join jitter, looks like on jitter people are more responsive |
yep, this change to the linker script looks correct.
Yep, you need to support relocations for RISC-V. First of all, I suggest listing all relocation types that you've met here.
|
|
We've private text discussion with @rushi47 about relocation types and how to handle them. I think at first we can go with a "naive way": This code is not tested because I've no RISC-V machine yet. So @rushi47, please try to play with that. It's a "naive" translation of the kernel implementation for these relocation types. |
|
@mihalicyn thanks ton for this. As per private conversation I tested this, so compel is compiling again fine but its still throwing error I think need to add more relocation types working on it On other hand, this is docker image can be used as emulator to test RISCV: |
|
Hi @rushi47! how things are going with RISC-V? I have some plan to play with that soon, as far as I'll get access to the real RISC-V device to test on it. Please let me know if you have a serious plans to work on it. |
Hey,
I had pause project for some time. I am quite keen about it & more than happy to work on it. So if you are starting working on it, I am more than happy to resume again. If someone can mentor me actively, I am more than happy to dedicatedly work for some hours on it daily. |
Disable criu support until upstream has support (checkpoint-restore/criu#1702)
Disable criu support until upstream has support (checkpoint-restore/criu#1702)
|
Hello everyone, I have been working on RISC-V recently, and was wondering if we could really advance on this feature. I would be happy to provide any RISC-V based simulations and testing. |
@mihalicyn I have followed this suggestion and added more relocation types I encountered (doing a If other people are interested in this, @mihalicyn @rushi47 and I also have a group chat on Gitter to discuss the details. It's quite active :) |
Hello @thesaadmemon We're actively working on this feature now. What simulation and testing environment are you referring to? Can you give us some more details? Thank you very much! :) |
|
@felicitia Hello Yixue,I have two VisionFive2(4c8g) board and am currently building the riscv64 version of Proxmox VE. |
That's great @jiangcuo ! I actually found a way to test CRIU on riscv64 QEMU and wrote the instruction since simulation is a lot easier to get started. CRIU needs to be cross compiled for |
|
@felicitia Hi Yixue, I am quite interested in using CRIU on RISC-V and have attempted to follow your instructions to perform a sample compel test on my own riscv64 QEMU. However, I encountered an issue during the cross-compilation process while running ...
DEP soccr/soccr.d
CC soccr/soccr.o
AR soccr/libsoccr.a
make[1]: 'soccr/libsoccr.a' is up to date.
criu/Makefile.packages:36: Can not find some of the required libraries
criu/Makefile.packages:37: Make sure the following packages are installed
criu/Makefile.packages:38: RPM based distros: protobuf protobuf-c protobuf-c-devel protobuf-compiler protobuf-devel protobuf-python libnl3-devel libcap-devel python3-future
criu/Makefile.packages:39: DEB based distros: libprotobuf-dev libprotobuf-c-dev protobuf-c-compiler protobuf-compiler python3-protobuf python3-future libnl-3-dev libcap-dev
criu/Makefile.packages:40: To run tests the following packages are needed
criu/Makefile.packages:41: RPM based distros: libaio-devel python3-PyYAML
criu/Makefile.packages:42: DEB based distros: python3-yaml libaio-dev libaio-dev
criu/Makefile.packages:43: *** Compilation aborted. Stop.
make[1]: *** [criu/Makefile.packages:49: check-packages] Error 2
make: *** [Makefile:263: criu] Error 2
Prior to this, I had successfully executed cross-compile/build_required_deps.sh, so all necessary dependencies should be located at /scratch/cross-compile-riscv64-artifacts/riscv64_pb_install. I am unsure of what could be causing this issue. Would you happen to know of any potential causes or perhaps a workaround? I noticed in your instructions that you have successfully built all artifacts. Could that possibly be a solution to this problem? Below is my #!/bin/bash
TOOLCHAIN_ROOT="/opt/riscv64"
# the path that contains cross compiling toolchain binaries, e.g., riscv64-unknown-linux-gnu-gcc, riscv64-unknown-linux-gnu-ld
TOOLCHAIN_PATH="$TOOLCHAIN_ROOT/bin"
TARGET_ARCH="riscv64"
# the root directory that contains CRIU's source code
CRIU_ROOT_DIR="/scratch/yixue-criu"
# the root directory that will contain the cross compiled artifacts (initially empty)
# e.g., the RISC-V binaries of protobuf (CRIU's required package)
BUILD_ROOT_DIR="/scratch/cross-compile-riscv64-artifacts"
mkdir -p $BUILD_ROOT_DIR
# no need to change it, unless you changed the build scripts for CRIU's dependencies (e.g., build_protobuf.sh)
# the path should be consistent with the prefix specified in the build scripts (e.g., build_protobuf.sh)
INCLUDE_DIR_CC="$BUILD_ROOT_DIR/riscv64_pb_install/include"
LIB_DIR_CC="$BUILD_ROOT_DIR/riscv64_pb_install/lib"
TOOLCHAIN_INCLUDE_DIR="$TOOLCHAIN_ROOT/sysroot/usr/include"
# the directory that contains the toolchain libraries, e.g., libpthread.so.0
TOOLCHAIN_LIB_DIR="$TOOLCHAIN_ROOT/sysroot/lib"
export PATH=$TOOLCHAIN_PATH:$PATH
I greatly appreciate your efforts and look forward to your response. Thank you :) |
|
Further investigation into errors when suffering from insomnia😭 By removing ...
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-rpath'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-rpath'
riscv64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-rpath'
...This CFLAGS=$(pkg-config --cflags libprotobuf-c)
CFLAGS+=" -I$INCLUDE_DIR_CC -L$LIB_DIR_CC"
LDFLAGS=$(pkg-config --libs libprotobuf-c)
LDFLAGS+=" -rpath $TOOLCHAIN_LIB_DIR"I tried a syntax fix suggested by chatgpt, updating the last line to /rivos/riscv-gnu-toolchain/lib/gcc/riscv64-unknown-linux-gnu/12.1.0/../../../../riscv64-unknown-linux-gnu/bin/ld: cannot find -lselinux: No such file or directory
/rivos/riscv-gnu-toolchain/lib/gcc/riscv64-unknown-linux-gnu/12.1.0/../../../../riscv64-unknown-linux-gnu/bin/ld: cannot find -lgnutls: No such file or directory
collect2: error: ld returned 1 exit status
criu/Makefile.packages:36: Can not find some of the required libraries
...Anyway at least I'm able to cross compile infect and rsys tests and run them on my riscv64 VM. I'm learning how compel and the arch-specified code works, hopefully I can make some contributions to this feature. Thank you so much :) |
|
@ancientmodern Ah sorry to hear about your insomnia I hope it's not because of CRIU! 😂 What you saw was exactly what's expected actually so great job!! 👏 This is very great news as you're the first one following my instructions and it looks like everything was correct! Now some explanations of the error you saw. However, you'll have other compiling issues after fixing this as well and this is a known issue. You probably already tasted how complicated the build process is when you did further investigation right? :) It wasn't easy to cross-compile CRIU that far and I'm only focusing on "compel" module as @mihalicyn suggested earlier in this issue (one of the earliest CRIU developers!). So don't worry about it for now as "compel" is already cross-compiled correctly and we're only working on "compel" for now. The short-term goal is to pass all 4 test cases under So long story short, you have everything correctly set up already! If you're interested in using CRIU for RISC-V, it's still under development and you can just watch this issue. If you're interested in contributing to the project, you can find me on Gitter to sync up the current status (just so you're not chasing the issues that I already faced :)). |
|
@felicitia Thank you for your detailed explanation and assistance. I'm glad I was on the right track! One interesting thing I found is that, despite build_criu.sh consistently resulting in some sort of error, it needs to be run once before executing build_compel_test.sh in order to build the required libraries. 😂 |
Yup, build_criu.sh cross-compiles the compel library and needs to be run first. build_compel_test.sh cross compiles compel's test cases (rsys, infect, for now). If you modify the tests (e.g., add more debugging info), you need to run |
|
Hello CRIU community, happy to report that However, when testing using New updates on |
Update: int compel_get_task_regs(pid_t pid, user_regs_struct_t *regs, user_fpregs_struct_t *ext_regs, save_regs_t save,
void *arg, __maybe_unused unsigned long flags)
{
// ......
if (regs->a7) { /* Not sure if this is adequate for syscall identification */
/* Restart the system call */
switch (regs->a0) {
case -ERESTARTNOHAND:
case -ERESTARTSYS:
case -ERESTARTNOINTR:
regs->a0 = regs->orig_a0; /* The user_regs_struct structure needs to be altered */
regs->pc -= 0x4;
break;
case -ERESTART_RESTARTBLOCK:
regs->a0 = regs->orig_a0;
regs->a7 = __NR_restart_syscall;
regs->pc -= 0x4;
break;
}
}
ret = save(arg, regs, fpsimd);
return ret;
}We only require the kernel (and riscv-gnu-toolchain) to add Original reply: // arch/riscv/kernel/signal.c
void arch_do_signal_or_restart(struct pt_regs *regs)
{
struct ksignal ksig;
if (get_signal(&ksig)) {
/* Actually deliver the signal */
handle_signal(&ksig, regs);
return;
}
/* Did we come from a system call? */
if (regs->cause == EXC_SYSCALL) {
/* Avoid additional syscall restarting via ret_from_exception */
regs->cause = -1UL;
/* Restart the system call - no handlers present */
switch (regs->a0) {
case -ERESTARTNOHAND:
case -ERESTARTSYS:
case -ERESTARTNOINTR:
regs->a0 = regs->orig_a0;
regs->epc -= 0x4;
break;
case -ERESTART_RESTARTBLOCK:
regs->a0 = regs->orig_a0;
regs->a7 = __NR_restart_syscall;
regs->epc -= 0x4;
break;
}
}
/*
* If there is no signal to deliver, we just put the saved
* sigmask back.
*/
restore_saved_sigmask();
}Here is my log related to this issue, demonstrating that Currently, the RISC-V ptrace capability (even in linux-next where vector support has been added), doesn't allow setting CSRs via ptrace in userspace. We could potentially submit a short patch to the Linux kernel for this, but it's uncertain whether providing |
Add orig_a0 to RISC-V ptrace's user register interface. This allows user-space programs to access and manipulate orig_a0 through ptrace regset APIs. In order to maintain the prefix relationship, the pt_regs struct has been reordered accordingly. Some other archs have similar practice, e.g. x86 exposes orig_eax to user space. The need for this change arose during the porting of a user-space checkpoint/restore tool to the RISC-V. The tool required access to orig_a0 to restart interrupted syscalls, as a0 gets overwritten by the syscall return value after execution. Detailed explanation and the associated issue can be found at: github.com/checkpoint-restore/criu/issues/1702#issuecomment-1605814529 Signed-off-by: Haorong Lu <haorong@rivosinc.com>
Hello Team,
I am fairly new to CRIU, and am quite interested in porting it to RISCV by my own but I feel kind of lost as there are lot of touch points. And am unable to wrap my head around the code.
So would love to know if are there are any plans to support for RISC-V, would love to pitch in.
Also would appreciate any help, if someone can point me to any documentation.
I use this link : https://criu.org/Category:Under_the_hood but things are sorted by Alphabetical order and not the way they are used in code or would say I am unable to find any documentation on how CRIU works. So it's difficult to read the components and try to know where it's used in code.
Would love to contribute to this project, would very much appreciate any help.
Looking forward to it.
Thank you in advance.
Here am trying few things but could very wrong and naive, apologies am very new to all systems programming as well :
criu-dev...rushi47:mod_criu
The text was updated successfully, but these errors were encountered: