Enable support for SMP Trampoline

[vinaychandra]

The current entry point is hardcoded to _start (or the entry_point! macro). This provides a single entry point to the code.

When bootstrapping SMP, the new processor starts up at a given physical address in the IPI (Inter-Processor Interrupt) in 16 bit mode. This requires us to start at 16 bit code and enable 64 bit mode again. Please provide a way to define multiple "entry points" which defaults to the current _start but can also be used to bootstrap SMP.

[64]

I’m getting near SMP in my own kernel, so I’ll probably investigate this a bit. It might be difficult to do in a platform agnostic way. For example, how should the page tables be set up?

[ethindp]

I've got PCI going and am just confused on how I should manage some kind of driver registration framework to pass the PCI devices off to. Heh, I don't even have networking, but SMP and APIC would be nice.

…

On 8/14/19, Matt Taylor ***@***.***> wrote: I’m getting near SMP in my own kernel, so I’ll probably investigate this a bit. It might be difficult to do in a platform agnostic way. For example, how should the page tables be set up? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #74 (comment)

-- Signed, Ethin D. Probst

[vinaychandra]

@64 the way I understand it, we don't need to setup new page tables, etc. We can leave other processors potentially in the same place as the bootstrap processor. The pagetables are all setup for it anyway. So, we assume the same and start up the new processor into the function provided by the user. Any other changes will have to be done by the developer.

[64]

You don’t need to set up page tables, but you need to load something into cr3 when switching to long mode. There would need to be a way to specify this value.

[vinaychandra]

Agreed, that's where i said we can use the same assumptions as the bootstrap processor. We can load with the same CR3 but point EIP to the new entry point rather than the old one.

[64]

Booting up the APs is usually done fairly late in the boot process. Some people may have switched to a different p4 table in this time, so I think it would be better to have some way of setting the value of cr3 at runtime rather than using the bootloader’s one. But that shouldn’t be too difficult.

[ethindp]

Is there a way we can make a separate rust crate for SMP and allow the kernel to do all the SMP handling, or is that only doable by the bootloader/BIOS/UEFI? I don't fully understand SMP myself, but having that kind of control seems apt for a kernel.

…

On 8/14/19, Matt Taylor ***@***.***> wrote: Booting up the APs is usually done fairly late in the boot process. Some people may have switched to a different p4 table in this time, so I think it would be better to have some way of setting the value at runtime. But that shouldn’t be too difficult. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #74 (comment)

-- Signed, Ethin D. Probst

[64]

The bootloader doesn’t need to be involved with SMP at all. What could be useful is a crate which can be used for interacting with the LAPIC / I/O APICs, since that’s mostly what SMP is about.

[vinaychandra]

I agree. The crate provides entry point for 16 bit bootstrapping, not just boot loader logic which i wanted to reuse for SMP rather than creating another setup which does the exact same thing. That was my request as well. Just provide a way to start in 16 bit code and end in 64 bit rust which can potentially be used for SMP

[vinaychandra]

@phil-opp , i have been trying to play with this but hit a roadblock. SMP's processors start in 16 bit mode at a page aligned address. I couldn't find a way to change the current bootloader code so that i can add an extra function that will be placed at a page aligned address on startup. I didn't want to go through copying of pages in kernel. It would be nice if the low level kernel loader supports that directly.

Can you make sure such a thing would be easier in your rewrite?

[bjorn3]

Would using .align 512 before the function asm work?

[vinaychandra]

align 4096 should work (we need page aligned) but we should also be guaranteed that the function will be in the first 1MB of ram because that is the limit of real mode. Does bootloader guarantee that?

[bjorn3]

align 4096 should work (we need page aligned)

oops, got confused with the old disk sector size :)

[phil-opp]

@vinaychandra

I think the best way to do implement something like this is to add a smp_trampoline field to the boot information that points to the physical address of the trampoline function. The function itself could be created in assembly in the real mode part of the bootloader along with the normal initialization code. I think this should be relatively easy to add so that we won't have to wait for the rewrite of the lower stages. Would this approach work for you?

[vinaychandra]

@phil-opp it should, as long as the final address of the function is page aligned, in first mb and we know the value in kernel.

The simplest way I think should work is to use statics in Bootloader crate along with some assembly (stripped from stages 1, 2, 3) to communicate with the main kernel. The only limitation as of now is to figure out the location of the function so that the application processors can start.

I can help test out any changes you make.

[phil-opp]

@vinaychandra Ok, sounds good. I don't have time to provide the full implementation right now unfortunately. How about I start with the boilerplate (adding a page aligned function in the first mb and exporting that address in the boot information) and let you (or others) create the real trampoline function?

[vinaychandra]

Yes, that would be perfect. I can help out in creating the real trampoline (my first time but still interested)

[phil-opp]

I created #97 with the required boilerplate. I hope this helps!

On a side note: Why the page alignment is required? I tried to google it, but didn't find anything…

[vinaychandra]

Example from this answer,

A SIPI contains a vector, this is similar in meaning, but absolutely different in practice, to an interrupt vector (a.k.a. interrupt number).
The vector is an 8 bit number, of value V (represented as vv in base 16), that makes the CPU starts executing instructions at the physical address 0vv000h.
We will call 0vv000h the Wake-up address (WA).
The WA is forced at a 4KiB (or page) boundary.

When a processor starts up, you can send it a "startup" address which it will offset with 000h which is page aligned.

[phil-opp]

Thanks!

[Andy-Python-Programmer]

Any updates on this?

[Lucky4Luuk]

Any updates on this? This would be a really nice addition for this crate.

[jasoncouture]

Copy/Paste from: #343 (comment)

I, personally, don't think this is a good idea.

Here's why:

The bootloader already has a lot to do, with limited support and system resources (Especially in the case of bios). There are also different gotchas with various hardware iterations stretching back to the initial MP spec by intel. (IE: It's hard to get right across a variety of hardware). Beyond that, it requires reading and parsing the ACPI tables, and setting up the APIC to do IPI. Could it be done? Yes. But it would be less of a bootloader, and more of a kernel. It's also more information the kernel needs to take into account, how IPI was setup as an example.

Perhaps a better way to go about this would be to provide a no_std library that made it easy to start CPUs instead, and people could drop that into their kernel. Similar things exist for ACPI, Serial, and a few more things.

And one more thought, each architechture has it's own method of doing SMP. ARM's process requires i2c communication to the other CPUs. Much different from INIT/SIPI on x86(_64) see docs here: https://developer.arm.com/documentation/102337/0000/Functional-description/System-Boot/Boot-flow-overview?lang=en
This makes it much harder to port the bootloader to other architectures, such as ARM and RISC-V, as you now need to implement SMP on for that hardware as well.

[jzbor]

I think it would be nice to provide a method to place the trampoline code through the bootloader, while leaving the whole IPI shenanigans to the actual OS. This would keep the implementation for the bootloader relatively simple while allowing the OS to rely on the memory management provided by this crate.

I tried (re-)implementing the SMP-bootcode from another project with a blogOS-like OS and the bootloader crate and could not get around the issue on where to put the bootcode without interfering with the bootloader/OS and without resorting to external tools for linking. I am sure that was a skill issue on my side, but I think it would help greatly if the bootloader would provide some space to put the trampoline code...

[jzbor]

Has anyone succeeded in bringing up SMP with Cargo + bootloader crate?

[jasoncouture]

SMP is outside of the scope of a bootloader. That said, the bootloader currently chews up the lower 1MB, which is the only place the AP trampoline can exist.

I made some modifications a while ago before getting sick that prevented the bootloader from using the lower 1mb of ram, and I was able to pull up the APs after I did that.

You can also use a linker script to place the trampoline, but due to system variations, it's best to do this at runtime.

[jasoncouture]

Has anyone succeeded in bringing up SMP with Cargo + bootloader crate?

Yes, see my previous comment.