Disabling hardware atomic instructions / providing custom implementation?

[teknoman117]

This is more of a question than an issue.

I'm wondering if anyone knows of a way to provide an alternative implementation of the atomic operations used within the alloc crate.

For my own amusement, I've been following along with this series and porting it to a really old single board computer that has an 80386EX in it. Figured, well, it's the first x86 cpu with an MMU, so what might go wrong? Turns out, quite a bit, as the 386 did not support atomic instructions, cmpxchg didn't show up until the 486. The compiler is emitting "lock cmpxchg %dl, (%ecx)", which crashes the system.

(this particular single board computer is super neat because it has a debugger built straight into the bios, although it doesn't understand 32-bit instructions very well)

using heap at 0x15630 (size: 0x415D0)

Embedded BIOS Debugger Invalid Opcode Trap
EAX = 00000000  CS:EIP = 0563:00001F19  EFL = 00000246  pl ZR .. na .. PE .. nc
EBX = 000064F4  SS:ESP = 0563:0000FF80  EBP = 0000FFEA  .. nt IOPL0 nv up EI ..
ECX = 00006500  DS:ESI = 0563:00006500  FS  = 0000      .. .. id vp vi al vm rf
EDX = 00000101  ES:EDI = 0563:000E0345  GS  = 0000
0563:00001F19   lock

EB40DBG:

rustc --print cfg --target 80386.json (a modified version of yours, but 32 bit) shows supports for atomic instructions (target_has_atomic="32", etc.), but I still haven't found a way to disable them. I realize doing so would probably break the alloc crate, which is why I'm wondering if there is a way to provide a custom implementation of atomics and sync, following the implementation that old operating systems used (disabling interrupts because there wasn't SMP support on those operating systems and doing so would prevent things such as your DMA controller from poking you in the middle of your operation).

This could also be useful for more embedded CPUs which may not have them.

(for fun, this is what I'm trying to run it on: https://wiki.embeddedarm.com/wiki/TS-3100)

[teknoman117]

Found a way to disable the atomics, needed to add this in the target json file (no quotes around the number is apparently important).

"max-atomic-width": 0

Now need to figure out how to provide my own version of those core::sync::atomic structs, as spin no longer compiles, pretty much expected that through. Kinda curious if I could get it to believe it has atomic load/store but not compare-and-swap (via a "xchg" instruction, which asserts exclusivity over the bus). (rust-lang/rust#51953)

[phil-opp]

I don't think that you have to disable atomics entirely. Instead, you could tell LLVM to compile for a i386 instead of the default generic x86 CPU, which uses more recent CPU features including cmpxchg. Try explicitly setting the cpu key to i386 in your target specification file (I'm not sure if it needs to be literally "i386" or a different string).

[teknoman117]

Here's my current target specification file. If I take out "max-atomic-width", it generates the compare and swap instructions.

{
    "llvm-target": "i386-unknown-none-code16",
    "data-layout": "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128",
    "linker-flavor": "ld.lld",
    "linker": "rust-lld",
    "pre-link-args": {
      "ld.lld": [
        "--script=com.ld"
      ]
    },
    "target-endian": "little",
    "target-pointer-width": "32",
    "target-c-int-width": "32",
    "max-atomic-width": 0,
    "arch": "x86",
    "cpu": "i386",
    "os": "none",
    "features": "-mmx,-sse,+soft-float",
    "disable-redzone": true,
    "panic": "abort",
    "executables": true,
    "exe-suffix": ".elf",
    "relocation_model": "static"
}

(it's set to code16 as I don't have my switch to protected mode quite setup yet, need to implement writing to the serial port for debugging as I don't have video output on this board).

[phil-opp]

Have you tried to additionally pass -C target-cpu=i386 to rustc?

[phil-opp]

You could also try adding -cx8 to the features in your target JSON. This should explicitly disable the CMPXCHG8B instruction.

Source: https://github.com/llvm/llvm-project/blob/f688570d5c5493e969d5fca599d6eb8a796e27ca/llvm/lib/Target/X86/X86.td#L42-L43

[teknoman117]

Doesn't look like they have a flag to disable it (that's the 8 byte version, the one that's getting generated for me is the original "CMPXCHG" instruction, the 4 byte / 32 bit one).

[phil-opp]

Ah ok. Still, LLVM should not create a cmpxchg instruction if you specify a CPU that doesn't support it. They even mention that the 80386 does not support cmpxchg in https://releases.llvm.org/9.0.0/docs/Atomics.html#atomics-and-codegen :

However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there are atomic load and store instructions, but no cmpxchg or LL/SC.

[phil-opp]

Have you tried disabling the atomic-cas feature flag in your target JSON?

Edit: Seems like the PR you linked in #707 (comment) uses exactly this flag instead of setting max-atomic-width to 0.

[teknoman117]

Ah, I spelled it wrong. had "atomic_cas", needed "atomic-cas". Thanks for that.

That does in fact disable compare and swap. Looks like the spin crate explicitly uses compare and swap, so I have some work to do.

nlewis@enceladus [02:10:23 AM] [~/Sources/rusty-dos] [master *]
-> % rustc --print cfg --target dos.json
debug_assertions
target_arch="x86"
target_endian="little"
target_env=""
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="none"
target_pointer_width="32"
target_vendor="unknown"

error[E0599]: no method named `compare_and_swap` found for type `core::sync::atomic::AtomicBool` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/mutex.rs:131:25
    |
131 |         while self.lock.compare_and_swap(false, true, Ordering::Acquire) != false
    |                         ^^^^^^^^^^^^^^^^ method not found in `core::sync::atomic::AtomicBool`

error[E0599]: no method named `compare_and_swap` found for type `core::sync::atomic::AtomicBool` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/mutex.rs:181:22
    |
181 |         if self.lock.compare_and_swap(false, true, Ordering::Acquire) == false
    |                      ^^^^^^^^^^^^^^^^ method not found in `core::sync::atomic::AtomicBool`

error[E0599]: no method named `fetch_add` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:199:31
    |
199 |         let value = self.lock.fetch_add(READER, Ordering::Acquire);
    |                               ^^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_sub` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:205:23
    |
205 |             self.lock.fetch_sub(READER, Ordering::Release);
    |                       ^^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_sub` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:224:19
    |
224 |         self.lock.fetch_sub(READER, Ordering::Release);
    |                   ^^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_and` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:236:19
    |
236 |         self.lock.fetch_and(!(WRITER | UPGRADED), Ordering::Release);
    |                   ^^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_or` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:328:22
    |
328 |         if self.lock.fetch_or(UPGRADED, Ordering::Acquire) & (WRITER | UPGRADED) == 0 {
    |                      ^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_add` found for type `&'rwlock core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:440:19
    |
440 |         self.lock.fetch_add(READER, Ordering::Acquire);
    |                   ^^^^^^^^^ method not found in `&'rwlock core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_add` found for type `&'rwlock core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:467:19
    |
467 |         self.lock.fetch_add(READER, Ordering::Acquire);
    |                   ^^^^^^^^^ method not found in `&'rwlock core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_sub` found for type `&'rwlock core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:511:19
    |
511 |         self.lock.fetch_sub(READER, Ordering::Release);
    |                   ^^^^^^^^^ method not found in `&'rwlock core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_sub` found for type `&'rwlock core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:521:19
    |
521 |         self.lock.fetch_sub(UPGRADED, Ordering::AcqRel);
    |                   ^^^^^^^^^ method not found in `&'rwlock core::sync::atomic::AtomicUsize`

error[E0599]: no method named `fetch_and` found for type `&'rwlock core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:531:19
    |
531 |         self.lock.fetch_and(!(WRITER | UPGRADED), Ordering::Release);
    |                   ^^^^^^^^^ method not found in `&'rwlock core::sync::atomic::AtomicUsize`

error[E0599]: no method named `compare_exchange` found for type `&core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:545:16
    |
545 |         atomic.compare_exchange(current, new, success, failure)
    |                ^^^^^^^^^^^^^^^^ method not found in `&core::sync::atomic::AtomicUsize`

error[E0599]: no method named `compare_exchange_weak` found for type `&core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/rw_lock.rs:547:16
    |
547 |         atomic.compare_exchange_weak(current, new, success, failure)
    |                ^^^^^^^^^^^^^^^^^^^^^ method not found in `&core::sync::atomic::AtomicUsize`

error[E0599]: no method named `compare_and_swap` found for type `core::sync::atomic::AtomicUsize` in the current scope
   --> /home/nlewis/.cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/spin-0.5.2/src/once.rs:104:33
    |
104 |             status = self.state.compare_and_swap(INCOMPLETE,
    |                                 ^^^^^^^^^^^^^^^^ method not found in `core::sync::atomic::AtomicUsize`

error: aborting due to 15 previous errors

Oddly, the every x86 processor does sort of support the fetch_{sub,and,add} operations, but it doesn't return the value, but it will add/sub/and to the memory location atomically.

[phil-opp]

According to the documentation linked in #707 (comment), LLVM should be able to emulate compare and swap instructions using mutexes. The question is whether the Rust compiler supports this.

[phil-opp]

Regarding the spin issues: I think it's easier to write a custom version of spin without relying on hardware support instead of reimplementing the full sync::atomic module.

Wikipedia lists a number of software solutions to the mutual exclusion problem, which all work without special hardware instructions. There is also an example spinlock implementation in x86 assembly that uses only xchg and should work on the 80386.

[teknoman117]

Thanks!