Finish LJ_GC64 mode

[MikePall]

What's already done:

• Core infrastructure.
• x64 interpreter. Currently needs to be explicitly enabled with -DLUAJIT_ENABLE_GC64.
• ARM64 interpreter.

What's missing:

• Changes to the compiler data structures. In particular, most IR constants are 32 bit right now.
• Changes to the compiler frontend. E.g. frames in snapshots.
• Changes to the compiler backends. x64 should be first, since the support is mostly there.

Note that LJ_GC64 implies LJ_FR2 (2-slot frame info), which causes its own complications for the compiler.

There are quite a few asserts for both LJ_GC64 and LJ_FR2 all over the code, wherever I knew support was missing. But this is incomplete, so don't believe it'll just work as soon as it builds cleanly.

Related discussion: http://www.freelists.org/post/luajit/Status-of-LJ-GC64-on-x64,1 http://www.freelists.org/post/luajit/Status-of-LJ-GC64-on-x64,3

[Considering the scope and the difficulty of the changes, only the current incomplete LJ_GC64 mode support will make it into 2.1.]

[DemiMarie]

I feel (purely as a logical guess) that a sponsorship is likely to be obtainable, as this issue is a blocker for no fewer than four sponsored projects. Is this accurate?

[xHasKx]

Any news here? I understand the difficulty of this issue, just want to be in touch

[stephenmax-zhang]

Any progress?

[NukeRusich]

I believe a sponsorship sounds like it would be logically available, but those who would sponsor it may be waiting for someone else to sponsor this prerequisite, since they would really be interested in the ports themselves in most cases.

That said, I would certainly welcome a sponsorship offer.

[DemiMarie]

This looks like much of the work might already be done.

Looking at the asserts for !LJ_GC64 or !LJ_FR2:

• There are 7 asserts for !LJ_FR2 in lj_record.c and 2 in lj_ffrecord.c. These can probably be fixed separately and individually, being replaced in the meantime by stubs that fall back to the interpreter. Performance will be bad (because these include basic operations like function call and return), but it might be enough for basic testing, and they don't need to all be fixed at once.
• There are 2 asserts for !LJ_FR2 in lj_snap.c, both referring to storing 64-bit program counters. There is also an assertion for !LJ_GC64 related to storing 64-bit references. This seems to be fixable by using 64-bit SnapEntry values on 64-bit platforms (a trivial patch).
• The LuaJIT IR seems to be already OK for LJ_GC64 — it uses GCRef and MRef, which are 64-bit aware.

I am not very familiar with LuaJIT internals, but this task might be easier than it seems.

[mraleph]

The LuaJIT IR seems to be already OK for LJ_GC64 — it uses GCRef and MRef, which are 64-bit aware.

No, IR is not OK. GC/MRef are used within a union and end up overlapping with wrong parts of it (should only overlap with op12 leaving the second word intact).

Furthermore everything that is currently marked IRT_P32 in the IR would not be of this type anymore.

An explicit untagging is now required to convert TValue into a pointer (previously one could just load lower word and that would yield a 32bit pointer).

The task is definitely not easier than it seems, a lot of details are involved.

[NukeRusich]

Dr. Bo, it says "Finish LJ_GC64 mode", not "LJ_GC64 mode".

Since it's rather important, is there any issue with the FFI with LJ_FR2/LJ_GC64 enabled?

I think I'd be willing to accept a sponsorship, but I honestly have not looked into the source details to know about what needs to be done. Presumably, they mean literally "64-bit garbage collection" and "two-slot frame info", and nothing more.

[obilaniu]

@MikePall I think you will get more attention and contributions if you explain just what is LJ_GC64, and what must be done for it to work.

Many might be interested in helping out, but first would need to understand whether they can help out. Unfortunately it's not at all clear what LJ_GC64/LJ_FR2 are for the layperson. Both of those strings do not occur within the source code as of master's tip, nor in any sort of documentation, nor do they occur anywhere on luajit.org. Searching for "it" on Google only yields hits that say "it" is needed, but amusingly, nothing at all discusses what "it" is.

Ping me back when you'll have LJ_GC64 documentation posted up anywhere.

[corsix]

@obilaniu: Check out the v2.1 branch rather than master, then these strings will occur quite a lot in the source code.

The LJ_FR2 setting relates to call frames on the Lua stack. When LJ_FR2=0, the information about a call frame occupies one register (aka. one TValue, or 64 bits). When LJ_FR2=1, the information about a call frame occupies two registers (aka. two TValues, or 128 bits). For completeness, note that so-called continuation frames require double the normal number of TValues, and that so-called C frames also store some information on the C stack in addition to the TValues on the Lua stack. For normal Lua frames, the information about the frame consists of:

• A marker indicating that the frame is a Lua frame.
• The address of the GCfuncL being called.
• The address of the bytecode instruction at which to continue execution when returning to the frame.

In 32-bit mode, both addresses are 32 bits, the marker is included for free courtesy of pointer alignment, and the whole lot ends up fitting very nicely into 64 bits. In 64-bit mode, both addresses are 64 bits, and so 128 bits are required.

The LJ_GC64 setting relates to the maximum size of the Lua heap. When LJ_GC64=0, the Lua heap is limited to the low 2³¹ bits of address space. When LJ_GC64=1, the Lua heap is instead limited to the low 2⁴⁷ bits of address space. Amongst many other things, this results in major changes to how TValues are laid out, how big various structures are, and how easy it is for JIT-compiled code to encode Lua heap pointers as immediate constants. In particular, note that LJ_GC64=1 requires LJ_FR2=1.

[NukeRusich]

@obilaniu, while I agree on your overall point that these are not very well explained (mostly because it seems that it's just a doubling of a number of variables and constants tied to "#if LJ_FR2 && LJ_GC64" etc., but it's probably going to be more intensive than that once I actually start working on it), it actually seems pretty clear from their names, even though Mike Pall does not seem the type to say "The name speaks for itself." to describe something.

[DemiMarie]

@NukeRusich When do you plan on beginning work? I plan on working on the ARM64 JIT compiler (#26) this summer, and this is a prerequisite.

[DemiMarie]

@NukeRusich Scratch that – I won't be able to work on the ARM64 JIT, but I might be able to work on this, inasmuch as affects the C part of the VM rather than the assembly portion.

[NukeRusich]

@drbo, sorry for taking awhile to get back to you. If there's no sponsorship, I usually don't work on things on weekdays, so that's your best bet.