Merging some lua53 changes back into app/lua

[TerryE]

Scope

This might seem a wasted step but what I am talking about is nothing to so with the Lua 5.1 language core functionality, but rather some eLua-specific changes that haven't been integrated into Lua 5.3 because they are not that effective so there was no argument to do so.

The advantage of doing this are multiple:

• The way that these eLua patches were implemented complicates the Lua runtime, increases runtime and save little if any RAM.
• Harmonising how these features across Lua 5.1 and Lua 5.3 simplifies making our C modules work as a common codebase across Lua5.1 and Lua 5.3
• This give me the chance to test and debug some of these implementation approaches within a working Lua 5.1 runtime which derisks their subsequent implementation in Lua5.3
• This remove the last of the code where we need GCC, MinGW, MSC variant botches to get luac.cross to compile on these platforms.

These changes are within the Lua runtime and transparent to running code, and as I said they make coding C modules easier not harder.

There is a fundamental difference in implementation approach between the eLua way and the core Lua approach.

• In core Lua implementation details are handled at as low a level as possible and hidden from the upper levels. So for example Lua tables all have a Table record header and the only functions which manipulates the internals of this are in ltable.c, through some Table access methods are provided in lobject.h. All Lua structures and access are strongly typed.

• eLua tends to use void * for all of its resources and wrap everything in runtime macros that generate conditional code across the runtime. This make compile time checking pretty much impassible. As an example eLua introduced a ROTable sub-type of Table but the implementation points to null terminated vector of luaR_entry records doesn't share any of the access methods of Table instead a set of macros replace every occurrence of a table method call with logic: "if the table address is in ROM then call the ROTable version of the method else call the standard Table method."

The changes that I am suggesting are:

• Replacing the ROTables internal implementation. The LROT_BEGIN() macro now inserts a ROTable header record which fits in the luaR-entry row size. It is a variant of Table that contains all of the fields accessed by the lobject.h macros. This means that we can back out all of the eLua changes except in ltable.c which contains the lrotable.c method variant functions and hooks into them where necessary. In practice this generates simpler, smaller, faster code.
• Removing the RO string subtype. In standard Lua when a TString is created the string itself is appended to the (16 byte long) TString record, so a (16+l+1) byte record is malloced. In eLua if the string is marked as a readonly address then the 4 bytes following the record are a pointer to the address in ROM.
  This sounds like a brilliant saving, but in practice these are only created when a C module pushes a new literal string that hasn't already been used as a string in the calling Lua, so based on my idebug instrumentation a running app might have perhaps a half dozen of these saving a few dozen bytes of RAM -- except that the the LFS dummy_strings module moves most of these into true ROM TStrings anyway.
  With this patch, every inline reference to the content of a TSring is now replaced by conditional code which hast to test a flag bit in the TString and either uses the following field as the string or a pointer to the string. More complicated, larger, slower code -- dump it.
• Some eLua API tidy up. See Impacts of migrating to Lua 5.3 #2808 (comment) for this discussion.

Deployment

At the moment I have a single working branch for this and, the module clean up and the Lua53 development. But that because I am stacked up waiting for @marcelstoer, @nwf, @HHHartmann to shepherd through the next master drop, so I can rebase and commit #2836

I will then split out the lua5.1 + modules changes as the next PR. This is a mega PR so I will try to break into sensible commit chunks. Integrating this should give a lua53-ready firmware.

We should also plan on a quick follow-up bugfix PR. Hopefully the first (very much alpha evaluation version of lua53 should follow 2-3 weeks later.

@jmattsson et al. review comments welcome.

[TerryE]

@jmattsson FYI, this slimline aligned version of Lua51 is now working. Will go back to working on Lua53 tomorrow.

[TerryE]

One example of why the new ROtables implementation is important is that the VM which often check on meta fields for example does the metafield exist? Before this change this involved a full table scan of the table for a __metatable entry, then a full table scan of the metatable for an __index entry and 99% of the time these failed. Now the VM just checks h->metatable && getflasgs(h->metatable)&1 which involves 2 word loads from flash to do the same check. This means in practice that the only accesses from a ROTable are when the keys exist and the lookaside cache handles these.

[TerryE]

Now implemented in PR #2912.