Using iterators with state in sol

[oxidase]

Hi, in OSRM project a change from luabind to sol2 revealed a regression Project-OSRM/osrm-backend#3452

My lua code :

print("way id : ", way:id());
for node in way:get_nodes() do
print(" node id : ", node:id())
end

The error exception :

way id : 5203906
TBB Warning: Exact exception propagation is requested by application but the linked library is built without support for it
terminate called after throwing an instance of 'tbb::captured_exception'
what(): lua: error: profile.lua:677: attempt to call a userdata value
Abandon (core dumped)

The current PR changes a usertype to a table Project-OSRM/osrm-backend#3468
but it is just a fall back to stateless iterators and requires use of ipairs

for _, node in ipairs(way:get_nodes()) do
  print("  node id  : ", node:id())
end

Is there a way in sol2 to support old luabind-like behavior?

[ThePhD]

Iterators and pair/ipairs calls are entirely funky based on which version of Lua you use. Seeing as OSRM looks to support Lua 5.1, 5.2, and 5.3, you're in for a bit of a bad time trying to write code that's agnostic to all three when it comes to iterating over something.

For a brief overview of why, see this issue and the information about container serialization.

I do not have a good solution to the problem. Lua 5.2 and 5.3 added mechanisms directly to solve this problem and have stateful iterators. Lua 5.1 does not respect any of the metamethods for doing so and typical iteration functions do not respect the index or newindex metamethods either.

Your recourse would be to tell people to iterate in a specific way by letting them manually get 2 iterators and then walking those manually with a for or while loop, or some other third scheme. There's lots of ways to iterate, but Lua 5.1 is going to tie your hands at every opportunity (the language just isn't there).

I pray one day that LuaJIT becomes entirely Lua 5.3 friendly in its next big update, and we can leave the older design flaws of Lua 5.1 behind permanently as people have no reason to go backwards to 5.1, but... well.

Wishful thinking and all.

Let me know if there's any way to help.

[ThePhD]

If I ever do find a good way to do iteration with Lua 5.1, I will almost 100% implement it in sol2 so people don't have problems with it anymore...

[TheMarex]

@ThePhD to be clear: We could get this behavior if we drop Lua 5.1. support?

I don't have any real objections to that from the OSRM side.

[ThePhD]

@TheMarex Try to hook up the __pairs and __ipairs methods (or define proper iterators with begin and end as well as value_type and other container typedefs on the object returned from obj:nodes()) and then hook it up to Lua using sol2. You should see the code work in Lua 5.2 and Lua 5.3, but break in 5.1.

Again, you can simply expose a obj:nodes_table() method or something to offer an alternative, or you can drop Lua 5.1 and just keep the regular for loop and pairs/ipairs syntax.

Remember that dropping Lua 5.1 generally means losing LuaJIT support. I don't know who your userbase is, but I would imagine that might not be the best?

[TheMarex]

@ThePhD thank you for the insights. 👍

In our use-case we have already discouraged LuaJIT for a while now and in our production deployments we run on Lua 5.2. LuaJIT gave us a 15-30% speedup, but ultimately wasn't worth the hassle since the scripting enabled part of our processing is only a small fraction of the overall computation time.

[ThePhD]

Sounds like a plan. Well, good luck!