Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable loading dynamic libraries from the LUAZIP virtual file system (with a fallback for ffi.load for regular apps) #488

Open
10 of 11 tasks
rdw-software opened this issue Feb 9, 2024 · 1 comment · May be fixed by #562

Comments

@rdw-software
Copy link
Member

rdw-software commented Feb 9, 2024

Complimentary to #482, it should also be possible to load dynamic libraries (using the FFI) from the virtual file system. In-memory extraction and loading doesn't work since DLLs can only be loaded from disk. So a self-extraction mechanism is needed.

Goals:

  • Applications can elect to use vfs.loadlibrary (or similar) if they want to support LUAZIP deployment
  • If a DLL/SO needs to be loaded, it is extracted to a temporary directory (using uv.os_tmpdir/fs_mktmp) first
  • Error messages should be clear in case of DLL load errors or if the file isn't found
  • The original ffi.load function shouldn't be overridden or modified in any way, similar to the require policy

Roadmap:

  • Implement vfs.dlname to enable resolving library names to platform-dependent paths (similar to ffi.load)
  • Implement vfs.dlopen to enable loading dynamic libraries from the VFS
  • Update the hello-world-app test to exercise the new functionality
    • Create a hello.dll (or libhello.so on macOS/Linux) library that's compiled into the hello-world-app archive
    • Modify the entry point (main.lua) so that it tries to load the library via vfs.loadlibrary
    • Run the main.lua entry point as part of the test so that it falls back to ffi.load when using vfs.loadlibrary
  • Update the relevant parts of the documentation that still mentions dynamic libraries cannot be loaded from the VFS
@rdw-software
Copy link
Member Author

rdw-software commented Sep 4, 2024

As for fallbacks: I don't think it makes sense to "replace" ffi.load with vfs.dlopen unless you actually want to deploy the app.

So the most ergonomic way of using it would be to turn something like this

-- FFI-based code that only works in regular LuaJIT scripts
local lib = ffi.load("mylib") -- Call to load may throw!

into an explicit opt-in for loading from the VFS:

-- FFI-based code that tries to unpack/load from VFS if built as a standalone app (no-op otherwise)
local lib = vfs.dlopen("mylib") or ffi.load("mylib") -- Call to load may throw!

That is to say, if a VFS was found (vfs.decode is called before running the app), this should "just work".


Notes:

  • Even though ffi.load is expected to throw, vfs.dlopen should always fail (return nil and an error message)
  • This is so the above pattern works and also because it's expected to fail in various scenarios
  • On the other hand, ffi.load does always fail loudly because failure to load a dynamic library is usually fatal
  • This pattern is preserved when chaining both calls, as the first might fail silently but ffi.load will still throw
  • So in case of internal load errors, the behavior is the same, but VFS/build errors must be detected

I'm not entirely sure if this divergence in behavior is desirable in practice, so let's just start with this and see how it turns out.


Edit: One issue with throwing inside vfs.dlopen (via ffi.load, which is currently pcalled) - temp dirs must be cleaned up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Oooh, shiny! (Highly experimental)
Status: It's finally happening! (WIP)
Development

Successfully merging a pull request may close this issue.

1 participant