Possibility of using pre-compiled shared libraries like sharp

[summera]

Thank you for this amazing library! Really appreciate your work on this. I'd love to get your thoughts on the best way to pin and update libvips and its dependencies. This can be a complicated and cumbersome task in a docker environment and many base images have very old versions of libvips. Currently, we're compiling libvips in our Dockerfile using a multi-stage build and copying files to the final stage. Compilation and copying files can take quite a bit of time and figuring out what files to copy over is nontrivial. Do you have any suggestions or examples for doing this? I'd love to find a better way.

Also, I'd be curious what your thoughts are on providing something similar to what sharp does with packaging scripts so that we could download pre-compiled shared libraries to be used with this gem. Is that possible?

I attempted to use sharp's shared libraries with this gem, but could not get it to work. In addition to easing the burden of installing libvips, this would also make it easier to use patched versions of dependencies, such as libwebp, which (as I'm sure you're already aware) has come up recently and been resolved for users of sharp.

Thanks in advance for your time!

[jcupitt]

Hi @summera,

We've talked about this in the past, but not got around to doing it. As you say, sharp has all these things precompiled, so pyvips and ruby-vips should just need a fairly small addition to automatically download a matching binary for you.

Other solutions depend on the platform. Heroku has buildpacks to automate deployment. Debian-derived systems have PPAs. I'm vague about redhat, though Remi has a nice RPM repository which includes a well-maintained libvips. AWS has something too, though I forget the details.

[fschwahn]

I attempted to use sharp's shared libraries with this gem, but could not get it to work

I tried the same thing, namely downloading the relevant package from https://github.com/lovell/sharp-libvips/releases/tag/v8.15.1 and moving lib/libvips-cpp.42.dylib to /usr/local/lib/libvips.42.dylib.

When I do this I get an error because glib is missing, so I do brew install glib. When I do that I can require "vips", but as soon as I try any operation ruby segfaults, such as:

(process:28060): GLib-CRITICAL **: 10:38:44.610: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.610: g_param_spec_pool_lookup: assertion 'pool != NULL' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.610: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'

(process:28060): GLib-CRITICAL **: 10:38:44.620: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.620: g_param_spec_pool_lookup: assertion 'pool != NULL' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.620: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'

(process:28060): GLib-CRITICAL **: 10:38:44.643: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.643: g_param_spec_pool_lookup: assertion 'pool != NULL' failed

(process:28060): GLib-GObject-CRITICAL **: 10:38:44.643: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'
/Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/operation.rb:225: [BUG] Segmentation fault at 0x0000000000000050
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]

-- Crash Report log information --------------------------------------------
   See Crash Report log file in one of the following locations:             
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0084 p:---- s:0456 e:000455 CFUNC  :vips_cache_operation_build
c:0083 p:0005 s:0451 e:000450 METHOD /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/operation.rb:225
c:0082 p:0239 s:0446 E:000da0 METHOD /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/operation.rb:481
c:0081 p:0072 s:0424 e:000423 METHOD /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/image.rb:283

...

This may be pretty naive, but it would be great to be able to use a precompiled version of vips.

[kleisauke]

https://github.com/lovell/sharp-libvips exclusively distributes the C++ bindings as a single shared library, which is unlikely to be compatible for use within Ruby bindings.

Could you try the https://github.com/kleisauke/libvips-packaging fork instead? I think that would work without any problems, at least I haven't encountered any issues in NetVips (NetVips.Native NuGet package), pyvips (experiments mentioned in PR libvips/pyvips#445) and weserv/images (see e.g. weserv/images#338).

[fschwahn]

@kleisauke Thanks for the recommendation! I downloaded libvips-8.15.1-osx-arm64.tar.gz, did mv libvips-8.15.1-osx-arm64/lib/libvips.42.dylib /usr/local/lib, and required the gem. That works. As soon as I try to use vips it fails however:

irb(main):001> require "vips"
=> true
irb(main):002> image = Vips::Image.new_from_file("/path/to/image.jpg")

(process:43411): GLib-CRITICAL **: 13:29:57.512: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:43411): GLib-GObject-CRITICAL **: 13:29:57.512: g_param_spec_pool_lookup: assertion 'pool != NULL' failed

(process:43411): GLib-GObject-CRITICAL **: 13:29:57.512: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'
/Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/operation.rb:228:in `build': jpegload: parameter filename not set (Vips::Error)
	from /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/operation.rb:481:in `call'
	from /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/ruby-vips-2.2.1/lib/vips/image.rb:283:in `new_from_file'
	from (irb):2:in `<main>'
	from /Users/fabian/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/irb-1.11.2/exe/irb:9:in `<top (required)>'
	from /Users/fabian/.rbenv/versions/3.2.3/bin/irb:25:in `load'
	from /Users/fabian/.rbenv/versions/3.2.3/bin/irb:25:in `<main>'
irb(main):003> exit

(process:43411): GLib-CRITICAL **: 13:29:59.931: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:43411): GLib-GObject-CRITICAL **: 13:29:59.931: cannot unreference class of invalid (unclassed) type '(null)'

(process:43411): GLib-CRITICAL **: 13:29:59.931: g_datalist_id_set_data_full: assertion 'key_id > 0' failed

(process:43411): GLib-GObject-CRITICAL **: 13:29:59.931: cannot unreference class of invalid (unclassed) type '(null)'

Do you have any idea what might be the problem?

[kleisauke]

Ah, it looks like 2 GLib instances are running in the same process. On Linux, we localize the g_param_spec_types symbol to avoid these kinds of collisions, but I'm not sure if we can do the same on macOS.
https://github.com/kleisauke/libvips-packaging/blob/v8.15.1/build/lin.sh#L463-L465

[kleisauke]

but I'm not sure if we can do the same on macOS.

... looks like we can use -Wl,-unexported_symbols_list nowadays on macOS. For example:
unexported_symbols_list.txt:

g_param_spec_types

-Wl,-unexported_symbols_list,unexported_symbols_list.txt

I'll try an experiment during the weekend, but I don't have access to a macOS device (so I will have to do a reverse shell in a macOS runner on GitHub to test this properly 😅).

[fschwahn]

Thanks for looking into this 😃 I'm happy to try out your experiment whenever you are ready.

[kleisauke]

Actually, I think it should work with this (short-circuit) patch:

--- a/lib/vips.rb
+++ b/lib/vips.rb
@@ -27,7 +27,7 @@ def library_name(name, abi_number)
   if FFI::Platform.windows?
     "lib#{name}-#{abi_number}.dll"
   elsif FFI::Platform.mac?
-    "#{name}.#{abi_number}"
+    "vips.42"
   else
     "#{name}.so.#{abi_number}"
   end

i.e. ruby-vips needs something similar to libvips/pyvips@c380493.

[fschwahn]

@kleisauke Yes, that works! Thanks for your help - I've now patched this in my local installation, but would be great if this could be done in the gem.

Maybe it would be possible to specify an ENV variable which can be picked up for library name?

[jcupitt]

Ah nice. I'll have a look, thanks Kleis.

Maybe it would be possible to specify an ENV variable which can be picked up for library name?

It should be possible to do this automatically. It's only needed for semi-statically linked libvipses.

[kleisauke]

I just opened (draft) PR #390 to support this.

[fschwahn]

@jcupitt #390 fixed this problem, so I think this issue can be closed.

could we maybe have a new release with this fix? I copy this fix locally from ruby release to ruby release, and there sure seem to be a lot of releases lately 😅

[jcupitt]

OK, I made 2.2.2. Good luck!

[fschwahn]

@jcupitt Thank you!