add dynamic linker support

[mohaalak]

Why do we need it?

with the work of @reduz and alket soon we can have an asset sharing ( addon ) but the missing system is that this addons is all in gd script but there are so many works that can be done in modules like add Admob,IAP,OpenCV,CameraCaputre,VRsupport and so many more , this modules should be compiled and it's scary for beginners just to clone the godot repo add the module then download dependencies and compile it but think about it if the modules is compiled and can be added to engine it will be great.

Problems

• first of all that this modules should be compiled with compilers that the godot was compiled
• every version of godot can load just the modules for that version
• our last problem how can we implement it that it will be simple and easy to use both for module developers and engine users

my solution is that we just support those release on steam and in site the stable versions and if someone wants a custom version he should download the modules code and compile it themselves, for compatibility issue the modules should have a supported version and for different platform the compiled version should exist.

@akien-mga I will appreciate it if you add feature request and discussion label to this issue

[neikeq]

Related to #2701

[reduz]

I understand, but this is too much work for too little gain.

What we could do is configure Godot so you can set a path to your Godot
repository that you compile yourself, and downloading the addon will add a
module into Godot, that you can recompile.

On Sat, Mar 5, 2016 at 4:46 PM, Ignacio Etcheverry <notifications@github.com

wrote:

Related to #2701 #2701

—
Reply to this email directly or view it on GitHub
#3936 (comment).

[mohaalak]

I think it's too little gain is for you and me but think about the developers that they hope with godot they can bring food for their children 😝 and does not know anything about android sdks , ndk, gradle, or some of them even does not anything about git , I just say if it is lots of work just direct me in the right direction let me support this system but if you think that it is bad design for godot and you don't want that in godot then I think I should close this issue.

[Zylann]

Would having a C API make the problem easier to solve?

[ghost]

@Zylann it would, in fact #3943 would make it possible to write wrappers directly in GDScript with no performance penalties.

You could (for example) just distribute an OpenCV module written in GDScript itself and it would call C code directly, much like it happens in other languages with FFI support (like this). In short, there would be no need for compiling.

But it should also be possible to write modules and link them dynamically. I think this is a huge issue Godot has - having to compile your own C++ modules along with all of Godot makes no sense - but perhaps the problem is being cross-platform. What about something like Pluma?

[bojidar-bg]

Linking modules dynamically is not impossible (even without additional libraries), but might be a pain to setup correctly. I have discussed the issue with @est31 once and we actually went rather deep into it.

I might do this one day, but given the current state of affairs, for the next few months I have enough other issues to tackle first. If anyone of you wants to take it, I'm free to speak about it on IRC.

[reduz]

The problem is not linking modules dynamically, this is super easy. The
problem is creating modules for every single platform Godot supports.

On Wed, Jun 8, 2016 at 2:13 PM, Bojidar Marinov notifications@github.com
wrote:

Linking modules dynamically is not impossible (even without additional
libraries), but might be a pain to setup correctly. I have discussed the
issue with @est31 https://github.com/est31 once and we actually went
rather deep into it.

I might do this one day, but given the current state of affairs, for the
next few months I have enough other issues to tackle first. If anyone of
you wants to take it, I'm free to speak about it on IRC.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3936 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AF-Z25uZVzcO9fhZqKDHirjbOIhYiBpbks5qJvgsgaJpZM4HqEgW
.

[bojidar-bg]

@reduz That would be the module developers' problem, not ours. So not a big issue, since even now, the modules have to be compiled for every single platform.

[reduz]

Yeah, the problem is that it's a lot of work for the module developer.

On Wed, Jun 8, 2016 at 2:41 PM, Bojidar Marinov notifications@github.com
wrote:

@reduz https://github.com/reduz That would be the module developers'
problem, not ours. So not a big issue, since even now, the modules have to
be compiled for every single platform.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3936 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AF-Z2_-qH36U3CKDl09COFfwWIzGNxPwks5qJv64gaJpZM4HqEgW
.

[neikeq]

@reduz I think a developer won't mind that if it means more users will use the module (since users don't need to build it nor the engine). Also a developer may just release the source and the users could share their builds (like unofficial builds of Godot).

[reduz]

An alternative I thought about could be to make a simplified, generated
from bindings, godot API for this to link from C or C++. It would at least
make it easier to link and keep compatibility between versions

On Wed, Jun 8, 2016 at 3:02 PM, Ignacio Etcheverry <notifications@github.com

wrote:

@reduz https://github.com/reduz I think a developer won't mind that if
it means more users will use the module (since users don't need to build it
nor the engine). Also a developer may just release the source and the users
could share their builds (like unofficial builds of Godot).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3936 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AF-Z285dFHCuBTDnh5wr0iGAwUs9-0uuks5qJwOxgaJpZM4HqEgW
.

[ghost]

@reduz You mean like a libgodot.so which can be called from C++?

[reduz]

yes, exactly, you link your program against a wrapper header/lib and
generate a dll/so. Godot then loads this .so dynamically for every platform

On Wed, Jun 8, 2016 at 3:40 PM, paper-pauper notifications@github.com
wrote:

@reduz https://github.com/reduz You mean like a libgodot.so which can
be called from C++?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3936 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AF-Z2y0eIsGDfvLhf2OMazSX0AXb0-82ks5qJwyTgaJpZM4HqEgW
.

[ghost]

@reduz That's a pretty good idea! Does this approach have any limitations? It would speed up C++ development a lot. +1

[neikeq]

@paper-pauper the limitations is that you have the same API access as GDScript.

[ghost]

@neikeq that would not affect wrappers (for libraries like OpenCV and MongoDB), which would only need to call the register hook. I think this is a good solution (but still think writing wrappers in GDScript itself via #3943 should be implemented at some point for easier distribution, much like languages like Python have wrappers written both in C++ via the API and in Python itself via ctypes/ffi).

[Zylann]

I don't really like the fact that GDScript would be required to bind dynamic modules, because GDScript is a module itself, it is optional. Modules depending on other modules is not a bad thing, but for such a core feature the engine's API should be enough.

[ghost]

@Zylann For what @reduz proposed, GDScript is not required at all.

For what I proposed in #3943, you raise a good concern but GDScript is always bundled with Godot, so it'd be the best way to distribute wrappers (aside from C++, which is always possible but requires compilation, which can be less comfortable). What other language aside from those two would you suggest for wrappers? Would you rather not have wrappers at all than have them in GDScript?

[Zylann]

I wouldn't use an intermediate language, actually, or maybe use the reflection/Variant system, which is already a dynamic interface to the engine. I understand that it would reduce the requirements for the module creator. However, since inheritance it's heavily used in the engine, how would you inherit from Node, Reference, or Resource?

Also, theoretically, if you setup a standard build system for a module on each platform you want to support, you can easily update modules for newer versions of Godot.

At the same time, I'm curious about what will be the best solution. I've worked on a few projects involving dynamic modules and realized a lot of problems need to be solved. My last engine project was litteraly designed around loading modules dynamically. Just like with Godot, they had access to all the engine (which was itself a library), assuming they won't change during execution because it's too much work to maintain validity of objects. So adding or removing native modules requires to restart the engine, but from a user's point of view, that's way better already than having to setup a whole environment to build everything for any platform.

[reduz]

I'll revive this thread with a proposal. Let me know what you guys think about it.

First a bit of context:

Distributing dlls/so files in Godot as external modules is to difficult due to the many different compilers, versions, ABI changes, etc. In a bit more detail.

1. C++ is terrible for binary compatibility in shared objects, the ABI is different between compilers and compiler versions.
2. Even if the same compiler was used, C++ requires symbols (class) sizes to be the same in order to link properly

This makes adding extra functionality as a .dll linking against Godot too difficult to even attempt.

An idea lurked around about creating a more stable mirrored C++ API that replicates API functions using a safer API, but this would only solve 2), not 1).

So, how about the following? Some functionality must of course be used as an extension, such as if you want to add a module that might replace the physics engine, but for the large amount of libraries and other stuff that might be nice to bind (ie, SQLite, ODBC, DBUS, Kinect, FMOD, whathever), having access via the script API to this would be fine. I think this could be solved via the following:

1. Create a very simple dynamic API via C, not C++ which can access most godot functionality, but only provide a minimal API.
2. C ABI is much more stable and compatible between compilers and compiler versions than C++.
3. C ABI would be simple and access most of Godot via the reflection API instead of headers, ie

godot_class c = class_db_get_class("Node2D");
godot_instance ins = godot_instance(c);
godot_call_method( ins, "set_pos", godot_vec2(10,22) );

the idea would be to simply add scripts using C, not sure how this would work exactly (CScript could be a class where you edit which which C class to instance) .

This would allow us to bind all sort of C and C++ libraries without any risk, given the ABI will always be C.

What do you think?

[reduz]

To add a bit more depth on how this would be used, we would just add a specific type of addon that is a CPlugin or somethin like this, like

addons/sqlite/sqlite.Linux.x86.32.so
addons/sqlite/sqlite.Linux.x86.64.so
addons/sqlite/sqlite.Windows.x86.32.so
addons/sqlite/sqlite.Windows.x86.64.so
addons/sqlite/sqlite.android.arm7.so

etc

Even WebAssembly has shared object loading in the roadmap.

This way it would be super simple to distribute this kind of content, and even distribute it in the asset library.

[SuperUserNameMan]

I've reread twice, but still not sure to understand what you mean by "the idea would be to simply add scripts using C".

For instance, let's say I want to make a Text To Speech plugin.
My first intuition would be that my plugin would have to create or extend an AudioStream with some custom methods and properties. Would it be possible ?
How would I have to do ?

[ghost]

@reduz Looks clean and easy to use. Can't wait to see it in practice!

@Zylann I think it would be possible simply by using the "extern" keyword, since it'd be all dynamically linked.

[bojidar-bg]

@reduz I hope those binding are going to be created automatically, is this correct? 😄

[Zylann]

@paper-pauper the problem is, if a module links another at compile-time as a dependency, how would it be usable as a Godot C module?
My last engine project supported this case by simply letting the OS load the dependencies, putting the libs next to each other. The system keeps a reference count, so it doesn't matters if the engine loads it a second time after.
Just wondering if that would be possible with Godot :)

@reduz I guess it will need some boilerplate for making C++ modules easier to develop.

[reduz]

@bojidar-bg the bindings will give you access to the regular Godot reflection API, you don't have to do anything from Godot side

@Zylann I don't think one of these modules will link another one

[Zylann]

@reduz It might not be that frequent, but it can happen, and is certainly possible to do. I already have module ideas in that would rely on this. Otherwise, the same issue that led Godot to support C modules would be present in modules again.

[ghost]

What about using dlopen to load shared libraries at runtime instead of linking?

In this case, where is the issue? All libraries will be loaded in the same address space, so simply using extern or the equivalent in other languages will allow modules to call other modules. (Or maybe my C is the issue, and I'm getting rusty)

[Zylann]

@paper-pauper Why use a library for dynamic loading when you can just link at compile time? #ItJustWorks

[ghost]

@Zylann here we are talking about dynamic loading in any case, otherwise you'd still have to recompile Godot in order to write modules, and in that case how would that be different from the current approach?

[Zylann]

@paper-pauper I was asking this about modules depending on another module #3936 (comment). If you don't allow them to depend on another, you'll have the same problem as Godot: having to modify a module's source code and recompile it.

[ghost]

@Zylann and I said that by using dlopen, it is possible for modules to depend on other modules :)

[reduz]

@Zylann @paper-pauper @bojidar-bg I think you guys might be misunderstanding what this is.

This is a simple API to write C plugins that can be loaded dynamically. It does not intend to replace Godot modules (which are compiled-in) or even be a way to extend Godot using C++.

This this C API is created to distribute binary modules as ..DLL, .SO or .DYLIB (depending on the platform), that can be opened by any Godot version, in run-time and keeping binary compatibility.

The main use cases are:

1. Binding Godot to third party libraries without having to recompile Godot
2. Adding support to Godot for unexposed APIs present in the host OS without having to recompile Godot
3. Adding extra (fast) compiled bits of code to add-ons written mostly in GDScript without having to recompile Godot.

Unlike C++ modules that are compiled-in, this very simple C API will be limited because:

1. It won't allow to add new object types
2. It's a lot more difficult to write for
3. It won't have any access to Godot internals

[ghost]

@reduz I still think it's very cool, please go ahead 👍

[jeremyjh]

Here's a draft of how the C API would look like. It would be a single file:
http://pastebin.com/Qepcp5Nm
👍 3 ❤️ 2

Would there be a way to subscribe to notification events and signals? Or that would be possible already using the godot_script_ functions?

[BeppeInfo]

I see there is a Cscript module at godot source. Is this already available ? If so, how do I integrate libraries created with this API with GDScript ?

[SuperUserNameMan]

@Bitiquinho : it's not yet functional.

[touilleMan]

I'm currently working on a C++ module and it's a huge pain to have to re-link the entire godot project each time something change (especially considering godot output is ~120mo big vs my 10 .o files totaling 1.5mo...).

Furthermore, according to @reduz

The problem is not linking modules dynamically, this is super easy.

So it would make life really easier to be able to have dynamic linking support for godot modules (and to update the doc accordingly) 👍

[est31]

@touilleMan have you heard of gold? Its a very cool tool and has an incremental mode where it reserves some space so that when you change only a little bit, it will be able to re-link much faster than before.

[vnen]

@touilleMan I believe you can make a static module that dlopen a dynamic one, as suggested before.

[touilleMan]

@est31 thanks for the tip ! I got a speedup from 14.12s to 9.30s using gold on a single line change.

I manage to get my module compiled as a shared library, the result is much better (see here)

I takes 6.77s by running scons at the root directory (by comparison running scons with no changes takes 6.11s so it's mostly dependencies check overhead)

Finally I could go as low as 1.37s by specifying the target using scons bin/libmymodule.x11.tools.64.llvm.so (0.7s with no changes) given it cuts a lot the amount of dependencies 😃

However the road to get there was quite painful, so I think it should worth it to add a word about this in the documentation. If you agree I could work on this 👍

[ghost]

However the road to get there was quite painful, so I think it should worth it to add a word about this in the documentation. If you agree I could work on this 👍

@touilleMan dude that'd be great! 👍

[touilleMan]

godotengine/godot-docs#308 enjoy ! 😄

[ghost]

@touilleMan is awesomeMan, I'm sure this is gonna help lots of people 👍

[karroffel]

Just want to say that the DLScript module implements what reduz is proposing here.

See the readme for an overview of what it does.

It's almost finished, only a few edges that need rounding but it's feature complete.

Just leaving that here so nobody wastes resources to try to implement it themselves.

[27thLiz]

Closing, as GDNative (aka DLScript, aka cscript) is now implemented in the master branch :) 🎉