This RFC, designed by @kabergstrom and myself, introduces a flexible approach to adding scripting support to the engine.
Amethyst is currently fundamentally a Rust project: its usage is exclusively in Rust, it depends heavily on cargo and the Rust toolchain, etc. In order to extend the scope of Amethyst outside of the Rust ecosystem, the engine needs to open its doors to other foreign languages. In order to achieve this successfully, a scripting API needs to be carefully crafted so that the experience is flawless even from those foreign languages.
But scripting integration is not only about the language barrier. Scripts are usually associated with flexibility in the way code is written. Indeed, it is not uncommon to see scripts being loaded at runtime (or even reloaded) and treated as pure assets. It is this kind of philosophy this RFC aims at introducing in the engine.
The architecture introduced in this RFC aims at fulfilling the following constraints:
- Support any language (the design should not assume a specific programming language)
- Enforce the safety of the ECS (it must be guaranteed that no behavior in the scripting languages can impact the safety of safe code in the ECS)
- Close to no overhead (Rust as a scripting language should be as fast as Rust natively)
- Expandable (adding new scripting languages should be easy, even for 3rd parties, and possible without modifying the engine itself)
- Little maintenance for each languages (adding new features to the scripting API should not require an update to the language-specific parts)
- Support the creation of components and resources from the scripting environment (without using Rust) including complex data structures
- Support hotswapping (components, systems and resources can be modified and created without restarting the engine)
- Independent from Rust (the engine must provide most of its feature to users that do not know Rust)
Now that the goals are clear, let's jump into the design proposal.
In this proposal, a script is a typical asset in the sense described by the Asset Management and Pipeline RFC (#875). On the building of the game, it goes through a first building pass, converting it in built artifacts which are then loaded into the engine and used.
Scripts have a very limited scope; generally, they only represent a single System or implement the methods for a single State. This heavy fragmentation allows for precise control over thread safety and the general execution pattern (and, as a bonus side effect, keep the useful aspects of ECS fragmentation, notably for testing purposes).
The communication between the scripts and the engine is done through "augmented FFI". The engine will expose a typical C-like API, that will be then discovered and augmented by a language driver (see details below). This will allow automated expansion of the API and high level concepts such as struct methods, iterators, and more. Normal FFI headers are first automatically generated by cbindgen, then additional metadata on the generated header data is provided to the builder. Discussed changes in cbindgen's architecture might make the augmentation of FFI even easier. That means exposing an API to the scripting environment could potentially be as simple as making a function in an extern block.
This proposal assumes we are ready to bring new features into specs. None of them should be too convoluted, so we do not detail them, as most of them are part of the specs 2.0 design work. Please assume reasonable changes to specs when reading the implementation details.
A language driver is the piece of software handling the language-specific endpoints of the API. Their work is divided in two major parts (following the concepts of #875): building and linking the scripts to the engine, and loading and running the scripts into the runtime. The modularity of this approach allows the user to choose what language is best for each tasks, which could be a very useful feature for large game studios (for example choosing Rust for hot rendering-related code and Lua for simple game logic).
The first role of the language driver is to take a raw script asset and convert it into a runnable format. In order to do that, the language driver first discovers the engine's API. This happens at build time, so that the driver does not have to be manually updated when we add new features to the engine. In other words, we only need to change the scripting API once, and all language drivers will discover the newly updated API. The API would contain types and functions a script can use.
In order to expose that API in an idiomatic way to the language they drive, the drivers are also provided with additional high level metadata, along with the FFI. For example, if a type is an iterator, the language driver will be noticed of this additional information so that it can, for example, do the necessary language-specific work for this iterator to be iterated over in a for loop (see the Lua example for more details).
The complete list of additional metadata provided by the engine is not entirely determined yet. But the existence of such metadata means language drivers will only have to implement wrapping for those general constructs (struct methods, iterators, etc.) to be compatible with the complete and future list of engine features, without any update needed when the new features land in. This extended API is called "augmented FFI".
Notice that as all of this is done at build time, the communication between the engine and the driver can be done without any mandatory overhead, directly through FFI.
Once the augmented FFI is resolved, it is exposed to the individual script files to build, and the generated artifacts are passed down the pipeline.
Please note that while it is implied that this step is happening at build time, it can also happen at runtime for specific use cases (such as, for example, modding). However, when possible, it is very beneficial to do it at build time.
Please also note that the building step should be able to be ran in a standalone manner, so that the asset pipeline remains an opt-in technology.
The runner part of the language driver is attached to the game's runtime. They load the built script and run them in their proper context depending on their type. For example, if a script is a System, then the runner adds a new system in the dispatcher that will handle the running of that specific script.
In practice, the runner will also have to handle parallelism for languages that do not natively support it. See the Lua implementation details below for an example of such a situation.
Additionally, the runner needs to enforce safety rules. Part of it can be enforced by restricting the available APIs at build time.
The first rule is isolation: in order to enforce mutability rules, Systems must not be able to communicate with each other directly. How this is done in practice depends on the scripting language (see Lua below for an example).
The second rule is enforcing ECS lifetime guarantees. We need to make sure the scripts cannot use the references the dispatcher is providing them in a dispatcher stage where it is unsafe. We found multiple solutions to guarantee this. The first one is to ban scripts from handling their own threading. This will make sure execution will stop at the end of the run method, guaranteeing the references can only be used when the dispatcher knows it is safe. However, banning threading entirely might be too much of a constraint. We do not believe so, but we designed alternatives anyway because we love you. To keep threading and still enforce lifetimes, we can either use a runtime lifetime check called "reference gates", or provide limited threading.
Reference gates act as a lifetime runtime check. When the dispatcher gives a reference to the language driver, it wraps it into a box with a gate. The gate is controlled by the dispatcher. As long as it is open, the reference can be used freely (it is however abstracted away, the script cannot copy it). But as soon as it is closed, it will panic if used.
This has the obvious drawback of totally happening at runtime, costing a tiny bit and potentially panicking in convoluted situations.
Another alternative would be to provide limited managed threading, in a way similar to JavaScript WebWorkers: jobs running in a parallel thread without shared memory, communicating with memory ownership transfers. Those transfers would be carefully monitored to disallow the passing of references.
A drawback of this approach is that communication between threads can not be fast because it is message-based. It might also involve a complicated implementation.
Anyway, keep in mind that what approach is chosen can be different for each language, as different languages give us more or less flexibility (Rust as a scripting language for example already enforces all of this out of the box).
With this approach, supporting hotswapping is pretty straight forward. The language driver's runner, depending of the nature of the scripting language, simply reload the script if the language permits it, or attach the scripts as a dynamic library it can reload entirely at any point.
This means that this would allow the hotswapping of Rust code. In fact, Rust could become an excellent scripting language for the engine as it would bring in hotswapping, asset-like management of Rust code and tighter incremental compilation.
Naturally, users will want to register components and resources from the scripting world. To do this, the user would define components and resources as an external schema that is processed by the language driver's builder. We will quote here what I said in amethyst/specs#462 regarding this issue:
- Creating a new component type at runtime could be handled by having a special non-generic storage type that would store
Vec<u8>. ThoseVec<u8>would contain the raw data of the new components. Then, those components would be exposed to the scripting runtime through newly generated C headers corresponding to the component schema the user provided. That however will mean that TypeId can no longer be the only thing determining ResourceId, as those storages would share the same type but a different ID. But it does not seem unreasonable to have ResourceId become a(TypeId, InstanceId), at least from my uneducated point of view. Also, at the end of the execution, the dynamic component should be collapsed into a statically generated component to no longer need to store the data in aVec<u8>. The typical wokflow would be: create your components on the fly, do your testing with it, and if it's too slow then reboot the game. It seems reasonable to me, and besides I don't really expect it to be that slow, especially if you're only using it for logic creation. - Modifying existing component types on the fly would only work on component types that were created from the scripting environment, but I think it's a reasonable constraint. Modifying a component in dynamic state would simply be a matter of iterating over the
Vec<u8>storage and moving the data inside around. Modifying a component in statically-collapsed state however is a bit trickier, but not that hard either. You need to replace all static instances of that component into a dynamic instance of the new component. - Removing an existing component type at least is easy. Just remove all instances of it and leave it all alone, scared, in a white void up to the next reboot where you end its sufferings.
On the matter of resources, they simply can behave the same way component storages do.
As specified, resources and components would declared using a schema language (the exact language is yet to be determined), offering the user numerous primitives such as different number types, strings, vectors, algebra primitives, etc. That way, the user can represent complex data structures from the schema. All those primitives must be sufficiently represented with C types so that they can be shared by all drivers. One important aspect is that, when possible, they do not hold references.
A user can potentially force a reference to be stored in a component of a resource. But this is out of the scope of the scripting support: local safety has already been traded off by not using Rust. We only need to ensure the scripting languages can not disrupt the safety of code outside of the scripting world.
Let us now take every goals one by one and explain how this proposal solves them specifically.
- Support any language: a language driver can be written for any programming language that supports C FFI. While this is not every language, most languages that are relevant to game scripting support some form of FFI that would be sufficient to put this proposal in practice.
- Enforce the safety of the ECS: the rules enforced by a language driver's runner part will guarantee that the ECS is safe.
- Close to no overhead: all the communication happens through simple C FFI without additional processing, which is the fastest possible solution.
- Expandable: language drivers do not require modifying the engine and are very modular, making them easy to plug in.
- Little maintenance for each languages: as the augmented FFI is automatically discovered at build time, the language drivers expand the API they expose by themselves, without a need for maintenance.
- Support the creation of components and resources: the primitives exposed in the schema will offer a large palette to design complex structures.
- Support hotswapping: see above.
- Independent from Rust: if enough features are available to scripts, it will be possible to make it no longer necessary to write any Rust code to use the engine to its full potential.
Languages drivers may be a bit hard to design for intricate languages. Please comment with additional stuff that has not already been mentioned in the explanations.
Language drivers can be implemented by any 3rd party, but we believe the Amethyst organization should at least maintain a Rust driver and another scripting language driver. We believe LuaJIT would be a good target for a first implementation. We will not enter into details regarding the Rust driver as it would be pretty minimal anyway (most of the work is already handled by the language).
- Extremely popular: thanks to its simplicity and flexibility, Lua is a very popular language in the game development world, and so would make Amethyst very accessible. This would also be an impressive move, as we would show it is possible to use languages users are familiar with in Amethyst's context.
- Portability: LuaJIT can be ran on a large amount of platforms and CPU architectures.
- Excellent performances: LuaJIT is the fastest JIT compiler available, and is extremely good at compiling performant code. This would be a good way to show that we too can offer excellent performances even in the context of an "easy" language. See this experimental LuaJIT implementation for example performances.
- Isolation: it is relatively easy to isolate Lua scripts from each other, even in the same VM, and control exactly what APIs are available to them.
The LuaJIT driver's builder make extensive use of the very fast LuaJIT FFI support. In its build stage, the driver will add a thin layer of idiomatic Lua over the FFI bindings. The driver will create Lua metatables for all types depending on the traits they implement, and insert them on all returned values of the corresponding types. For example, this can allow for operator overloading for math types like Vector3. Once this step is applied to scripts, their bytecode is dumped as built artifacts to make runtime loading faster.
Lua is fundamentally not parallel. Therefore, in order to run a Lua System in parallel in the context of the dispatcher, we need to introduce a Lua VM pool. As its name implies, this pool will contain LuaJIT VMs that a System added by the driver can request to run.
Note that a Lua System will have to remain VM-local. To do that, we simply attribute a VM to each system in a round-robin fashion, and hope for the best. Unfortunately there is no really better way than doing it this way, beside a complicated heuristics-based approach. We definitely are open to suggestions on this.
This work does not cover utilities for par_join()-like functionality in all languages. This could be done by implementing a Unity-like job system over this current approach. Considering it would be a drop-in addition, we decided that this feature could land later.
Implementing such feature would boil down to having a system that would add a layer of joining before calling the script, so that the script itself is the core of the parallel for loop.