GDScript 2.0: .gdc bytecode files not created with export option

[KoBeWi]

Godot version

5d806b4

System information

Windows 10 x64

Issue description

This option

does not work. The pck/zip still contains original .gd files, with no .gdc.

Steps to reproduce

1. Export project with Compiled Bytecode option (enabled by default)
2. Look for gdc files in the exported project

Minimal reproduction project

No response

[vnen]

This option needs to be removed, this feature won't be added to Godot 4.0.

For the record, my plan is to add an intermediate representation, so we can store compiled scripts in the export, which is a much better solution than the current one (present in 3.x). The problem is that this won't happen before 4.0 is out. But adding the mentioned feature here would be a good amount of effort for a half-measure that would be obsoleted in the next Godot version.

[Koalamana9]

@vnen Could you explain in more detail what intermediate representation means and how this new method differs from the current script tokenization in 3.x? Thanks!

[vnen]

Could you explain in more detail what intermediate representation means and how this new method differs from the current script tokenization in 3.x? Thanks!

A bit late, but here's an explanation.

What is done in 3.x is converting the tokens in the file to a binary format. For example, if in the source script you have var x = 1 it is converted to TK_PR_VAR TK_IDENTIFIER("x") TK_OP_EQUAL TK_CONSTANT(1) (names here for visualization, in the file it's only their numeric representation). When loading this the tokenizer can skip actually looking for the source string, so it doesn't have to deal with whitespace or comments for instance. Given the binary data has a strict format, it's much faster to tokenize than looking at the source code.

That's the only thing done though. The tokenization phase is almost free in this case but the script still has to be parsed and compiled when loading.

With an intermediate representation, the script is stored as a file already tokenized, parsed, and compiled. For a similar example, the line var x = 1 would be stored as something like stack(0) = constant(1) (local variable names don't need to be stored, they are referred by their stack addresses). When loading, it only needs to convert from the IR to the actual in-memory bytecode used by the GDScript interpreter. This process is similar to loading the .gdc file, except it doesn't have to go through the more demanding parser and compiler steps.

This has more benefits besides loading performance:

1. We could potentially remove the GDScript parser and compiler on release builds, leaving the executable smaller.
2. This also applies to type information. A lot is needed in release because GDScript has to figure things out when compiling. If it doesn't need to do so, then this information is not necessary after export.
3. The parser can become more thorough in type-checking without worrying about load times in release builds.
4. The code is somewhat obfuscated, for people who worries about having scripts peeked or stolen. It ignores local variable names and the instructions are much simpler, similar to assembly, which makes it harder to follow (and this after the effort of converting the binary format to something readable).

[Calinou]

We could potentially remove the GDScript parser and compiler on release builds, leaving the executable smaller.

This may have to remain something done in custom export template builds regardless, as it'll prevent hot-loading of custom hand-written GDScript code in exported projects (which can be desired in some situations). That said, this is an uncommon use case, so maybe it's OK to have this be something you explicitly have to compile support for.

[Riteo]

@Calinou we could just keep that for debug releases. That might be a problem if the editor can also run release binaries (can it?), but I think that this would be a nice compromise.

[vnen]

That might be a problem if the editor can also run release binaries (can it?), but I think that this would be a nice compromise.

@Riteo AFAIK the editor doesn't run any template executable, it always runs the same executable as the editor itself.

Also, my idea was to make it optional, I should've made that clear. We could consider whether or not we want to remove it from default/official builds but we can take that decision when the time comes.