Add suppport for external ports

[ypujante]

Purpose

The purpose of this PR is to add support for ports that are not embedded in emcc distribution at all, so they are external.

Syntax

From a user point of view the syntax is simple to use and understand:

# use of an embedded/built-in port
./emcc --use-port=sdl2 ./test/hello_world.c -o /tmp/out.js
# use an external port
./emcc --use-port=/tmp/foo.py ./test/hello_world.c -o /tmp/out.js
# use an external port (can be a relative path)
./emcc --use-port=my_ports/foo.py ./test/hello_world.c -o /tmp/out.js

Why is this useful?

As mentioned in one of the PR (#21276) about contrib ports, there are several additional use cases that are missing. In particular this change allows:

• work on a port (either new version of an existing port or brand new) without the need to either cloning the project or messing up with the latest distribution locally
• create/distribute a port that is not part of emcc at all (could be promoted to contrib ports if there is traction in the future for example)
• not relying on whether the port will be accepted or not in emscripten (if the port is not accepted then without this feature a port cannot be used)
• not relying on the emscripten release cadence and worfklow (for example if there is a bug in a port, fixing the bug and releasing a fix, or simply a new version, will take time dependent on emscripten release/test cycle)
• lock down a version of a port (which I was trying to achieve via the tag option): simply copy the ports file in a project and use that copy instead, which is even better than the tag option because it copies both the version of the tarball fetched, as well as the recipe to download it and build it

Changes

The changes to the code are fairly straightforward: if the port name ends with .py then assume it is a local file, and load it. Because it is possible to use a port that is not part of emscripten at all, I was able to create a fairly comprehensive test case (in other) which, as a neat side effect, also tests more of the port functionality in general.

Alternative solution

Another possible way to implement a similar concept would be to add the ability to have additional ports directory, something like:

# reusing the example from earlier (foo.py is in /xxx/my_ports)
./emcc --ports_dir=/xxx/my_ports --use-port=foo ./test/hello_world.c -o /tmp/out.js

This would allow to achieve a similar outcome. I did not go down this route because it would require the addition of yet another emcc option and I know it is not your preference. I also feel like it is a little bit more cumbersome to use. That being said I can implement it this way if you think that would be a better way to go.

[sbc100]

Nice!

I like the idea of supporting out-of-tree ports in this way.

I do worry that it means that we might have to start worrying about the port ABI though. Especially since we are running python and there is nothing really stopping port authors from depending on any emscripten internals they want to.

We can try to lock it down and not break it, but also maybe we should just have a previos that says "be aware that the port API used by emscripten is not 100% stable and could change between versions". Perhaps just adding that to the docs is enough.

[ypujante]

@sbc100 I am really happy you seem to like the basic idea. I am pretty satisfied with the end result too!

I agree with you about the ABI concern and I will make sure to update the documentation properly to mention it before ready to merge to main.

[ypujante]

@sbc100 I have made the changes we discussed (except the documentation which I will update once you are good with everything).

It is probably not a big issue, but if I change the code in a port (for example, the code in the create inner function which generates the library), the library does not get regenerated and I have to manually clear it. I am not too sure if that could be done automatically but wanted to bring it up to your attention if you have any idea on the topic.

[ypujante]

I think I found a way to check for out of date library:

def build_single_port(port, settings):
  libs = port.get(Ports, settings, shared)
  if port.last_modified_time > 0:  # 0 for built-in ports
    need_rebuild = False
    for lib in libs:
      need_rebuild |= os.path.getmtime(lib) < port.last_modified_time
    if need_rebuild:
      port.clear(Ports, settings, shared)
      libs = port.get(Ports, settings, shared)
  return libs

What do you think?

[sbc100]

@sbc100 I have made the changes we discussed (except the documentation which I will update once you are good with everything).

It is probably not a big issue, but if I change the code in a port (for example, the code in the create inner function which generates the library), the library does not get regenerated and I have to manually clear it. I am not too sure if that could be done automatically but wanted to bring it up to your attention if you have any idea on the topic.

Indeed, there is no kind of dependency system in the ports tree. Its up to users to force a rebuild via ./embuild build --force xxx or ./embuild clean xxx if changes are made to xxx.

Not that whenever a new version of emscripten is tagged/released all builds are cleared and rebuilt so in general this doesn't effect our users, only folks who are working on emscripten or the ports themselves.

[sbc100]

I think I found a way to check for out of date library:

def build_single_port(port, settings):
  libs = port.get(Ports, settings, shared)
  if port.last_modified_time > 0:  # 0 for built-in ports
    need_rebuild = False
    for lib in libs:
      need_rebuild |= os.path.getmtime(lib) < port.last_modified_time
    if need_rebuild:
      port.clear(Ports, settings, shared)
      libs = port.get(Ports, settings, shared)
  return libs

What do you think?

I assume port.last_modified_time is the timestamp on the port file itself?

We could maybe consider something like this a followup. If we do I don't see why we wouldn't also this for builtin ports for folks that are making changes there? But then also we might want to rebuild if the shared code in __init__.py changes too? I think it might be forever-imperfect way of detecting if a rebuild is necessary since so many thing can also trigger then need for a rebuild (e.g. changed the llvm/clang version).

[ypujante]

I assume port.last_modified_time is the timestamp on the port file itself?

Yes I set it this way (for external ports):

  # path is the path provided in --user-port=<path>
  port.last_modified_time = os.path.getmtime(path)

We could maybe consider something like this a followup. If we do I don't see why we wouldn't also this for builtin ports for folks that are making changes there? But then also we might want to rebuild if the shared code in __init__.py changes too? I think it might be forever-imperfect way of detecting if a rebuild is necessary since so many thing can also trigger then need for a rebuild (e.g. changed the llvm/clang version).

I would argue that built-in ports, __init__.py, llvm clang and others are by definition included with emscripten and are not really supposed to change unless somebody is tinkering with the internals, which is fine that it is not supported.

On the other end, and obviously by definition, external ports are not included and so can be modified. Here is a use case which for an end-user would be extremely confusing (and wrong) and it is not a use case of a port developer, but a use case for a port user:

• let's say raylib creates a port for each of their release (their release cadence is somewhat frequent): raylib.py
• in my project I use raylib so I copy their port locally and write emcc --use-port=local_ports/raylib.py ...
• a few days later, raylib publishes a new version of their library and update their port accordingly (which contains a new tag)
• in my project I copy the latest version of the ports file write emcc --use-port=local_ports/raylib.py ...
• => the port is not rebuilt, hence I believe I use the latest version of raylib when I am not, no error message, no nothing

I feel like that would be a pretty bad experience and also quite an unreasonable expectation that a user of a port (not a developer of a port) would have to tinker with the emscripten cache.

[sbc100]

I assume port.last_modified_time is the timestamp on the port file itself?

Yes I set it this way (for external ports):

  # path is the path provided in --user-port=<path>
  port.last_modified_time = os.path.getmtime(path)

We could maybe consider something like this a followup. If we do I don't see why we wouldn't also this for builtin ports for folks that are making changes there? But then also we might want to rebuild if the shared code in __init__.py changes too? I think it might be forever-imperfect way of detecting if a rebuild is necessary since so many thing can also trigger then need for a rebuild (e.g. changed the llvm/clang version).

I would argue that built-in ports, __init__.py, llvm clang and others are by definition included with emscripten and are not really supposed to change unless somebody is tinkering with the internals, which is fine that it is not supported.

On the other end, and obviously by definition, external ports are not included and so can be modified. Here is a use case which for an end-user would be extremely confusing (and wrong) and it is not a use case of a port developer, but a use case for a port user:

• let's say raylib creates a port for each of their release (their release cadence is somewhat frequent): raylib.py
• in my project I use raylib so I copy their port locally and write emcc --use-port=local_ports/raylib.py ...
• a few days later, raylib publishes a new version of their library and update their port accordingly (which contains a new tag)
• in my project I copy the latest version of the ports file write emcc --use-port=local_ports/raylib.py ...
• => the port is not rebuilt, hence I believe I use the latest version of raylib when I am not, no error message, no nothing

I feel like that would be a pretty bad experience and also quite an unreasonable expectation that a user of a port (not a developer of a port) would have to tinker with the emscripten cache.

Is that unreasonable to ask the emcc user to rebuild the raylib port explicitly each time they update raylib (via git fetch or whatever)? If you think thats too much to ask then lets discuss adding auto-rebuild feature as followup PR.

[ypujante]

Is that unreasonable to ask the emcc user to rebuild the raylib port explicitly each time they update raylib (via git fetch or whatever)? If you think thats too much to ask then lets discuss adding auto-rebuild feature as followup PR.

I am fine for a follow up PR. There is just NO way right now to rebuild an external port except manually tinkering with the internal emscripten cache. embuild know nothing about external ports.

[sbc100]

Is that unreasonable to ask the emcc user to rebuild the raylib port explicitly each time they update raylib (via git fetch or whatever)? If you think thats too much to ask then lets discuss adding auto-rebuild feature as followup PR.

I am fine for a follow up PR. There is just NO way right now to rebuild an external port except manually tinkering with the internal emscripten cache. embuild know nothing about external ports.

I think being able to build everything with embuilder is very useful, and also the direction I would like to move things in going forward.

Having ports built magically at compile or link time is just a little too magical, so having this non-magical explicit way to build everything for users who want that is good IMHO.

So I think the logical next step would be to ensure that embuilder can be used to build ports directly and not just a side effect or building some other thing.

[ypujante]

Is updating embuilder a requirement for this PR or a follow up one?

[sbc100]

Is updating embuilder a requirement for this PR or a follow up one?

Followup would be great!

[ypujante]

Ok sounds good.

I know none of the current port use this pattern, but a way to alleviate the new version/not rebuilt scenario is to use the tag in the library name:

TAG = '1.0.4'
HASH = 'xx'

def get_lib_name(settings):
  return (f'lib_contrib.glfw3_{TAG}.a')

It is actually quite a fairly common practice to have the version number part of the library name...

And if the tag changes, hence the library name changes, hence the new code is fetched/rebuilt

[sbc100]

It is actually quite a fairly common practice to have the version number part of the library name...

Sometime the major version is added to the end of the library (e.g. glfw3) yes. This does mean that if you want to support referencing the library in the normal way you would then need to do emcc -lglfw3-1.0.4 .. which looks a little odd but maybe is fine. BTW how does the 1.0.4 version here relate the version of glfw? Is it completely independent?

[ypujante]

All tests are passing. I did check that the test for external ports which cannot run in other (due to frozen cache) are correctly running in the cross platform tests:

test-windows:
test_external_ports_simple (test_other.other) ... ok (4.30s)
test_external_ports (test_other.other) ... ok (58.71s)

test-mac:
test_external_ports_simple (test_other.other) ... ok (1.01s)
test_external_ports (test_other.other) ... ok (3.61s)