Skip to content

The State of the PPX Transition

Jump to bottom
Sonja Heinze edited this page Jan 17, 2022 · 4 revisions

Issues ppxlib was Facing

The issues we were facing and are tackling with our current plan are the following:

  1. To keep the PPX ecosystem cross-compiler compatible

    a) ppxlib was handling and abstracting away parts of the unstable compiler-libs API;

    b) OMP/ppxlib was keeping the AST migration information up-to-date.

    That situation required us and the compiler developers to coordinate for each compiler release in order to guarantee that the new compiler could be used in projects with ppx rewriters.

  2. To guarantee new feature support, ppxlib needs to bump the ppxlib AST to the newest version, which implies a breaking change. That was an issue for a homogeneous and up-to-date PPX ecosystem.

  3. Not all ppx rewriters had migrated from OMP to ppxlib. That was also an issue for a homogeneous and up-to-date PPX ecosystem.

Our Three-Part, Multi-Step Plan and its Current State

Part One: Implementing Astlib in Six Steps

The first part of our plan works towards continuous cross-compiler compatibility (Issue 1. above) while making the situation of still having ppx rewriters based on OMP (Issue 3. above) even more of a problem. It consists of implementing an interface module called Astlib between ppxlib and the compiler, then upstreaming it to the compiler. Here are its six steps:

  1. Moving the OMP driver and other OMP features from OMP to ppxlib. That was done in August 2020, and it introduced OMP2. This was the start of having the PPX ecosystem split into the two incompatible worlds of OMP1 ppx rewriters on one hand and ppxlib ppx rewriters on the other hand.
  2. Reducing the compiler-libs usage in ppxlib as much as possible to keep Astlib minimal. That has been completed over time.
  3. Implementing a ppxlib internal version of Astlib as a layer between ppxlib and the compiler. The first ppxlib version containing Astlib is 0.23.0.
  4. Working on an upstream version of Astlib for the compiler and adapting the tooling to update the AST migration logic to the compiler. That's next on our ToDo list.
  5. Discussing the upstream version and the workflow to update the AST migrations with the compiler developers, with the aim to upstream it.
  6. Simultaneously to upstreaming Astlib, implementing the logic to use our internal Astlib on old compilers and the upstream Astlib on new compilers into ppxlib.

This part of the plan got a bit delayed when the compiler got frozen to integrate multi-core support. With our current roadmap, we're aiming for having Astlib upstreamed into OCaml 5.1.0.

Part Two: Porting ppx rewriters to Re-unite the Ecosystem

Moving the driver from OMP to ppxlib (Step 1. above) made the fact that some ppx rewriters were still based on OMP (Issue 3. above) even more of a problem since it split the PPX world in two. For some PPX developers, the reason to avoid porting their ppx rewriters to ppxlib was that ppxlib depended on base and stdio, so we decided to tackle this situation by three means:

  • Dropping the base and the stdio dependencies, which was done in August 2020.
  • Porting and reviewing some of the most important ppx rewriters ourselves. So far we've ported js_of_ocaml, bisect_ppx, and tyxml with the help of the respective maintainers, and we've also reviewed several ports.
  • Spreading the word about the need to port ppx rewriters and asking for help. A lot of people have joined the effort!

In summer 2020, we made a non-exhaustive list of ppx rewriters to be ported. By now (January 2022), all ppx rewriters on that list have been ported.

Part Three: Sending Patch PRs when Bumping the AST

Our current plan doesn't provide a solution for the problem that a few ppx rewriters break whenever we bump the ppxlib AST (Issue 2. above), but it does make handling the problem more efficient and takes away the burden from the PPX developers. Since the AST bump to 4.10, whenever we bump the AST, we create a workspace with all ppx rewriters fulfilling a certain standard, fix the rewriters that break all at once, and open patch PRs. We call that workspace the ppx-universe. You can find a ppx-universe snapshot on our repo.

Our plan is to make the tooling that creates and interacts with the workspace more sophisticated. We use opam-monorepo as main tool. For the rest, we are currently using a ppxlib-specific bash script.

Clone this wiki locally