Consider migrating to SILE org and/or core

[alerque]

I am personally of the opinion that while it is a good thing that this can be a third party package and I'm glad we hashed out the issues involved in making enough API surface area for this to work as an external package, this is actually a fundamental enough functionality that will be useful to enough other packages that it should be kept a little closer to SILE's core.

Would you consider one or both steps of moving this into the SILE org and publishing stable versions from the API key there; and secondarily re-splicing the code into the SILE core so it can be tested and refactored on the same release cycle?

Keep in mind that since 3rd party packages take precedence over core ones if any aspect of the core one was not to your liking you could continue to use the forked/3rd party one over the official/core one without anything different that what you're doing now.

[alerque]

Specific proposal:

1. Transfer repo to SILE org.
2. Fork a copy back to this location for ongoing development / author's use.
3. Publish from SILE's luarocks API key.
4. Either add it as a LuaRocks dependency by default or splice a copy (history and all) back into the SILE core repo as a default package.

[Omikhleia]

I am embarrassed... Really.

First, this package is important to me (it's used everywhere in my book, and above all it's the key component in a "choose your own adventure"-like chapter, which would have been much harder without it¹). It's one of the first I completed to what I considered a pretty satisfying state in the SILE 0.12.x era, and proposed in April 2022 as a PR - since the topic was mentioned in an issue, and in a (now removed) Wiki page.

Now, countless rebase operations afterwards, the requests comes both too late and too soon. ..

Too late... This package now has its life as a 3rd-party module... I could say: "No man ever steps in the same river twice"... But even though, this package is now also cluttered with technical debt and issues, in the SILE 0.14+ era...

• It severely breaks in case of multiple package instantiation (Cross-referencing package sile-typesetter/sile#1366 (comment)). The rationale for multiple package instantiation eludes me (Multiple package initialization case? sile-typesetter/sile#1531) and the obscure reasoning that SILE shall have it this breaking way because "some packages need it" eludes me too². Having to struggle with it regularly is not a pleasant conclusion...
• As it plays at low-level (e.g. with SILE.Commands, leveraging tocentry etc.), it will break it the grand scheme of "package unloading" (a "feature" I don't get the ultimate need either). One way of solving this is obvious, but unpleasant... (I am not really interested in leveraging the "standard" tableofcontents package and book class, which I don't use and are, IMHO, mere little "toys" with fairly low quality, so...)
• As noted at the time (... and footnotes have the same issue), things like this work poorly when inadvertently transferred to running headers, table of contents, etc. How to address the very issue of contextual "fragile" commands? I have ideas, but they require deep changes to the core SILE processing - without any hint that SILE is ready for this...³

Too soon... While it went to its "fireproof" in my book, that doesn't mean it's so clever...

• I'm needing it too for cross-references in Markdown (Cross references support markdown.sile#36) and a first PoC shows how lacking it is...
• It could suffice for simple use cases, but I do feel it's amiss something... Once the main work on markdown is done, it's likely something I'll want to re-check... Though I can't say when, with all the other missing bits to the Plan "as envisioned". I'm lagging behind my objectives...

Still, some good questions are raised...

so it can be tested and refactored on the same release cycle?

Or 3rd-party packages would need a way to be tested against several versions of SILE and Lua... (Omikhleia/markdown.sile#26)

Keep in mind that since 3rd party packages take precedence over core ones if any aspect of the core one was not to your liking you could continue to use the forked/3rd party one over the official/core one without anything different that what you're doing now.

So I'd be using a forked version of my own package, not the one SILE would provide. Er. What's the point for anyone?
Doh, it's MIT-licensed. If SILE wants to fork it, I cannot oppose... but do not turn the tables, now, heh.

But still, I feel embarrassed (and too tired to push my thought further, barely recovering from the sickness that struck me last week... Yeah, that was it, with the loss of smell and taste and all ^^ ;)

Footnotes

1. As a writer, the ability to reshuffle the sections until they fit well, without having to bother about the numbering and linking, was important during all the rewritings of that very chapter... which went through to at least 4 full rewriting attempts. (What else to expect, from a "base" story that was written more than 20 years ago, in a different time of imagination. Lol). ↩

2. Pray tell, what are the packages in question. Footnotes comes to mind. Is it right in what it does, though? I could challenge that: it's current implementation is, in my opinion, defective by design. It's as bad as it can seem: frame definitions only know how each frame is placed with respect to other, so how and where space should be taken or not - and it might differ on different page masters. But frames and insertions, as implemented, are thus defective by design too... ↩

3. I think SILE core has a serious governance issue and lack of direction. And this is not a criticism of your actions @alerque - It's more global. We are dealing with a pre-alpha quality software (poor implementations, real poor code quality, half-baked concepts, bad code commenting... Do I need to go on, here? I'll sound all too negative again). That was ok 8+ years ago when Simon initiated it, but it would be time to grow up. In other terms (more positively oriented?), I'd prefer to see the core components improve and fix issues, rather than adding an additional one, with all its flaws and limits... ↩