Eliminate site key in myst.yml?

[fwkoch]

Currently the myst.yml file has a site key and a project key - there is lots of confusion around the difference between these, how inheritance may or may not happen, where new keys go, etc. (as one example, see the discussion on this PR: jupyter-book/myst-theme#325)

If you look in the docs about frontmatter, site isn't even mentioned: https://github.com/executablebooks/mystmd/blob/main/docs/frontmatter.md In the code, there there isn't even a validateSiteFrontmatter(...) function; "site frontmatter" isn't really ever used. The closest we get is when building the "site manifest" we validate the frontmatter against the site theme template; this looks much more like export validation than frontmatter validation.

Why does the site section of the myst.yml exist at all? I think the only reason for this comes from the idea that a site may have multiple MyST projects: https://github.com/executablebooks/mystmd/blob/main/packages/myst-config/src/site/types.ts#L72 - that means site is a separate entity higher in the hierarchy than project.

In practice, however, project:site is always 1:1, especially as we improve "intersphinx/inter-MyST" - there's no reason to combine multiple projects into one site; just link two sites.

---

Because of all of this, I think it would be useful to eliminate the site from the project config. Configuring site would then work the same way as configuring another export - specify the format, a template, and any options required by the template.

---
exports:
  - format: site
    template: book-theme
    logo: my-logo.png
    ...
---

There would no longer be the same ambiguity around inheritance; project frontmatter is site frontmatter, and anything defined on the site export would override project frontmatter.

[choldgraf]

It sounds like my mental model of site: isn't what it was originally intended for. I had assumed it was short for website: and was only for configuration that was related to HTML themes.

The way you describe it here sounds reasonable to me, at least as a starting point. It feels like this would be a natural progression to start with:

1. Begin with the assumption of 1 myst project per myst.yml file. Use the minimal number of keys / config options to support this (e.g., prefer one project: key over 2 keys)
2. See what people do with this limitation. See if key stakeholders or many users hit an issue of "I want to have configuration that spans multiple myst projects)
3. If so, then do some UX research to understand their needs more deeply.
4. Use this to decide if / how to expand the scope of a MyST build to incorporate multiple MyST sites.

It sounds like @fwkoch's proposal is roughly following 1 above, which feels reasonable to me. IMO it's always better to start simple and only add complexity when there's a clear need.

That said, I do think that websites expose a unique "configuration surface area" compared to most other exports, so it might be helpful to have a way to easily group website-related config¹. Websites just tend to have a ton of bells and whistles, especially if you want to let people customize them with extensions or their own custom themes.

Footnotes

1. Sphinx does this with a dedicated key called html_theme_options. ↩

[stevejpurves]

We were previously working with multiple projects and composing those into myst sites in various ways but although it allowed you to do interesting things, it did get overly complex - stepping back to 1 project per myst.yml has made things much clearer.

This change though takes things further making the project the container or superset for potentially multiple sites and I'm not sure that makes sense beyond the 1:1, project:site mapping, although the change does make sense for collapsing frontmatter into one place for the 1:1 case. The export format options patterns already in place will also then let theme specific options to be piped through without requiring frontmatter / config changes.

Just left with the nagging doubt that maybe there was some other advantage of the separate site section that we are not seeing... cc @rowanc1?

[agoose77]

We spoke about this change a couple of weeks ago, and I'm still in favour. I tend to think about MyST sites as "exports", so formalising that is something I'd support.

I recognise that e.g. mystmd.org has multiple projects, but IIUC this is done using a different mechanism to projects:. As @fwkoch says, I feel that if we need cross-project links, we should be using our cross-project tools (intersphinx et al.) for that. If someone wants to do a "custom" kind of deployment, they can just build the contents and aggregate them together with a custom theme.

Regarding @stevejpurves point, we could opt to make certain classes of exports singletons if like, such as sites. I don't see a reason to impose that limitation, but it's always available to us.

[choldgraf]

I agree that assuming "one project per myst.yml" is sensible. It feels like there's general consensus around this, are we blocking on something to move forward?

For future consideration, I'd like to see us make multi-project MyST sites by leaning into the MyST-as-data idea. For example:

• If you wanted to standardize things like "header buttons" across multiple MyST projects (similar to our mystmd.org docs), I'd like to be able to define one source of truth, and have other MyST projects point to that source of truth for their own configuration to re-use. Similar to {include} but at the config level.
• If you wanted to share things like search across MyST sites within the same website structure, you could allow MyST to compile a shared search index by scraping pre-defined sites (similar to what Algolia does, but just from within the MyST build workflow).

[choldgraf]

One use-case I just thought of that might require some consideration. What do other folks think about this one:

Multi-lingual MyST projects

Within a book context (e.g. community knowledge bases), it is fairly common to want multiple languages of the same MyST project. For example, mysite.org/en/page1/ and mysite.org/fr/page1/. In that case, you'd want things like the MyST interface to use the language of the respective pages (so en/ used Table of content and fr/ used the phrase Table des matières).

Would the decision in this issue result in users needing to maintain two separate MyST configurations for this?

[agoose77]

@choldgraf I am not so experienced in the norms in the space of multi-lingual projects. If it's desirable to separate different languages entirely, then perhaps we want to have multiple site exports and set a locale key in the MyST config, so that we export each language as its own site?

@fwkoch how do the ergonomics of that sound from the perspective of e.g. hosting multi-locale sites on a single vercel server? I'm thinking that the current direction that we're going in sounds like independent sites (one for each export).

By the same token, I think we can also just move the locale key into the site export, such that it takes care of all of this itself. So, I'm not worried.

[choldgraf]

My thinking was a structure something like this:

English version

myproject/en/myst.yml
myproject/fr/myst.yml

en/myst.yml has all the configuration of the project, and project.locale = en

fr/myst.yml has a line like file: ../en/myst.yml which pulls in all the config from that file. It's then followed by a line with project.locale = fr which over-writes the locale config from above.

I wonder if we can learn from the way that Docusaurus or other heavy-duty docs / website frameworks handle this?

[rowanc1]

Very much in favour of moving to a 1:1:1 mapping of myst.yml : project : site. It will simplify a lot in the code. Choosing a 1:1:1 is a large-ish code change/refactor, that closes off a path in favour of dramatically simplifying the development. It isn't that much of a change user-facing because we have already removed most of the docs around this.

I also want to echo what @choldgraf said on the importance of the site:

I do think that websites expose a unique "configuration surface area" compared to most other exports, so it might be helpful to have a way to easily group website-related config.

IMO, keeping a top-level site key in the config is helpful, and ensures that there is a 1:1 of project : site, and emphasizes this as the primary way to use myst. The site is different than an export as there can only be one of them per project.

For the internationalization, I think we can combine some thinking with #1123 to improve the sharing of important config elements (e.g. all the downloads), and having two myst.ymls one for en one for fr, etc. will work well. That will require some build/deployment work to stitch them together, but something that netlify, for example, will do super easily. This is also similar to the versioned content, which are multiple myst.ymls in time and then stitched together with a hosting provider like RTDs/Curvenote.

My take away: there are not a lot (any?) user facing changes to this, but there are a ton of things we can remove from the code if we make this call, most of which are not in the docs (nested myst.ymls, traversing upwards to find if you are in a nested project, having a state that keeps track of your projects, loading the site and project configs separately, etc.). There is a ton of complexity in supporting this currently, especially around the config discovery and loading, which we don't advertise and won't need.

---

The possible user facing change is that we could move from:

version: 1
project: 
   title: My project
   authors: []
site:
   template: article-theme

to treating a myst.yml as a single project always, and drop the project nesting.

version: 1
title: My project
authors: []
site:
   template: article-theme

Not too fussed about that change, and it is easy to change from a validation point of view after we have made some other decisions, but it might be nice to say "your myst project" rather than "the project in your myst project".

[choldgraf]

IMO this is an easier structure to grok:

version: 1
title: My project
authors: []
site:
   template: article-theme

So I think I'm in favor of @rowanc1's proposal. It sounds to me like the only reason not to do this is if we think we're closing the door on some critical future functionality. But it doesn't feel like that's the case...

And generally speaking, I think "reducing complexity" should be a guiding principle unless there's a clear value proposition for the functionality that would require the extra complexity.

[fwkoch]

Closing this, as we have decided not to eliminate the site key - it is not just "another export" but a fundamental part of a myst project. Flatting of myst.yml has it's own issue here: #1308