Fix experimental_indexers importPath being ignored

[svallory]

Closes #25554

What I did

Added a simple "or" to try to use indexers.importPath but fallback to the original file path.

Couple of things to note

1. importPath is currently required. I think it is safe to make it optional with the default being the path of the file being indexed. I haven't added a test case because of this.
2. Contribution guide didn't mention anything about bumping the version, so I didn't

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

[svallory]

@JReinhold when you have some time, please take a look at the questions I had

[svallory]

Hey @JReinhold, I got my storybook sandbox working yesterday to test this PR and there was a piece missing. The builder-vite package also needed to be updated to handle the importPath correctly.

Initially I thought of importing StoryIndexGenerator from core-server but I quickly ran into issues related to esm vs cjs. So instead I assumed that a custom importPath provided by an indexer will ALWAYS be virtual. I think that's

Similar problems may arise for other builders. Since I'm still learning about the inner workings of Storybook, it would be great if someone from the core team could take at this. It would take me quite some time to verify since I would still need to setup sandboxes for each builder.

If someone finds and issue with a builder and points me to the right location, I can quickly fix other builders applying the same logic.

After I've received an approval for the implementation, I'll add tests, an importPath validation, and a note to the docs to inform of the importPath requirements. For example, that the importPath needs to be either the original path or a virtual module path following vite/rollup convention (if that's what we decide)

[svallory]

crap... @ndelangen I wanted to pull and did the opposite 🤦🏻 and I just realized it's too easy to do a force push with Lazygit.

I noticed there was a commit from you before I did this atrocity. Was it just a rebase on top of latest-release?

[svallory]

I've rebased the branch on top of latest-release and added a commit that fixes the ExtractPresetType (renamed to UnwrapPresetType) and (hopefully) all the build issues

[svallory]

Hey @JReinhold or @valentinpalkovic, it looks to me that what we need now is to update the snapshots due to the change of the import path. But I think it is better if someone from the core team take a look at it

[JReinhold]

So instead I assumed that a custom importPath provided by an indexer will ALWAYS be virtual.

Can you tell me more about this, why do you assume this?

[JReinhold]

I'm seeing lots of changes here unrelated to importPath - mostly preset handling and typing. Are they a result of a bad rebase/merge or are all these changes intentional?

Either way I think we should keep them out of this PR.

[JReinhold]

Is there a scenario you can think of where an indexer would ever create a virtual path for a story file?

I believe that's actually the point of this whole exercise @tmeasday , and the only use case for indexers to specify their own importPath. (We explored this topic further up in the PR but I understand it's hard to keep up with this)

If you have an input file - allFixtures.json that you want to split up into multiple components, the best way to do that is to generate a virtual module for each component, that the indexer then sets as the importPath.
I'm not sure there's any other use case for setting an importPath different from the original

We can enforce a specific prefix for virtual paths (traditionally virtual:pkg-name), and informed the user of the requirement on every import error "in case they are trying to use a virtual path.

Vite already establishes a convention for this with the virtual: and \0virtual: prefix and I think we should just lean into that. If the importPath starts with either fo those, consider it a virtual module. If not, assume it's a file path and convert it to a relative path if needed.

https://vitejs.dev/guide/api-plugin#virtual-modules-convention

[svallory]

I believe that's actually the point of this whole exercise @tmeasday , and the only use case for indexers to specify their own importPath.

That's exactly right. In my case I'm creating an interceptor between Marko and Storybook to allow stories to be written in Marko. I can mess with the files, otherwise I'll break Marko's build.

I actually tried (a lot) to get around the need for a virtual path, but not matter what I did, at one point or another, things would go haywire. Vite simply isn't very reliable or predictable when it comes to plug-in execution :(

Vite already establishes a convention for this with the virtual: and \0virtual: prefix and I think we should just lean into that.

That's my recommendation too. But just to give credit where credit is due, that mechanism and convention was established by Rollup ;)

(Sorry, I guess I'm a bit bitter that Vite become the standard and now I have to endure it hahaahaha but seriously... I've lost countless hours because of it)

[tmeasday]

Ok, thanks for filling me in (and sorry for not reading the whole thread). That approach sound sensible to me!

[dunklesToast]

just came across this thread because i think this would also solve an issue i am currently facing. would it be possible for somebody from the @storybookjs/core team to create a canary release for pr so i can check if this works for my usecase as well or if i need to dig somewhere else?

[storybook-bot]

Failed to publish canary version of this pull request, triggered by @valentinpalkovic. See the failed workflow run at: https://github.com/storybookjs/storybook/actions/runs/9356213289

[storybook-bot]

Failed to publish canary version of this pull request, triggered by @valentinpalkovic. See the failed workflow run at: https://github.com/storybookjs/storybook/actions/runs/9356213289

[nahtnam]

Same here, happy to test a canary. Thank you @svallory for working on this

[svallory]

Quick update:

Sorry for being MIA. I'm working on a big launch due next week and have no time at all to work on anything else. If anyone wants to take this over the finish line, please go ahead! There isn't much left to be done.

I'm available to answer any questions. Just send me a DM on twitter @svallory_en or discord

[dunklesToast]

I'd love to take over but after reading the thread I am not 100% sure whats left to be done here? Probably fixing the TypeScript types that break the build right now and some manual testing?

[svallory]

I'd love to take over but after reading the thread I am not 100% sure whats left to be done here? Probably fixing the TypeScript types that break the build right now and some manual testing?

Hey @dunklesToast, that's great! There isn't much left to do. IIRC, all that is left is:

• Make sure absolute paths are turned into relative ones
  This will affect StoryIndexGenerator.ts and maybe can be entirely solved there.
  For context, check these comments:
  • #issuecomment-2135635954
  • #issuecomment-2136405529
  • #issuecomment-2136852474
  • #issuecomment-2137047417

Note

Be aware of Rollup's virtual module convention for virtual modules adopted by vite:

• Updating the tests accordingly
• Create a simple sandbox example that displays the use of experimental indexers.
  The documentation has some examples at the end that you may use so you don't have to create one from scratch.
  Just make sure the path you are using to load the stories is a virtual one or, at least, not the original path

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit aff1957. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 1 target

• nx run-many -t build --parallel=3

---

Sent with 💌 from NxCloud.

[tobiasdiez]

Note that another alternative to virtual paths is to add certain query segments / "sub parts" to the import url, e.g. Button.stories.ts?story=abc. According to https://vitejs.dev/guide/api-plugin#virtual-modules-convention, this can be used if the code in the virtual module is derived from a real file:

Note that modules directly derived from a real file, as in the case of a script module in a Single File Component (like a .vue or .svelte SFC) don't need to follow this convention. SFCs generally generate a set of submodules when processed but the code in these can be mapped back to the filesystem.

(the vue vite plugin uses this, eg https://github.com/vitejs/vite-plugin-vue/blob/ac1255989d53bcd98f72f2e6c16946a00402eac6/packages/plugin-vue/src/main.ts#L314-L320)

[tobiasdiez]

Thinking about this a bit more, would it make sense to write the story index (storybook-stories.js) in the form

const imports = {
   story-id: () => import("import-path")
   ....for each story
}
function importFn(storyId) {
   return await imports[storyId]
}

where both story-id and import-path are obtained via the story indexer. And import-path can be any path vite can handle as import (e.g. an absolute path, something with alias or a virtual module).
Then

storybook/code/core/src/preview-api/modules/store/StoryStore.ts

Lines 141 to 147 in 6388f93

	async loadCSFFileByStoryId(storyId: StoryId): Promise<CSFFile<TRenderer>> {
	const { importPath, title } = this.storyIndex.storyIdToEntry(storyId);
	const moduleExports = await this.importFn(importPath);
	// We pass the title in here as it may have been generated by autoTitle on the server.
	return this.processCSFFileWithCache(moduleExports, importPath, title);
	}

could be rewritten as

async loadCSFFileByStoryId(storyId: StoryId): Promise<CSFFile<TRenderer>> {
    const moduleExports = await this.importFn(storyId);

    // We pass the title in here as it may have been generated by autoTitle on the server.
    return this.processCSFFileWithCache(moduleExports, importPath, title);
  }

[svallory]

Note that another alternative to virtual paths is to add certain query segments / "sub parts" to the import url, e.g. Button.stories.ts?story=abc. According to https://vitejs.dev/guide/api-plugin#virtual-modules-convention, this can be used if the code in the virtual module is derived from a real file:

It's not an alternative to the issue at hand if it requires an existing file.

Also, depending on the language/framework you are creating the plugin for, even if there's a file, you may not be able to process the request before the language/framework plugin does

[svallory]

Thinking about this a bit more, would it make sense to write the story index (storybook-stories.js) in the form

@tobiasdiez I think the entire process would benefit from a little cleanup (well, more like a big refactoring TBH)

The flow is confusing and hard to follow. I suspect over the years the need for optimization, or handling new use cases, led to adding more functions, breaking the process of indexing and loading into so many specialized functions.

But now that you guys have a clearer picture, I think a do-over with a cleaner separation of concerns (introducing a CSFFile class/module for example) would go a long way

[tobiasdiez]

Note that another alternative to virtual paths is to add certain query segments / "sub parts" to the import url, e.g. Button.stories.ts?story=abc. According to https://vitejs.dev/guide/api-plugin#virtual-modules-convention, this can be used if the code in the virtual module is derived from a real file:

It's not an alternative to the issue at hand if it requires an existing file.

Also, depending on the language/framework you are creating the plugin for, even if there's a file, you may not be able to process the request before the language/framework plugin does

Sure! That remark was meant to provide an alternative solution as long as this PR is not finished. I would love to see support for virtual modules.
For many purposes (eg your marko plugin ?) the stories should be defined in an existing file, so add a sub part query may work.

Thinking about this a bit more, would it make sense to write the story index (storybook-stories.js) in the form

@tobiasdiez I think the entire process would benefit from a little cleanup (well, more like a big refactoring TBH)

The flow is confusing and hard to follow. I suspect over the years the need for optimization, or handling new use cases, led to adding more functions, breaking the process of indexing and loading into so many specialized functions.

But now that you guys have a clearer picture, I think a do-over with a cleaner separation of concerns (introducing a CSFFile class/module for example) would go a long way

From what I have seen in the codebase, I agree with your judgement.

[svallory]

For many purposes (eg your marko plugin ?) the stories should be defined in an existing file, so add a sub part query may work.

I see your point, and I think the existing Marko plug-in works using a similar technique.

The plug-in I created though is not for rendering Marko components in stories, but to write the stories in Marko.

And just in case someone doing a similar plug-in finds this...

In general, for files that need compilation before becoming a JS CSF File, there are two options:

1. Intercept the transpilation: cut off the language plug-in and instrument the compiler (this is what the Svelte stories plug-in does)

2. Use a virtual path: add a virtual path to storybook index, when that path triggers the plug-in, convert it to the real file path, import it and the language plug-in will do its thing and you'll get a JS module (this is what this PR will allow)

[tmeasday]

Hey @svallory @tobiasdiez thanks for the discussion.

The flow is confusing and hard to follow. I suspect over the years the need for optimization, or handling new use cases, led to adding more functions, breaking the process of indexing and loading into so many specialized functions.

I won't speak to the confusing part but the architecture of the indexers/importFn is still as designed. @shilman and I talked about it briefly and sketched out this diagram:

The key point here is that the starting globs (stories field in main.ts) are fed into both the story index generator and the builder (which produces storybook-stories.js / the importFn). The generated index is not fed into the builder.

The reason for this is that the webpack importFn is not really similar to what you've sketched above. It uses the glob directly to ensure performance, and that added and removed CSF files are picked up by webpack:

storybook/code/lib/core-webpack/src/to-importFn.ts

Lines 46 to 58 in ab66906

	async (path) => {
	if (!${importPathMatcher}.exec(path)) {
	return;
	}
	const pathRemainder = path.substring(${directory.length + 1});
	return import(
	/* webpackChunkName: "[request]" */
	/* webpackInclude: ${webpackIncludeRegexp(specifier)} */
	'${directory}/' + pathRemainder
	);
	}

--

Another element of this is that the importPaths are intended to be builder agnostic, and are used in other places besides simply calls to importFn(importPath). For instance the importPath is used in the index for docs entries to relate which CSF files are imported by a given MDX file.

Another future use case is relating dependent CSF files by importPath.

---

In general, for files that need compilation before becoming a JS CSF File, there are two options:

I guess I wonder about what's the issue with the first option?

[tobiasdiez]

Thanks @tmeasday for the nice overview.

One problem with this design is that whatever the indexer returns as the importPath is never reaching the builder and thus cannot be used to construct a correct importFn.

As an idea, would it work to use the story index index.json (via importing it) to build the correct importFn? For this maybe the generation of index.json needs to be moved to the builder.

[tmeasday]

One problem with this design is that whatever the indexer returns as the importPath is never reaching the builder and thus cannot be used to construct a correct importFn.

As an idea, would it work to use the story index index.json (via importing it) to build the correct importFn? For this maybe the generation of index.json needs to be moved to the builder.

But this is my point. Webpack in particular does not and would not use the generated index to construct it's importFn, so although what you are suggesting might work in Vite, it would not have any effect in Webpack.

[tobiasdiez]

I'm completely ignorant when it comes to webpack, but is there a fundamental issue with having the following dependency chain?

stories (via glob) -> indexer (index.json) -> importFn script

Then if the stories are changed, the indexer output changes as well; which invalidates importFn and triggers the HMR of it I'd assumed that this would work once the indexer output is part of the builder (so that webpack/vite can monitor it).

Another idea would be to call the indexer for the generation of the importFn script.

Or how would you envision how the importFn makes use of the importPath returned by the indexer?

[JReinhold]

In general, for files that need compilation before becoming a JS CSF File, there are two options:

1. Intercept the transpilation: cut off the language plug-in and instrument the compiler (this is what the Svelte stories plug-in does)

2. Use a virtual path: add a virtual path to storybook index, when that path triggers the plug-in, convert it to the real file path, import it and the language plug-in will do its thing and you'll get a JS module (this is what this PR will allow)

I guess I wonder about what's the issue with the first option?

@tmeasday AFAIK only the second option would provide a way to define multiple separate metas from the same file. Eg if you had a json-file from which you wanted to create 3 components with 10 stories each. Because CSF only allows one component per file, you'd instead generate 3 virtual files from that single json file path.

Although TBH I haven't actually tried returning IndexInputs with different titles from the same file path.

The second option does sound slightly interesting to me, based on the Svelte work I've done. Given you could instead create a virtual CSF file that wraps the original Svelte component I imagine the complexity around transforming an already-internally-transformed file would be smaller. A few cases spring to mind:

• today we have an issue with variable collision, if the generated story export variable names collide with user-imported variables. [Bug] cannot use some names in imports addon-svelte-csf#167
• we depend on referencing the Svelte-internal component variable. If they changed that variable name in a patch (because it's an internal construct) it would break the addon.
• we have to be careful (and the user too) about which order the Svelte CSF Vite plugin is added, because its transformer must run after the Svelte transformation, but before the experimental Vitest integration addon.

I'd expect all of the above would be solved by moving away from transforming the file inline to generating a wrapper CSF file instead.

[svallory]

@tmeasday thanks for the overview!

I guess I wonder about what's the issue with the first option?

To add to what @JReinhold said about this, the main issue with "intercepting the transpilation" is that the development of a "simple" plugin for storybook suddenly requires knowledge of how the language compiler works.

In the case of Svelte, the plugin needed to run after the Svelte transpilation. In my case, since Marko integrates tightly with the server for extensive runtime optimization (which I believe may also be the case for Qwik and Imba), that wasn't an option. I would need to take control of the Marko compiler, proxying it's integration entirely. Even with deep knowledge of its internals, the implementation would end up fragile and break easily with future changes.