Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve MDX rendering performance #8533

Merged
merged 1 commit into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/thin-starfishes-love.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@astrojs/mdx': patch
---

Improve MDX rendering performance by sharing processor instance
1 change: 1 addition & 0 deletions packages/integrations/mdx/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@
"remark-rehype": "^10.1.0",
"remark-shiki-twoslash": "^3.1.3",
"remark-toc": "^8.0.1",
"unified": "^10.1.2",
"vite": "^4.4.9"
},
"engines": {
Expand Down
50 changes: 13 additions & 37 deletions packages/integrations/mdx/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
import { markdownConfigDefaults } from '@astrojs/markdown-remark';
import { toRemarkInitializeAstroData } from '@astrojs/markdown-remark/dist/internal.js';
import { compile as mdxCompile, type CompileOptions } from '@mdx-js/mdx';
import { markdownConfigDefaults, setVfileFrontmatter } from '@astrojs/markdown-remark';
import type { PluggableList } from '@mdx-js/mdx/lib/core.js';
import type { AstroIntegration, ContentEntryType, HookParameters, SSRError } from 'astro';
import astroJSXRenderer from 'astro/jsx/renderer.js';
import { parse as parseESM } from 'es-module-lexer';
import fs from 'node:fs/promises';
import { fileURLToPath } from 'node:url';
import type { Options as RemarkRehypeOptions } from 'remark-rehype';
import { SourceMapGenerator } from 'source-map';
import { VFile } from 'vfile';
import type { Plugin as VitePlugin } from 'vite';
import { getRehypePlugins, getRemarkPlugins, recmaInjectImportMetaEnvPlugin } from './plugins.js';
import { createMdxProcessor } from './plugins.js';
import type { OptimizeOptions } from './rehype-optimize-static.js';
import {
ASTRO_IMAGE_ELEMENT,
Expand Down Expand Up @@ -84,21 +81,7 @@ export default function mdx(partialMdxOptions: Partial<MdxOptions> = {}): AstroI
),
});

const mdxPluginOpts: CompileOptions = {
remarkPlugins: await getRemarkPlugins(mdxOptions),
rehypePlugins: getRehypePlugins(mdxOptions),
recmaPlugins: mdxOptions.recmaPlugins,
remarkRehypeOptions: mdxOptions.remarkRehype,
jsx: true,
jsxImportSource: 'astro',
// Note: disable `.md` (and other alternative extensions for markdown files like `.markdown`) support
format: 'mdx',
mdExtensions: [],
};

let importMetaEnv: Record<string, any> = {
SITE: config.site,
};
let processor: ReturnType<typeof createMdxProcessor>;

Comment on lines -87 to +84
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these options are moved to createMdxProcessor

updateConfig({
vite: {
Expand All @@ -107,7 +90,10 @@ export default function mdx(partialMdxOptions: Partial<MdxOptions> = {}): AstroI
name: '@mdx-js/rollup',
enforce: 'pre',
configResolved(resolved) {
importMetaEnv = { ...importMetaEnv, ...resolved.env };
processor = createMdxProcessor(mdxOptions, {
sourcemap: !!resolved.build.sourcemap,
importMetaEnv: { SITE: config.site, ...resolved.env },
});

// HACK: move ourselves before Astro's JSX plugin to transform things in the right order
const jsxPluginIndex = resolved.plugins.findIndex((p) => p.name === 'astro:jsx');
Expand All @@ -134,23 +120,13 @@ export default function mdx(partialMdxOptions: Partial<MdxOptions> = {}): AstroI
const code = await fs.readFile(fileId, 'utf-8');

const { data: frontmatter, content: pageContent } = parseFrontmatter(code, id);

const vfile = new VFile({ value: pageContent, path: id });
// Ensure `data.astro` is available to all remark plugins
setVfileFrontmatter(vfile, frontmatter);

try {
const compiled = await mdxCompile(new VFile({ value: pageContent, path: id }), {
...mdxPluginOpts,
elementAttributeNameCase: 'html',
remarkPlugins: [
// Ensure `data.astro` is available to all remark plugins
toRemarkInitializeAstroData({ userFrontmatter: frontmatter }),
...(mdxPluginOpts.remarkPlugins ?? []),
],
recmaPlugins: [
...(mdxPluginOpts.recmaPlugins ?? []),
() => recmaInjectImportMetaEnvPlugin({ importMetaEnv }),
],
SourceMapGenerator: config.vite.build?.sourcemap
? SourceMapGenerator
: undefined,
});
const compiled = await processor.process(vfile);

return {
code: escapeViteEnvReferences(String(compiled.value)),
Expand Down
143 changes: 31 additions & 112 deletions packages/integrations/mdx/src/plugins.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,101 +4,49 @@ import {
remarkPrism,
remarkShiki,
} from '@astrojs/markdown-remark';
import {
InvalidAstroDataError,
safelyGetAstroData,
} from '@astrojs/markdown-remark/dist/internal.js';
import { nodeTypes } from '@mdx-js/mdx';
import { createProcessor, nodeTypes } from '@mdx-js/mdx';
import type { PluggableList } from '@mdx-js/mdx/lib/core.js';
import type { Literal, MemberExpression } from 'estree';
import { visit as estreeVisit } from 'estree-util-visit';
import rehypeRaw from 'rehype-raw';
import remarkGfm from 'remark-gfm';
import remarkSmartypants from 'remark-smartypants';
import type { VFile } from 'vfile';
import { SourceMapGenerator } from 'source-map';
import type { Processor } from 'unified';
import type { MdxOptions } from './index.js';
import { recmaInjectImportMetaEnv } from './recma-inject-import-meta-env.js';
import { rehypeApplyFrontmatterExport } from './rehype-apply-frontmatter-export.js';
import { rehypeInjectHeadingsExport } from './rehype-collect-headings.js';
import rehypeMetaString from './rehype-meta-string.js';
import { rehypeOptimizeStatic } from './rehype-optimize-static.js';
import { remarkImageToComponent } from './remark-images-to-component.js';
import { jsToTreeNode } from './utils.js';

// Skip nonessential plugins during performance benchmark runs
const isPerformanceBenchmark = Boolean(process.env.ASTRO_PERFORMANCE_BENCHMARK);

export function recmaInjectImportMetaEnvPlugin({
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin is moved to recma-inject-import-meta-env.ts as-is without changes

importMetaEnv,
}: {
interface MdxProcessorExtraOptions {
sourcemap: boolean;
importMetaEnv: Record<string, any>;
}) {
return (tree: any) => {
estreeVisit(tree, (node) => {
if (node.type === 'MemberExpression') {
// attempt to get "import.meta.env" variable name
const envVarName = getImportMetaEnvVariableName(node);
if (typeof envVarName === 'string') {
// clear object keys to replace with envVarLiteral
for (const key in node) {
delete (node as any)[key];
}
const envVarLiteral: Literal = {
type: 'Literal',
value: importMetaEnv[envVarName],
raw: JSON.stringify(importMetaEnv[envVarName]),
};
Object.assign(node, envVarLiteral);
}
}
});
};
}

export function rehypeApplyFrontmatterExport() {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin is moved to rehype-apply-frontmatter-export.ts as-is without changes

return function (tree: any, vfile: VFile) {
const astroData = safelyGetAstroData(vfile.data);
if (astroData instanceof InvalidAstroDataError)
throw new Error(
// Copied from Astro core `errors-data`
// TODO: find way to import error data from core
'[MDX] A remark or rehype plugin attempted to inject invalid frontmatter. Ensure "astro.frontmatter" is set to a valid JSON object that is not `null` or `undefined`.'
);
const { frontmatter } = astroData;
const exportNodes = [
jsToTreeNode(`export const frontmatter = ${JSON.stringify(frontmatter)};`),
];
if (frontmatter.layout) {
// NOTE(bholmesdev) 08-22-2022
// Using an async layout import (i.e. `const Layout = (await import...)`)
// Preserves the dev server import cache when globbing a large set of MDX files
// Full explanation: 'https://github.com/withastro/astro/pull/4428'
exportNodes.unshift(
jsToTreeNode(
/** @see 'vite-plugin-markdown' for layout props reference */
`import { jsx as layoutJsx } from 'astro/jsx-runtime';

export default async function ({ children }) {
const Layout = (await import(${JSON.stringify(frontmatter.layout)})).default;
const { layout, ...content } = frontmatter;
content.file = file;
content.url = url;
return layoutJsx(Layout, {
file,
url,
content,
frontmatter: content,
headings: getHeadings(),
'server:root': true,
children,
});
};`
)
);
}
tree.children = exportNodes.concat(tree.children);
};
export function createMdxProcessor(
mdxOptions: MdxOptions,
extraOptions: MdxProcessorExtraOptions
): Processor {
return createProcessor({
remarkPlugins: getRemarkPlugins(mdxOptions),
rehypePlugins: getRehypePlugins(mdxOptions),
recmaPlugins: getRecmaPlugins(mdxOptions, extraOptions.importMetaEnv),
remarkRehypeOptions: mdxOptions.remarkRehype,
jsx: true,
jsxImportSource: 'astro',
// Note: disable `.md` (and other alternative extensions for markdown files like `.markdown`) support
format: 'mdx',
mdExtensions: [],
elementAttributeNameCase: 'html',
SourceMapGenerator: extraOptions.sourcemap ? SourceMapGenerator : undefined,
});
}

export async function getRemarkPlugins(mdxOptions: MdxOptions): Promise<PluggableList> {
function getRemarkPlugins(mdxOptions: MdxOptions): PluggableList {
let remarkPlugins: PluggableList = [remarkCollectImages, remarkImageToComponent];

if (!isPerformanceBenchmark) {
Expand All @@ -125,7 +73,7 @@ export async function getRemarkPlugins(mdxOptions: MdxOptions): Promise<Pluggabl
return remarkPlugins;
}

export function getRehypePlugins(mdxOptions: MdxOptions): PluggableList {
function getRehypePlugins(mdxOptions: MdxOptions): PluggableList {
let rehypePlugins: PluggableList = [
// ensure `data.meta` is preserved in `properties.metastring` for rehype syntax highlighters
rehypeMetaString,
Expand All @@ -152,38 +100,9 @@ export function getRehypePlugins(mdxOptions: MdxOptions): PluggableList {
return rehypePlugins;
}

/**
* Check if estree entry is "import.meta.env.VARIABLE"
* If it is, return the variable name (i.e. "VARIABLE")
*/
function getImportMetaEnvVariableName(node: MemberExpression): string | Error {
try {
// check for ".[ANYTHING]"
if (node.object.type !== 'MemberExpression' || node.property.type !== 'Identifier')
return new Error();

const nestedExpression = node.object;
// check for ".env"
if (nestedExpression.property.type !== 'Identifier' || nestedExpression.property.name !== 'env')
return new Error();

const envExpression = nestedExpression.object;
// check for ".meta"
if (
envExpression.type !== 'MetaProperty' ||
envExpression.property.type !== 'Identifier' ||
envExpression.property.name !== 'meta'
)
return new Error();

// check for "import"
if (envExpression.meta.name !== 'import') return new Error();

return node.property.name;
} catch (e) {
if (e instanceof Error) {
return e;
}
return new Error('Unknown parsing error');
}
function getRecmaPlugins(
mdxOptions: MdxOptions,
importMetaEnv: Record<string, any>
): PluggableList {
return [...(mdxOptions.recmaPlugins ?? []), [recmaInjectImportMetaEnv, { importMetaEnv }]];
}
65 changes: 65 additions & 0 deletions packages/integrations/mdx/src/recma-inject-import-meta-env.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import type { Literal, MemberExpression } from 'estree';
import { visit as estreeVisit } from 'estree-util-visit';

export function recmaInjectImportMetaEnv({
importMetaEnv,
}: {
importMetaEnv: Record<string, any>;
}) {
return (tree: any) => {
estreeVisit(tree, (node) => {
if (node.type === 'MemberExpression') {
// attempt to get "import.meta.env" variable name
const envVarName = getImportMetaEnvVariableName(node);
if (typeof envVarName === 'string') {
// clear object keys to replace with envVarLiteral
for (const key in node) {
delete (node as any)[key];
}
const envVarLiteral: Literal = {
type: 'Literal',
value: importMetaEnv[envVarName],
raw: JSON.stringify(importMetaEnv[envVarName]),
};
Object.assign(node, envVarLiteral);
}
}
});
};
}

/**
* Check if estree entry is "import.meta.env.VARIABLE"
* If it is, return the variable name (i.e. "VARIABLE")
*/
function getImportMetaEnvVariableName(node: MemberExpression): string | Error {
try {
// check for ".[ANYTHING]"
if (node.object.type !== 'MemberExpression' || node.property.type !== 'Identifier')
return new Error();

const nestedExpression = node.object;
// check for ".env"
if (nestedExpression.property.type !== 'Identifier' || nestedExpression.property.name !== 'env')
return new Error();

const envExpression = nestedExpression.object;
// check for ".meta"
if (
envExpression.type !== 'MetaProperty' ||
envExpression.property.type !== 'Identifier' ||
envExpression.property.name !== 'meta'
)
return new Error();

// check for "import"
if (envExpression.meta.name !== 'import') return new Error();

return node.property.name;
} catch (e) {
if (e instanceof Error) {
return e;
}
return new Error('Unknown parsing error');
}
}
49 changes: 49 additions & 0 deletions packages/integrations/mdx/src/rehype-apply-frontmatter-export.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import { InvalidAstroDataError } from '@astrojs/markdown-remark';
import { safelyGetAstroData } from '@astrojs/markdown-remark/dist/internal.js';
import type { VFile } from 'vfile';
import { jsToTreeNode } from './utils.js';

export function rehypeApplyFrontmatterExport() {
return function (tree: any, vfile: VFile) {
const astroData = safelyGetAstroData(vfile.data);
if (astroData instanceof InvalidAstroDataError)
throw new Error(
// Copied from Astro core `errors-data`
// TODO: find way to import error data from core
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we have AstroError:

import { AstroError } from "astro/errors";

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly copied the existing code over. We can follow-up a PR for this improvement 😬

'[MDX] A remark or rehype plugin attempted to inject invalid frontmatter. Ensure "astro.frontmatter" is set to a valid JSON object that is not `null` or `undefined`.'
);
const { frontmatter } = astroData;
const exportNodes = [
jsToTreeNode(`export const frontmatter = ${JSON.stringify(frontmatter)};`),
];
if (frontmatter.layout) {
// NOTE(bholmesdev) 08-22-2022
// Using an async layout import (i.e. `const Layout = (await import...)`)
// Preserves the dev server import cache when globbing a large set of MDX files
// Full explanation: 'https://github.com/withastro/astro/pull/4428'
exportNodes.unshift(
jsToTreeNode(
/** @see 'vite-plugin-markdown' for layout props reference */
`import { jsx as layoutJsx } from 'astro/jsx-runtime';

export default async function ({ children }) {
const Layout = (await import(${JSON.stringify(frontmatter.layout)})).default;
const { layout, ...content } = frontmatter;
content.file = file;
content.url = url;
return layoutJsx(Layout, {
file,
url,
content,
frontmatter: content,
headings: getHeadings(),
'server:root': true,
children,
});
};`
)
);
}
tree.children = exportNodes.concat(tree.children);
};
}
Loading
Loading