Rewriting Asset Links to Absolute Paths

[noio]

I'm trying out Eleventy because I did not like the way I'm forced to organise my files in Jekyll. I want to keep my posts (written in markdown) together with the assets (images, mostly) in those posts.

The creator of Jekyll Postfiles explains it better than I can.

The gist is: I'd rather not type absolute paths (/assets/2019-01-12-some-post/an_image.png) everywhere in my markdown files. I use Typora, and being able to just drag & drop images from the same folder is a delight. I also have markdown files (private logbooks) that are not meant for online, so I usually start out with relative file paths (![alt](an_image.png)). Sometimes I want to publish a part of those logbooks, and I would rather not rewrite all the image-URLs to absolute ones.

I have since discovered why this is not a 'solved problem' in Jekyll, as I realise that it's not entirely trivial.

The solution I have come up with is a quick hack but it might do the job. Imagine a folder structure as follows:

.
├── 2019-01-12-some-post
│   ├── index.md
│   └── an_image.png
└── _site
    └── 2019-01-12-some-post
        ├── index.html
        └── an_image.png

As shown above, when simply using ![alt](an_image.png), this works, because both the .md and the image will end up in the same folder.

As soon as the post uses a permalink, however, this obviously breaks:

.
├── 2019-01-12-some-post
│   ├── an_image.png
│   └── index.md
└── _site
    ├── 2019
    │   └── 01
    │       └── some-post
    │           └── index.html
    └── 2019-01-12-some-post
        └── an_image.png

In order to have the least opinion about where post assets should be located, my strategy would be to rewrite local asset URLS to an absolute path.

• The assets would still be pass-through copied to their original locations. In the example above, the image would still be located at /2019-01-12-some-post/an_image.png
• The URL in the rendered template would be rewritten to point to this absolute path

Is there a way to preprocess templates in this way in Eleventy?
I realise it's hardly an ideal solution, but I think it's important to preserve a "human-first" file structure for posts.

Alternative solutions would be:

• Not use permalinks, which would change my site's URLs (which is not cool)
• Use some {% liquid tag %}, which would break WYSIWYG image drag & drop in Typora.

[zachleat]

A ha. Interesting idea. I’ve been thinking about this for a bit and the best idea I’ve come up so far is to let you apply a directory data file (with a permalink set there) and have the image files use that permalink too for processing. As I’m sure you’re aware, passthrough copy files are not processed in this way and do not interpret data currently but I do like the idea.

Somewhat related: #344 (and maybe a stretch but thinly related to #117)

[noio]

Ah yes! That was something I was also wondering, whether I could change the 'permalinks' of images. In that case one would leave the images as relative paths, but just make sure they end up in the same folder, correct? That is the method of the Jekyll-Postfiles plugin too: it tries to figure out which directory the "adjacent post" will get built to, and copies the images to that same folder.

But...

I know this is going to sound like I want to 'have it both ways', but I think that method still restricts the file structure too much, because then you can have only one permalink per directory data.

I will illustrate what led to this, bear with me. For a project I've been keeping a logbook that was not initially intended to end up online. I just happily typed away in Typora and drag & dropped images into the document. When it got too long, I started working in a new document. This logbook consists of a few markdown files and a single folder of almost 300 screenshots that are used in those markdown files:

logbook
├── logbook-2017.md
├── logbook-2018-part-1.md
├── logbook-2018-part-2.md
├── logbook-2018-part-3.md
├── logbook-2019.md
└── screenshots
    ├── 2017-11-01\ at\ 14.34.57.png
    ├── 2017-11-03\ at\ 13.28.44.png
    ├── etcetera...
    ├── etcetera...
    └── 2019-01-10\ at\ 14.59.30.png

Because I just drag & dropped images into the document, all images are relative links: ![alt](screens/2018-10-08 at 17.42.01.png)

Ideally, I would just add some front matter to each of the logbook files, then dump the whole folder into my blogs posts directory. Obviously, each of the posts will have a different permalink, and thus end up in a different folder. Ideally, they would still source the screenshots from the same folder.

What I'm trying to avoid:

1. Rewriting all image links to absolute links manually . Not just because it's work, but also because I don't like including too much information about the site generator into the content itself. (E.g. incorporating /assets/ because that's what Jekyll happens to call it). In an ideal world my content would stay 'generator-agnostic'.
2. The even worse task: Figuring out which screenshots are used in which logbook, splitting them out, and copying the ones that are used in multiple logbooks to different folders.

[kleinfreund]

Can we maybe make Eleventy documents (e.g. markdown files) give the option (via front matter) to drag surrounding files with them, like a local version of passthroughCopy?

---
passthroughFiles:
- screenshots
---
# Logbook 2019

I’m just thinking out loud here. One could configure that via data files, too.

[noio]

Can we maybe make Eleventy documents (e.g. markdown files) give the option (via front matter) to drag surrounding files with them?

The problem is the same as with "data file permalinks" in the folder: which of the multiple documents in the folder will "drag" the files with them?

Rewriting the links to the absolute paths would be more flexible, when doing that it won't matter where the images actually end up.

Perhaps if passthrough resources were treated as 'first class' things, (having inputPath and outputPath) such a rewriting would not be so hard..

[kleinfreund]

The problem is the same as with "data file permalinks" in the folder: which of the multiple documents in the folder will "drag" the files with them?

Right, that would only work if each document exists in its own directory in which all its related assets are stored. I think this would be a sensible approach. Then, all content of a document’s directory would be copied to the same location if a certain configuration was used.

[noio]

I think this would be a sensible approach.

Sensible, maybe, but it is a restriction on the content structure. And honestly, it's not so fundamentally different from Jekyll's "one asset sub-folder per post" method.

See "What I'm trying to avoid..." above ☺️

I want to have my cake and eat it too. 😂 I want posts to be able to share content from arbitrary folders, without having to restructure my content. I do not want to go through and split up (and duplicate!) a folder of assets based on which posts they are used in.

It probably sounds like a lot to ask, but I think it would be the ultimate example of "it does not matter how your content is structured"

Edit:
Unless, in your solution of..

--- 
passthroughFiles: 
- screenshots 
--- 
# Logbook 2019

It is the case that EVERY post COPIES ALL ASSETS to it's own output folder. So the entire screenshots folder gets copied 5 times, once for every post, regardless of actual usage. Not the most elegant, but it would work.

I'm still more inclined to think that URL rewriting is a better solution.

[dweidner]

I like the approach proposed by @kleinfreund in #379 (comment). It could be extended even further to allow the use of glob patterns, regular expressions or a list of mime types:

---
passthroughFiles:
 - *.(jpg|png)
 - *.css
---
# Example Post

or

---
passthroughFiles:
 - \.(jpg|png)$
 - \.css$
---
# Example Post

or

---
passthroughFiles:
 - image/*
 - text/css
---
# Example Post

Of course the problems described by @noio remain.

[igr]

Hugo has a concept of a page bundle: it's a folder with an index.md file and everything in that folder belong logically to that file.

[zachleat]

Also possibly related to #237.

[robdodson]

+1 to the ideas proposed by @kleinfreund and @igr. I was wondering if it's possible to have a very basic system to start, something that says that all local files get carried over to the directory created by the permalink. And then layer on a more advanced system as a second round when folks have had more time to mess around with the idea.

If I need to use a permalink today is the recommended approach to either have all images live in a single top-level assets directory or write an additional build step that looks at my content and manually moves the images to the right permalink directories?

[igr]

@robdodson I ended up creating simple framework where I can simply set the slug in folder options and all the files will have the same slug (among the other features).

The thing is, after trying all these tools (from Hugo to this one), I realized I just need a basic framework that will utilize the power of existing JavaScript libraries, and where I can organize my site content it any way I want. Might be reinventing the wheel; sorry for that, but hey I just need something very basic that I can easily extend...

[zachleat]

This repository is now using lodash style issue management for enhancements. This means enhancement issues will now be closed instead of leaving them open.

View the enhancement backlog here. Don’t forget to upvote the top comment with 👍!

[veleek]

Just wanted to add another example of where re-writing asset links would be helpful. I've got a site with a structure vaguely like like.

events
├── 2016
│   ├── index.md
│   ├── route.md
│   └── things.jpg
├── 2017
│   ├── index.md
│   ├── route.md
│   └── stuff.jpg
├── etc..

If I add a image in one of the markdown files ![](things.jpg) then it all works great. I can review the markdown for both index.md and route.md and it shows up properly. But after I process it, route.md gets moved to /events/2016/route/index.html, and now the "correct" path for the image is ../things.jpg.

In a similar way, a user needs to be aware of where all their links are going so that they add an additional ..\ for anything that's not an index page.

[veleek]

@ndorfin I'm not sure this solves the original problem with permalinks. The actual resulting path for the document is different, but the image references are not updated and including the images in template formats won't pull the images into the permalink paths so the references will still break.

[ndorfin]

@ndorfin I'm not sure this solves the original problem with permalinks. The actual resulting path for the document is different, but the image references are not updated and including the images in template formats won't pull the images into the permalink paths so the references will still break.

Ahh, gotcha. Thanks, I've removed my comment.

[victornpb]

I just made this plugin to solve this limitation
https://www.npmjs.com/package/eleventy-plugin-page-assets

It will keep the same relative structures under the permalink directory, keeping everything organized.

This folder structure

📁 src/posts/
  📁 some-title/
    📄 index.md <-- when a template file is processed
    🖼 cover.png    assets relative to it are automatically
    🖼 image.jpg    copied to the final permalink folder
  📁 good-title/
    📄 index.md 
    🖼 cover.png

Output

📁 dist/
  📁 perma-some-title/
    📄 index.html 
    🖼 cover.png 
    🖼 image.jpg 
  📁 perma-good-title/
    📄 index.html 
    🖼 cover.png

---

I plan to add a html parser so only user images are copied to the dist folder.
Another feature would be to re-write the output paths, similar to what webpack does with loaders.

[hidegh]

Code for assets copy - assets stored in the same directory as the article/post itself. Logic tries to imitate how HUGO does the copy.

const path = require("path");
const fs = require("fs");
const glob = require ("glob");

module.exports = function (eleventyConfig) {

    eleventyConfig.addTransform("local-images", function(content, outputPath) {
        // HUGO logic:
        // - if multiple *.md are in a folder (ignoring _index.html) - then no asset-copy over will occure
        // - if single index.html (allowing extra _index.html), then no further sub-dirs will be processed, all sub-dirs and files will be copied (except *.md)
        //
        // Alg:
        // - get all md/html/njk in the directory and sub-dirs, ignoring _index.* (_index.* - could be later used to create list-templates)
        // - if only 1 found = we copy the entire sub-content
        // - otherwise do no copy-over nothing
        
        const template = this;

        console.warn(`TRANSFORM - input: ${template.inputPath}, output: ${outputPath}`);
        
        const outputDir = path.dirname(outputPath);       
        const templateDir = path.dirname(template.inputPath).replace(/^\.\//, "");
        const templateFileName = path.basename(template.inputPath);

        const extensionsRegex = template._config.templateFormats.join(",");
        const mdSearchPattern = path.join(templateDir, `**\\*.{${extensionsRegex}}`);
        const mdIgnorePattern = path.join(templateDir, `**\\_index.{${extensionsRegex}}`);

        const entries = glob.sync(mdSearchPattern, { nodir: true, ignore: mdIgnorePattern });

        // only 1 page template allowed when copying assets
        if (entries.length > 1) {
            console.info(`Skipping copying over files from: ${templateDir} as multiple tempaltes found in directory!`);
            return;
        }

        // copy all hierarchically, except templates
        const fileSearchPattern = path.join(templateDir, `**\\*`);
        const fileIgnorePattern = path.join(templateDir, `**\\*.{${extensionsRegex}}`);

        const filesToCopy = glob.sync(fileSearchPattern, { nodir: true, ignore: fileIgnorePattern });

        for (let filePath of filesToCopy) {
            // strip template dir
            // prepend output dir
            const destPath = path.join(
                outputDir,
                filePath.substr(templateDir.length)
            );

            const destDir = path.dirname(destPath);

            fs.mkdirSync(destDir, { recursive: true });
            fs.copyFileSync(filePath, destPath);
        }

        // keep original content
        return content;
    });

}

[AustenLamacraft]

This is great. I found I got TemplateWriterWriteError if I didn't return content where you just have return

[RichardJECooke]

I'll use hidegh's plugin for now, but can't we just make this a simple solution? A one-line to put in the eleventy.js file. Like:

config.setOutputFolderFormat
(
    `posts/`,   //input 
    `articles/item.date.getYear()/item.title.toFileSlug()/`   //output
);

and that will copy all files that aren't templates to the same folder. Then we don't have to go and put a permalink in every single file or a .json file in every folder.

[RichardJECooke]

Basically this, #1072, we need a really simple way in the config file to configure output paths

[zachleat]

eleventyConfig.addGlobalData("permalink", …); should do something similar, see https://www.11ty.dev/docs/data-eleventy-supplied/#changing-your-project-default-permalinks