Is there a way to include files based on an externally generated list

[alerque]

I see from #148 (thanks @mbme) where proc macros for include/exclude were added, but the documentation is a bit sparse on how the order works. What I'd actually like to have happen is use the multiple-includes feature to iterate over a known list of resources. I have pre-generated lists of files I'd like to embed already at build time. Unfortunately there is other stuff mixed in the source folders, so just specifying a top level folder isn't too practical, nor is manually maintaining a list of thousands of includes. I can have the list available as an ENV var or written to a file before build, so potentially I could have build.rs do some lifting here ... is there a mechanism I can use to specify an explicit include list, then exclude=* to block everything else from the folder?

[pyrossh]

Hi @alerque
We don't have support for that yet. But something like reading from an file that lists all the files/folders that need to be included can be implemented I guess. But why don't you move all the required files/folders into a new folder and embed that. This would require a it of scripting I guess wouldn't this be the same a generating a file with the includes.

[alerque]

I thought I was being pretty tricky here, but just realized another design issue has me blocked.

I came up with a full file list to include. I auto generated this using the build system that had the data anyway, and used it to write out a file that is just a flat list of #[include = "..."] statements which a whitelist of what I want to include. That list I stuffed into the rust file setting up the embeds using a template system.

The trouble is, it seems a whitelist is useless. Exclude takes total priority, so the only way this is going to work is with a blacklist of every file in the file system I don't want to include, which is much more complicated.

I'm now in a catch 22. Copying files to an isolated location defeats a large chunk of the purpose of using this crate in the first place: being able to use filesystem data while working in the development tree and only having assets embeded in release builds.

Is there any way we can get some sort of prioritized globing worked out, perhaps more like .gitignore rules where includes and excludes can be combined to filter out most of the noise while including a specific list?

[pyrossh]

Yep you are right this library is mainly used to improve dx. Maybe if we move the priority of exclude below that of the include it might work. I agree something like .gitignore rules might be better. Or maybe a .embedignore file.
Another solution is add a glob pattern which supports regex like most cli's offer.

[alerque]

Lowering the priority of exclude= would certainly work for my use case. Honestly I'm more than a bit confused about what the include= is even supposed to do given that folder= already iterates over everything on the disk, so you basically start out with include=* and then work from there. Given that starting point I assorted uses of exclude= make sense, but what can include=... even do?

One other note is about performance: the current compile times don't seem to be related to how many files or how much data gets embeded, but they are very related to how big the file system in include= is, and it seems to be exponential. If you have a big file system it doesn't matter if you exclude=* or exclude all but a few files, the resulting compile times are quite slow.

[pyrossh]

@alerque Does this PR #244 improve compile times in anyway?

[alerque]

Sorry it took me a bit to spin up the project with this in it without my other workarounds to avoid this. I dug up some code from about the time I reported this and the warm build time for Cargo with 0.8 was running about 64 seconds. Switching to a Git dependency, I'm seeing warm builds between 30-40 seconds. I'm not sure why it now varies so much, but it seems clear it's at least somewhere close to halved my build times.

It's a little harder to get a grip on how much of my total build time is related to embeding, but disabling all the various rust-embeded brings warm build times down to about 8 seconds. There is a bit of work building an asset list and a little bit more prepping some of them for embeding, but it seems like there might be a bit of overhead still involved in handling a large list of includes.