Separate cache hash length and folder depth

[ronaldbarendse]

Currently, the length of the created cache hash can be configured on the middleware options:

ImageSharp.Web/src/ImageSharp.Web/Middleware/ImageSharpMiddlewareOptions.cs

Lines 87 to 91 in a411d0e

	/// <summary>
	/// Gets or sets the length of the filename to use (minus the extension) when storing
	/// images in the image cache. Defaults to 12 characters.
	/// </summary>
	public uint CachedNameLength { get; set; } = 12;

And this is used as a parameter on ICacheHash.Create() (so it makes sense that this isn't a setting specific to the ICacheHash implementation and a more general one defined in ImageSharpMiddlewareOptions):

ImageSharp.Web/src/ImageSharp.Web/Middleware/ImageSharpMiddleware.cs

Line 264 in c84f4ba

string key = this.cacheHash.Create(uri, this.options.CachedNameLength);

But this setting is also used by PhysicalFileSystemCache to specify the depth of the nested cache folders:

ImageSharp.Web/src/ImageSharp.Web/Caching/PhysicalFileSystemCache.cs

Line 83 in af70a40

this.cachedNameLength = (int)this.options.CachedNameLength;

So if you currently want to increase the cache hash length (e.g. to reduce collisions), it will also increase the cache folder depth. And because this is linked, it will only ever have a single image and metadata file inside a folder. I recon the folder nesting is done to prevent enumerating large amount of files in a single directory, but the additional nested folders might actually reduce I/O performance (because each folder still requires something to be written into the MFT record).

I would suggest the following changes:

• Change setting name from CachedNameLength to CacheHashLength, since is a more suitable name for where it's actually used and the IImageCache is otherwise responsible for creating the final file name;
• Rename the default CacheHash implementation to SHA256CacheHash, to make it more explicit how the hash is generated;
• Add a new CacheFolderDepth setting to PhysicalFileSystemCacheOptions, so this can be configured separately;
• Optionally: because the nested folders already contain the first n characters from the file name, these can be stripped from the final name, e.g. 027ee1e815be.meta (12 characters) could be stored as 0/2/7/e/e/1/e/8/15be.meta (with a depth of 8), instead of 0/2/7/e/e/1/e/8/027ee1e815be.meta. This decreases the path length, so you're also less likely to hit maximum path length limits...

I've already created a branch with the first 3 changes, which I can submit as PR if you'd like.

[JimBobSquarePants]

Dude. I’m not changing the cache.

Apologies... I need to stop reading these things first thing in the morning, I'm always grumpy.

If we make these changes we need to ensure that the default behavior is backwards compatible, we've got users out there with millions of cached files. I've already made changes to #206 to ensure that users can turn on the old behavior and keep their existing cache structure.

We have to do some sanity checking also to ensure that the depth cannot be greater than the hash.

Optionally: because the nested folders already contain the first n characters from the file name, these can be stripped from the final name, e.g. 027ee1e815be.meta (12 characters) could be stored as 0/2/7/e/e/1/e/8/15be.meta (with a depth of 8), instead of 0/2/7/e/e/1/e/8/027ee1e815be.meta. This decreases the path length, so you're also less likely to hit maximum path length limits...

This is an interesting suggestion. We'd have to think up a decent property name in the cache options -perhaps TrimNestedCharacters? (suggestions welcome) and default it to false, again for backwards compatibility. We'd also have to ensure that trimming leading characters is limited so that at least one char remains.

For information as why nesting occurs. Here's a feast of information on NTSF file/folder performance.
https://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories

[ronaldbarendse]

We don't need an additional property though I need to re-read more as well 😇 Yes, we still need something to make this backwards compatible: I suggest adding a setting like UseFullName or UseLegacyName on PhysicalFileSystemCacheOptions.

[ronaldbarendse]

I've created the PR: #212

[ronaldbarendse]

And thanks for the link on NTSF file/folder performance 👍🏻 If we're using a default cache hash length of 12 and cache depth of 8, the resulting file name length would be 6 characters, so you would also solve any performance issues with short-file name generation (because the first 6 characters are always unique) 🎉

[ronaldbarendse]

I stand by my original statement: we don't need an additional property/setting: #213 😈

[hey-red]

It's also a nice option for linux based OS, because most of distributions uses ext4 as default filesystem.
So, in default installations we have limited number of inodes and cache takes one inode for every created directory. This can cause exhaustion of the pool of inodes and your application has stopped working.

As I know, with ext4 we cannot resize inode table(only way to recreate volume and specify number of inodes).