Feature request: branched cache structure

[shawnkhall]

Currently QC stores all of it's cache files within the same folder (usually "/wp-content/cache/"). For a large site, multisite network or so on, this results in a very large number of cache files within a single folder.

Most file systems have issues with storing too many objects/links within a single folder. NTFS, for example, starts experiencing noticeable delays after about 3000 files exist in one folder. While you can safely store over 4b in one directory, performance to locate, read or write a file suffers when it exceeds a certain threshold. Similarly, EXT3 and EXT4 suffer from noticeable delays after there are about 7000 files in a single folder, and EXT3 effectively stops responding once you hit about 20k.

Other caching systems provide a branched structure to address this problem. QC should take advantage of their example and build a tree based on the first character from salt values. On one of my sites, this would result in folders like these (based on the last few minutes of cache files created for this site):
/wp/content/cache/0/5/5
/wp/content/cache/0/7/7
/wp/content/cache/0/e/2
/wp/content/cache/0/f/5
/wp/content/cache/1/4/1
/wp/content/cache/1/4/8
/wp/content/cache/1/4/f
/wp/content/cache/1/8/e
/wp/content/cache/2/4/7
/wp/content/cache/2/9/5
/wp/content/cache/2/e/8
/wp/content/cache/3/2/5
/wp/content/cache/4/4/4
/wp/content/cache/4/4/6
/wp/content/cache/4/7/8
/wp/content/cache/5/3/5
/wp/content/cache/5/8/c
/wp/content/cache/5/9/5
/wp/content/cache/5/a/f
/wp/content/cache/6/3/8
/wp/content/cache/7/c/5
/wp/content/cache/8/3/5
/wp/content/cache/8/8/2
/wp/content/cache/8/e/8
/wp/content/cache/9/2/2
/wp/content/cache/9/5/5
/wp/content/cache/9/9/5
/wp/content/cache/b/0/5
/wp/content/cache/b/5/5
/wp/content/cache/b/5/8
/wp/content/cache/b/9/5
/wp/content/cache/b/b/5
/wp/content/cache/c/3/5
/wp/content/cache/d/1/5
/wp/content/cache/d/a/5
/wp/content/cache/d/c/5
/wp/content/cache/e/0/2
/wp/content/cache/e/6/6
/wp/content/cache/e/9/5
/wp/content/cache/e/a/5
/wp/content/cache/e/b/5
/wp/content/cache/e/e/5
/wp/content/cache/e/f/2
/wp/content/cache/f/0/5
/wp/content/cache/f/1/b
/wp/content/cache/f/3/8
/wp/content/cache/f/7/1
/wp/content/cache/f/8/5
This splits the potentially 750k cache files from this one site into an average of approximately 200 per folder, with the improved file seek that accompanies fewer files in a given folder. Even if one parameter tended to be duplicated often (during this quick check approximately 70% of the third parameters were duplicates), it would still keep the number of files in any given folder below 3000.

This would require you to build the 4096 folders (16x16x16) within the cache tree, but that's a one-off, or can be compiled dynamically the same way you build the files. In any case, the performance would be improved on large sites by caching this way.

[jaswrks]

Agreed. I will take a closer look at this before the next maintenance release. Thank you.

[shawnkhall]

Awesome! This has been my only serious complaint about QC since I started using it years ago.

[jaswrks]

Thanks, we're looking this over again; and @raamdev is joining me in the development of this plugin also. He'll be reviewing this also in the near future; and perhaps he and I will come up with something that works together with existing functionality while still breaking things down into more sub-directories.

@raamdev Quick Cache currently uses glob() quite extensively, so calls to glob() would need to be modified in consideration of this new directory/file structure.

[jaswrks]

Sorry, had a few typos in that message that I corrected.

[bridgeport]

Could you please also consider a matching folder name directory structure in addition to what @shawnkhall is suggesting? So, if a site has the following pages:

example.com/
example.com/blog/
example.com/blog/hello-world/

The cache structure would be like:
/wp-content/cache/index.html
/wp-content/cache/blog/index.html
/wp-content/cache/blog/hello-world/index.html

This makes it easy to manually delete a cache(s) file or directory via FTP (if necessary) or PHP script, directly targeting directories for deletion or examination outside the Quick Cache internal operations. I'd be great to have a cache structure that's browsable instead of one that's obfuscated.

Thanks.

[raamdev]

@bridgeport Actually, I like the idea of using a named-branch structure as you suggested (WP Super Cache uses a named-branch structure as well). It's far more readable and much easier to navigate.

@shawnkhall Do you have any opposition to a named-branch structure over a salt-based structure (e.g., /wp/content/cache/0/f/5)? It seems like for large sites, including multisite networks (which would benefit by having each site's cache stored in a directory for that site, e.g., cache/childsite1.example.com/), the named-branch structure would work just fine. But you may have more experience with large sites than me, so I'd love your input if you have any. :)

[raamdev]

Reopening this issue until the new branched cache structure code is actually released. The code is in the development trunk. I'll close this again when I release this feature.

[raamdev]

My apologies for the noise. I'm closing this issue because it's now being tracked by the Next Release milestone.