Why is the filter set not consistent with expectations?

[zzl221000]

When using -F on Windows, I want to package folders and files whose paths start with "2024-07". I set the following rules:
-F '+ 2024-07*/**/*.log' -F '- *.log' -F '- *'
However, when testing with --debug-filter included-files, I found that all files were included in the packaging.

[mhx]

Hi and welcome!

I'm surprised that you're saying "all files were included". I have a potential explanation for why all files would be excluded, but that doesn't seem to be your problem.

Let's start with the basics:

• Which version of mkdwarfs are you using? There was a bug in filter rules processing prior to the latest release, but I don't think this is affecting your use case.

• Can you share the output of mkdwarfs with --debug-filter included-files, or even better just --debug-filter?

[zzl221000]

My bad. I have found the issue:

1. The problem with all files existing should be related to the + *.log issue; I apologize again for the mistake in the previous message.
2. Previously, without the + *.log, no files would be outputted; it's still an issue with the path separator, which should consistently use "/" for dwarfs.

+ \\?\Q:\dwarfs_test\server_data\data/2024-06-01/
+ \\?\Q:\dwarfs_test\server_data\data/2024-07-01/
+ \\?\Q:\dwarfs_test\server_data\data/2024-06-01/somedir/
+ \\?\Q:\dwarfs_test\server_data\data/2024-06-01/somedir/somesubdir/
- \\?\Q:\dwarfs_test\server_data\data/2024-06-01/somedir/somesubdir/exclude.log
+ \\?\Q:\dwarfs_test\server_data\data/2024-07-01/somedir/
+ \\?\Q:\dwarfs_test\server_data\data/2024-07-01/somedir/somesubdir/
+ \\?\Q:\dwarfs_test\server_data\data/2024-07-01/somedir/somesubdir/include.log

PM
     ___                  ___ ___
    |   \__ __ ____ _ _ _| __/ __|         Deduplicating Warp-speed
    | |) \ V  V / _` | '_| _|\__ \      Advanced Read-only File System
    |___/ \_/\_/\__,_|_| |_| |___/         by Marcus Holland-Moritz

mkdwarfs (v0.9.10 [2024-05-30])
built for AMD64, Windows-10.0.22631, MSVC 19.39.33523.0

using: FLAC++-1.4.2, boost-1.82.0, brotlidec-1.0.9, brotlienc-1.0.9,
       crypto-3.1.1, fmt-10.0.0, lz4-1.9.4, lzma-5.4.3, xxhash-0.8.1, zstd-1.5.5

Usage: mkdwarfs [OPTIONS...]

Working very well, thank you, I really like dwarfs.

[mhx]

Glad you got it working!

it's still an issue with the path separator, which should consistently use "/" for dwarfs

Yeah, I agree this is not ideal on Windows; I'll consider changing this, but I need to think about whether or not this causes other issues.

One more thing: while your filter rules work, I'd probably have chosen a different set of rules:

-F '+ /2024-07*/'
-F '- /*/'
-F '+ *.log'
-F '- *'

• The first rule includes only the top-level directories matching 2024-07*.
• The second excludes all other top-level directories. This has the advantage that any files below these directories are not even processed any further. If you have a long list of directories, potentially on a network share, this can be significantly faster.
• The next two rules simply include all *.log files and exclude everything else.

It's a different approach and both work fine.

[zzl221000]

Thanks a lot, it seems like the filter rules you've laid out are really hitting the mark for me.

Right now, I'm stuck with a bit of a snag. I've got all these files that were zipped up in 7z, and I'm looking to switch them over to dwarfs. Do you know if there's a slick way to pull that off?

[mhx]

I guess you'd want something like what was discussed already in #181, just for 7z instead of plain tar. This is going to be really hard.

Unfortunately, I don't have a convenient solution. Your best bet is to just extract the original files and then create a DwarFS image.

There are a few tools out there that apparently can mount 7z archives as volumes in Windows, similar to what DwarFS does. I haven't tried any of them, though. If you can use these tools to mount your archives, you should be able to create a DwarFS image from these mounts without the need to unpack all files.

[mhx]

it's still an issue with the path separator, which should consistently use "/" for dwarfs

Yeah, I agree this is not ideal on Windows; I'll consider changing this, but I need to think about whether or not this causes other issues.

FWIW, dwarfs-0.10.0 now consistently uses /.