dump.exfat should do more than just disk/volume information

[hgkamath]

dump.exfat tool should do more than just disk/volume information.

The advantages of exfat filesystem are that (A) it is simple, (B) has huge maxsize limit and (C) has a bit for indicating contiguous-ness allowing for read optimization (D) absence of journaling is better for lifespan of SSD.
The simplicity is a trade off against not having a journal and disk failure tolerance, robustness.

A use case of exfat filesystem is to store large static container files like virtual-disk images, that are seldom resized/moved. They may also be defragmented in the case of hdd (not ssd) as contiguous files.
More importantly, knowing where the used file blocks are helps in case of disaster recovery.

dump.exfat /dev/sdx|/mnt/exfat <arguments> . Following arguments are being suggested
-s in formatted text show only information from superblock(s)
-u in formatted text show comma separated (or line separated) offset ranges of used sectors in increasing order and the file to which the sector range belongs to specified in bytes/blocks/clusters.
-f <filepath> in formatted text show comma separated (or line separated) offset ranges of bytes/blocks/clusters associated with a file
-R <dirpath> in formatted text recursively descend a directory show pathname and comma separated (or line separated) offset ranges of bytes/blocks/clusters associated with contained files (this functionality may also be simulated by using a shell script to loop over output of find command and use -f argument on each file)
-o <bkup_tableflname> in a binary stream output/dump the FAT sectors to a offline backup file

The dump information saved as a periodical (nightly) backup ahead of any disaster, so that in the event disaster happens, it may be used for disaster recovery.
The formatted text dump information will be helpful while using a command like dd during disaster recovery .

dd if=/bad/partition_blockdev of=/good/fs/opath/frag1 bs=512 offset=off1 count=len1 
dd if=/bad/partition_blockdev of=/good/fs/opath/frag2 bs=512 offset=off2 count=len2  
:
dd if=/bad/partition_blockdev of=/good/fs/opath/fragN bs=512 offset=offN count=lenN
dd if=/bad/partition_blockdev of=/good/fs/opath/fragNb bs=1 offset=$((512*(offN+lenN))) count=lenbytes  
cd /good/fs/opath/
cat frag1 frag2 ... fragN fragNb > reconstructed_file

One may then append the fragments / concatenate them together to recover the original file on a new filesystem
Such metadata can be saved on a different disk/partition and will be helpful during disaster recovery.

The following also requires a option parameter feature addition to the mount -t exfat command
Saving a binary dump of the enitire FAT would be used the following way.
dump.exfat /dev/sdx|/mnt/exfat -o <bkup_tableflname>
This could then be used in conjunction with mount.exfat
mount.exfat /dev/sdx /mnt/tmpexfat -o ro,fat=<bkup_tableflname>
This allows using regular cp/dd/ddrescue/filecopy commands to copy data off the corrupted FAT/partition using the offline backup table.

[namjaejeon]

Okay, I can not understand fully what you want to do. Can you implement it and show the execution log and usage at real ?

[hgkamath]

Essentially, dump tool for kernel-driver based exfatprogs should give information similar to the fuse-driver based exfat-utils
Please compare
https://manpages.debian.org/experimental/exfatprogs/dump.exfat.8.en.html
https://manpages.debian.org/unstable/exfat-utils/dumpexfat.8.en.html
Unfortunately, I do not presently code.
Currently, on my system with 2Tb drives, I have many small 100G partitions. I am fearful of file-system crashes. Small partitions can be repaired using fsck in reasonable, lesser amount of time. Corruption in one partition does not affect filesystem in other partitions. The problem with this partitioning approach is the disk space is badly utilized. As the partition fills up, the total free-space of the disk is fragmented within a paritition and between partitions. The rigid space-limits of each partition makes it difficult create large virtual-disks and move them around.
I was investigating the idea of combining the partitions back to one large exfat partition, and put Virtual-disk-images that are dynamic sized and have max size 100G. The virtual disks have their own file-systems on them, which can be mounted. Usually the disks are much smaller than the max size. Hence, the combined free-space is larger and allows for easier manipulation of large virtual-disks.
In this regard, I considered how I would do disaster recovery. If the large exfat filesystem experienced degradation/corruption, then dump information saved previously from tools like dump.exfat may help.

One should be able recover a file by knowing the every file's FAT-CHAIN of bytes/sectors/clusters.

misc:

• hdparm --fibmap </path/to/file> slow; brute forces its way through the file to determine the file's absolute sectors on disk.
• nfi.exe - Windows NT File System (NTFS) File Sector Information Utility
• ntfscluster /dev/sda17 -F /test1.img
• fsutil file layout path\to\file
• contig -a
• filefrag -v /opt/tmp/test1.bin notice that filefrag for btrfs is faster and more informative than exfat

[namjaejeon]

fsck is ready on exfat-next branch. Please check whether it is covered for your request.

git clone --branch=exfat-next https://github.com/exfatprogs/exfatprogs

[hgkamath]

check whether it is covered for your request.

Not really, the RFE/issue was regarding for precaution-sake being able to get the file-fragment info dumps, in case of needing to do manual-scraping/recovery before even attempting fs-repair. Online fs-repair tools irreversibly modify on-disk data. As a user, one can only say "Y" to whatever confirmation fsck asks. The fsck may attempt to rebuild FAT as well find/reallocate/relocate sectors. When data in volume is too important, one dd-s the whole corrupted volume into a volume-image on larger fresh disk, makes a duplicate copy, and attempts fs-repair on the copy.
Pre-saved FAT file-fragment info dumps is one contingency option. ExFAT being a fs that is descendant of FAT technology, does not have the same corruption robustness as other large filesystems (ntfs, btrfs, bcachefs, etc). This contingency option is useful for an ExFAT volume, which has a few large changing virtual-disk-files that rarely every grow or change-size, and so for long periods of time, the virtual-disk-files can be expected to always occupy the same sector-locations on host-exfat-volume, unless defragged/moved/copied etc. When a virtual machine writes to the virtual-disk-file, it is expected to only overwrite already-allocated sectors. This option allows for manual sector by sector disk-scraping as was done on a bad harddisk in the old days.

Nevertheless it is good to know exfat fsck is ready. Good-work and thankyou for your contributions!

As of now, my filesystems/hdds are fine, and I don't have any corruption to test against, and I hope they remain that way.

[root@fedora gana]# rpm -qa | grep exfat
exfatprogs-1.1.3-2.fc36.x86_64

koji builds for fc37 are at exfatprogs-1.1.3.
https://koji.fedoraproject.org/koji/packageinfo?packageID=32895

[namjaejeon]

@hgkamath We have released exfat 1.2.0 version. Please check it.