Unable to run '--verify_only' workload on a read-only filesystem even though this is a read-only workload

[GavinPLucas]

Please acknowledge the following before creating a ticket

• I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:
Verify_only workloads are, by definition, read only. However fio throws an error "error=Read-only file system" when running a verify_only workload on a read-only filesystem.

When the '--verify_only' flag is used, fio should ignore the checks for a read-only filesystem, even if '--rw=write' is used, since this workload will not actually perform any write operations, just read and verify the data that would have been written by this workload.

I may be wrong, but I'm guessing that it's still trying to open the file read/write, the same as the original workload, even though this is actually a read only workload.

The point of trying to do this is that I'm attempting to verify data written by a previous fio job on a read-only snapshot of the filesystem. Unfortunately this isn't possible due to this issue.

Environment: Debian 11.8

fio version: fio-3.36-19-gbdf99

Reproduction steps

1. Run an fio workload that creates a file on a test filesystem. In this case an EXT4 filesystem mounted on /mnt:

glucas@debian:~/src/fio$ ./fio --name=fiotest --filename=/mnt/fiotest --bs=4k --offset=0 --size=2G --ioengine=libaio --iodepth=32 --direct=1 --fallocate=none --rw=write --refill_buffers=1 --verify=crc32c --do_verify=0
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.36-19-gbdf99
Starting 1 process
fiotest: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=83.6MiB/s][w=21.4k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=24378: Wed Dec  6 11:30:10 2023
  write: IOPS=27.2k, BW=106MiB/s (112MB/s)(2048MiB/19258msec); 0 zone resets
    slat (usec): min=22, max=15227, avg=35.78, stdev=69.20
    clat (usec): min=3, max=25422, avg=1139.02, stdev=590.69
     lat (usec): min=29, max=25471, avg=1174.80, stdev=602.65
    clat percentiles (usec):
     |  1.00th=[  906],  5.00th=[  971], 10.00th=[  988], 20.00th=[ 1012],
     | 30.00th=[ 1029], 40.00th=[ 1045], 50.00th=[ 1057], 60.00th=[ 1090],
     | 70.00th=[ 1139], 80.00th=[ 1205], 90.00th=[ 1254], 95.00th=[ 1319],
     | 99.00th=[ 1926], 99.50th=[ 4359], 99.90th=[10290], 99.95th=[15008],
     | 99.99th=[19792]
   bw (  KiB/s): min=84136, max=117000, per=100.00%, avg=109384.63, stdev=6866.87, samples=38
   iops        : min=21034, max=29250, avg=27346.16, stdev=1716.72, samples=38
  lat (usec)   : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=15.54%
  lat (msec)   : 2=83.52%, 4=0.40%, 10=0.43%, 20=0.10%, 50=0.01%
  cpu          : usr=5.31%, sys=25.91%, ctx=524519, majf=0, minf=22
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=2048MiB (2147MB), run=19258-19258msec

Disk stats (read/write):
  nvme0n2: ios=15/522027, sectors=120/4176512, merge=0/37, ticks=0/14674, in_queue=14674, util=99.54%

2. remount the filesystem read-only:

glucas@debian:~/src/fio$ sudo mount /mnt -o remount,ro

3. Run the previous job, but replace the '--do_verify=0' with '--verify_only':

EXPECTED OUTCOME:
The fio job should read and verify the contents of the file that was created by the original workload.

ACTUTAL OUTCOME:
fio throws 'error=Read-only file system' even though this is, by definition, a read-only job:

glucas@debian:~/src/fio$ ./fio --name=fiotest --filename=/mnt/fiotest --bs=4k --offset=0 --size=2G --ioengine=libaio --iodepth=32 --direct=1 --fallocate=none --rw=write --refill_buffers=1 --verify=crc32c --verify_only
fiotest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.36-19-gbdf99
Starting 1 process
fio: pid=24385, err=30/file:filesetup.c:794, func=open(/mnt/fiotest), error=Read-only file system


Run status group 0 (all jobs):

Disk stats (read/write):
  nvme0n2: ios=0/0, sectors=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

[axboe]

You still set --rw=write for the verify portion...

[GavinPLucas]

Yes. Which is the correct thing to do.

The point of '--verify_only' is that you pass all the options of the job that created the data and it verifies that what was written matches what would have been written by that job.

If I changed that to --rw=read then it would be thinking that I was trying to verify the output of a job which is all reads, i.e. nothing, which would make no sense to do.

From the man page:

verify_only=bool
Do not perform the specified workload, only verify data still matches previous invocation of this workload. This option allows one to check data multiple times at a later date without overwriting it. This option makes sense only for workloads that write data, and does not support workloads with the time_based option set.

[vincentkfu]

It seems reasonable to omit the O_RDWR flag for a verify_only workload. Does the patch below resolve your issue?

# git diff
diff --git a/filesetup.c b/filesetup.c
index 816d1081..bf040cb1 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -746,7 +746,7 @@ int generic_open_file(struct thread_data *td, struct fio_file *f)

 open_again:
        if (td_write(td)) {
-               if (!read_only)
+               if (!read_only && !td->o.verify_only)
                        flags |= O_RDWR;

                if (f->filetype == FIO_TYPE_FILE && td->o.allow_create)

[axboe]

If readonly is set, then we should be safe and never open the device for read/write at all. Just as a safe guard in case there's a bug that does make it issue a write. I think it'd be saner to clear this differently, eg if verify-only is set, then clear any write flags in data direction?

[vincentkfu]

If read only is set then the condition will fail and the RDWR flag won't get set (which is the same behavior as before).

The change just makes it so that RDWR gets set only if both readonly iand verifyonly are not set.

[vincentkfu]

The above patch did take some effort to convince myself that it does the right thing. And your suggestion is easier to think about. So I will go with your suggestion.

[axboe]

Ah yeah I'm blind, it does do the right thing!

[vincentkfu]

@GavinPLucas Can you try the patch below?

diff --git a/filesetup.c b/filesetup.c
index 816d1081..86e5f937 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -749,6 +749,9 @@ open_again:
                if (!read_only)
                        flags |= O_RDWR;

+               if (td->o.verify_only)
+                       flags &= ~O_RDWR;
+
                if (f->filetype == FIO_TYPE_FILE && td->o.allow_create)
                        flags |= O_CREAT;

Also available at https://github.com/vincentkfu/fio/commits/1681/

[GavinPLucas]

@vincentkfu Many thanks! I'll give it a go as soon as I can and get back to you.

[GavinPLucas]

@vincentkfu I've had one of my team re-test with a build I made that includes the patch, and I can confirm that it works for us. Many thanks!