Hybrid 64 GB RAM + SSD plotting - C30 under 5 minutes

[miwojcik]

Hi! Here is my idea how can we utilize our setups with 64GB of RAM to produce C30 compressed plots quicker and reduce SSD wear.

My setup

OS: Debian 12
HW: Ryzen 5950x, RTX 3070 Ti, 64GB/3200MHz RAM, 2 x 1TB Corsair MP510 NVMe SSDs, small SSD for OS

Problem

Because of B-550I mini-ITX board and case I cannot easily upgrade my ram to 128GB. While my SSDs are quite fast for my needs, during plotting they quickly run out cache space or overheat and slow down. While placing additional fan to cool them helped a little I wanted to put my RAM to use.

Solution

After experimenting with tmpfs + swap, loop devices placed in tmpfs combined with SSD by LVM etc. I think I found the best candidate - linux brd module allows to create block device directly in ram, that we can later use in mdadm RAID0 with SSD.

How to:

The commands below assume that the system will be used only for plotting. If you need it for other things, you may need to decrease the size of ram disk and increase the size of NVMe partition. Their sizes don't have to match - mdadm will use all available space. For C30 compressionIn you should end up with size reported by mdadm -D /dev/md0 that's at least: Array Size : 102305792 (97.57 GiB 104.76 GB). Instructions can be also tuned for setups with 32GB of RAM by reducing the RAM disk to 22-23GB, but YMMV.

1. Create RAM disk
   modprobe brd rd_size=55500000 max_part=1 rd_nr=1
   55500000 is the size in KB
2. Partition your fast SSD (it will wipe all data on it)
   parted /dev/nvmeXnX mklabel gpt
parted -a optimal /dev/nvmeXnX mkpart primary 0% 48000MB
3. Create mdadm array
   mdadm --create --verbose /dev/md0 -l 0 -n 2 /dev/ram0 /dev/nvmeXnXp1
4. Partition your slow SSD (it will wipe all data on it)
   parted /dev/nvmeYnY mklabel gpt
parted -a optimal /dev/nvmeYnY mkpart primary 0% 400GB
   If you use only one SSD for plotting, create second partition on it instead.
5. Create ext4 filesystems without safety features
   for ramdisk raid:
   mkfs.ext4 -O ^has_journal,bigalloc -T largefile4 -m 0 /dev/md0
   for slow SSD:
   mkfs.ext4 -O ^has_journal,bigalloc -T largefile4 -m 0 /dev/nvmeYnY
6. Mount
   mount -o noatime,barrier=0 /dev/md0 ~/tmp_fast/
mount -o noatime,barrier=0 /dev/nvmeYnY ~/tmp_slow/
7. Plot!
   ./cuda_plot_k32_v3 -C 30 -2 ./tmp_slow/ -3 ./tmp_fast/ -t ./tmp_slow/ -d @REMOTE -c $POOL_CONTRACT -f $FARMER_KEY
   -t defaults to current dir, but setting it the same as -2 prevents moving the plots again when they are done

Removing the ramdisk

Before shutdown or to start from scratch again run the following:
umount /dev/md0
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/ram0 /dev/nvmeXnXpX
rmmod brd

Results

Data written: 240GB to fast SSD, 193GB to slow SSD

[P1] Setup took 0.688 sec
[P1] Table 1 took 11.753 sec, 4294967296 entries, 0 GB/s up, 3.06313 GB/s down
[P1] Table 2 took 22.494 sec, 4294908510 entries, 1.4226 GB/s up, 3.00084 GB/s down
[P1] Table 3 took 28.775 sec, 4294617958 entries, 2.08511 GB/s up, 3.90968 GB/s down
[P1] Table 4 took 36.304 sec, 4294075051 entries, 2.75429 GB/s up, 4.09049 GB/s down
[P1] Table 5 took 60.459 sec, 4293094568 entries, 2.18284 GB/s up, 2.82838 GB/s down
[P1] Table 6 took 27.616 sec, 4291107899 entries, 2.75083 GB/s up, 3.86334 GB/s down
[P1] Table 7 took 22.217 sec, 4287055786 entries, 3.77749 GB/s up, 2.83571 GB/s down
Phase 1 took 210.933 sec, 4287055786 proofs (0.998158)
Phase 2 took 9.107 sec, 4.82255 GB/s up, 0.111535 GB/s down
[P3] Setup took 0.104 sec
Phase 3 took 32.34 sec, 2.75131 GB/s up, 0.863632 GB/s down
[P4] Setup took 0.121 sec
Phase 4 took 20.247 sec, 2.76075 GB/s up, 1.14535 GB/s down
Total plot creation time was 272.936 sec (4.54893 min)

I hope that helps somebody! If so, please share your results :)

[trxman]

Any info what should be min temp space for C29 ?

[miwojcik]

I built two C29s and the highest usage of tmp_fast reported by df -BM was: 98625M leaving 1017M free - so pretty much the same.
However, I observed that after few hours of C30 plotting tmp_fast usage occasionally increased, so you may need to use nvme partition bigger by one or two GBs.

[miwojcik]

I added instructions how to remove the ramdisk and start from scratch if you want to tweak it.