ipfs add -r <dir> (298GB average<10MB) spends over 3 days

[hoogw]

ipfs add -r

289GB( average file size < 10MB)
after add 70GB, speed noticeable slowing down,
spend 3 days to reach 200GB,

CPU and memory usage is under 20%, bottleneck is not CPU and RAM.

I have not try --pin=false

ipfs add (by default --pin = true)

To turn off pin to speed up,

D:\test>ipfs add --pin=false IMG_1427.jpg
4.18 MiB / 4.18 MiB [========================================================================================] 100.00%�added QmekTFtiQqrhiqms8FXZqPD1TfMc9kQUoNF8WVUNBGJF8h IMG_1427.jpg
4.18 MiB / 4.18 MiB [========================================================================================] 100.00%
D:\test>

Can you advise how to speed up?

[bonedaddy]

Can you give your system specs (RPM of hard drive, CPU, Memory) and IPFS version?

[hoogw]

@postables CPU and memory, disk usage way low from full.
Bottleneck is not CPU and RAM, RPM.

[Stebalien]

@hoogw could you this without running the daemon? That is, turn off the daemon and then run ipfs add -r .... That will rule out some potential issues.

[hoogw]

@Stebalien ok, I will try and let you know test result.

[hoogw]

@Stebalien Yes. if turn off daemon, I can see visible, noticeble, speed up, a lot faster.
Looks like --pin=false does not helps, but turn off daemon will definitely speed up and back to normal speed it should be.

[hoogw]

If turn off daemon, with full speed, the CPU usage is only 19%, very low, memory is only 15MB, with FULL speed. Disk read/sec 9MB/s , Disk write/sec 1MB/s. It is way low than the computer potential. Is there any other bottle neck to stop it run full computing potential?

[Stebalien]

I can see visible, noticeble, speed up, a lot faster.
Looks like --pin=false does not helps, but turn off daemon will definitely speed up and back to normal speed it should be.

Yeah, this looks like sending out provider records is slowing things down (probably). We're working on that in #5870.

It is way low than the computer potential. Is there any other bottle neck to stop it run full computing potential?

How many cores do you have? We don't currently parallelize hashing on add. We're probably also doing a bunch of small writes (assuming you're using the default datastore type) so we're context switches (syscalls) are probably costing us a lot of performance.