Improving transfer speed for small files for MinIO/S3

[KirklandIceCream]

I have a few hundred million (~400M) small files (1MB or less). I have been using rclone to transfer the files from my PC to the MinIO instance. I have a 10Gbps network (NIC and switches). However, whenever I try to transfer more than 1 at a time (in rclone settings), the MinIO instance will eventually crash. TrueNAS will report a disk has faulted, and the pool is degraded. The whole system will be become unreachable. The only solution will be to physical force restart, then it clears the fault and starts as if nothing happened. This process will repeat itself if I try to copy more than 1 file simultaneously.

I am imagining that this occurs because it's hitting some IOPS limit, or just writing more than the disks can handle? Is there any solutions on how I can move more simultaneously? The current transfer rate is about 1 MB/s, and at ~400M files, that's literally a decade...

My current setup is:

• 1 vdev at 6 disks (RAIDZ1) 7200 RPM.
• 192GB RAM.
• Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz CPU.

 

[NugentS]

You seem to be missing a motherboard, PSU and a mechanism for connecting the drives to the motherboard. Not sure how you are copying any files anywhere to be honest!!!