Horrible ZFS performance no matter the config

[lmiksche]

Hi, I have an issue with data transfers.
I tried striped mirrors, 8 drive Z2, 5 drive Z1 and for some reason im unable to get passed 200MBps on read and write over 10Gb link. When its in RAM cache i get up to 500MB/s, but once it reads or writes to the drive its in region of 160-200Mb/s no mater the config and on top of that it uses like 70% of CPU where assigned 8cores / 16 threads on PROXMOX. Any help? I dont know what to try anymore. It should be getting 6-700MBps on read!

 

[sretalla]

How are you connecting the drives? what's your hardware? (forum rules in red at the top here) how are you testing to confirm your throughput rate? (lots of small files or small number of big ones?)

 

[lmiksche]

1. 8x DELL (Hitachi) SAS2 drives on PERC H310 mini mono in PowerEdge R720
2. Large files as that is continuous read.

 

[Ericloewe]

Have you crossflashed the controller? The RAID firmware is rather sucky.

 

[lmiksche]

No idea what u r talking about. I flashed the IT firmware in. That's the only thing.

 

[Ericloewe]

That's a yes, then.

How exactly are you doing file transfers? SMB? NFS?

 

[Samuel Tai]

Did you pass the controller through to the TrueNAS VM in ProxMox? Or are you passing individual drives? What resources have you allocated to the VM in the host? Also, what's the HW configuration of the host?

 

[lmiksche]

Controller passthrough - "1. 8x DELL (Hitachi) SAS2 drives on PERC H310 mini mono in PowerEdge R720" as already stated,
tried different configs for testing from 16-64GB RAM with 8-12cores 16-24threads.
Tried SMB and FTP.
But not sure its the protocols as if the file is cashed in RAM then I´m getting reasonable 550MB/s. Still not full 10Gbit, but at least half.
I even tried to dedicate and passthrough the 10Gbit intel PCIe NIC, situation unchanged. Getting really desperados here.
Is it possible to do raid 6 on H710p and just give it the RAID as single drive? As it didnt like my H310 before firmware flash to the extend the TRUENAS didnt even start even install hang up half way thru booting.

 

[G8One2]

Pretty sure you will only get the speed of the slowest disk in any 1 vdev. You need multiple vdevs to increase your IOPS. I currently have 18 disks, in 3 vdevs on a 10Gb network and only get around 650-700Mbps read/write across my LAN. I will need to add another vdev to get closer to saturating my 10Gb network. Im not expert here, so someone could correct me if I'm wrong here, but its to my understanding, that's how it works.

 

[lmiksche]

Thanks for the info. If anyone else can offer any more explanation, please do so else,
this is not normal. My arm based NAS does close to 500MB/s with 4 drives in RAID5. In that case i guess they can stick the whole zfs... im installing PERC 710p and running RAID6.

 

[HoneyBadger]

No idea what u r talking about. I flashed the IT firmware in. That's the only thing.

Important distinction here. Did you flash the Dell PERC H310 IT mode firmware, or follow the more complicated "cross-flash" process to load the vanilla LSI/Broadcom IT firmware? The Mini/Mono cards have a special process that @fohdeesha has documented kindly in the Resources section.