FreeNAS to ESXI, using ISCSI. All over the place transfer rate. Help please. All Info in post.

[nafeasonto]

First and foremost:

• Motherboard make and model - ASUS Z10pA-U8 LGA2011
• CPU make and model - Intel Xeon E5 2650 V3
• RAM quantity - 60GB DDR4 ECC RAM
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives - HP Seagate 4TB SATA 7.2K 6Gb/s 3.5" Hard Drive ST4000NM0033 - BRAND NEW. RAIDZ2
• Hard disk controllers -2x LSI SAS9220-8i - flashed to IT mode.
• Network cards - 2x 10GB Network card for ISCSI, then the normal gigbits on Motherboard, for FreeNAS interaction

I have a 56GB Optane Memory, PCIE as a ZLOG device. My writes are set to standard ASYC and not ALWAYS on the pool.
I have a 250GB SAMSUNG 850 EVO as one cahce device on the pool.
Every single DRIVE has been SMART tested, all passed with flying colors.
All OSES on my VMs are Windows.

When I do transfers from a drive on the POOL to another drive on the pool, the transfer rate goes from 700MB a sec, down to 50MB a sec, in 5 seconds. Then fluctatues anywhere from 50-100, down to 50, to to 20, t hen to 50, then to 200, then to 50.

It's ALL over the place. Can someone tell me what's going on, or help me with this. Any suggestions please tell me. If I am an idiot, please tell me.

 

[Herr_Merlin]

First why async writes.
Second how high is the utilization of the pool?
Are you using dedup?
Why the fuck are you running async for block level storage?
Why is the Evo set up as l2arc?
How many VMs?
How many drives?
Why no mirror setup as you need random prefomance? ( I know I ran a Z2 (21 drives I think..but split into 4 drives vdevs) for VMs as well, but that are 10k 1.8TB drives with except the optane cache way faster plus the system has 192GB ram and faster CPUs... And my constant prefomance isn't much higher than 1Gbs..)

 

[UdoB]

• RAIDZ2

... on the pool, the transfer rate goes from 700MB a sec, down to 50MB a sec, in 5 seconds.

Well, RaidZ2 gives you the performance of a single drive. (As you have only a single vdev this is valid for the pool.)

Reading *and* writing 50 MB/s means 100 MB/s traffic for that pseudo-drive.

Read and write happens on different locations in the drive, so head movement is relevant. In my opinion the performance you get is absolutely expected...

Best regards

 

[nafeasonto]

First why async writes.
Second how high is the utilization of the pool?
Are you using dedup?
Why the **** are you running async for block level storage?
Why is the Evo set up as l2arc?
How many VMs?
How many drives?
Why no mirror setup as you need random prefomance? ( I know I ran a Z2 (21 drives I think..but split into 4 drives vdevs) for VMs as well, but that are 10k 1.8TB drives with except the optane cache way faster plus the system has 192GB ram and faster CPUs... And my constant prefomance isn't much higher than 1Gbs..)

I don't appreciate your attitude first off. Second off my CPU never even breaks a sweat.

 

[nafeasonto]

Well, RaidZ2 gives you the performance of a single drive. (As you have only a single vdev this is valid for the pool.)

Reading *and* writing 50 MB/s means 100 MB/s traffic for that pseudo-drive.

Read and write happens on different locations in the drive, so head movement is relevant. In my opinion the performance you get is absolutely expected...

Best regards

I did not realize that is how RAIDZ worked.

 

[Herr_Merlin]

I don't appreciate your attitude first off. Second off my CPU never even breaks a sweat.

still you did not answer on how many disk, vms, etc...
That would be nice to know.
Than as mentioned by UdoB RAIDZ works the way as it works so sequal speed equals all disks IOPS equal a single disk.
So how did you build your RAIDZ2? For example 1x 10 drives as RAIDZ2 or 2x 5 drives as RAIDZ2, with the second one you would get the IOPS of 2 disks.. still a mirror might be a better setup for your choice. As mirrors scale with IOPS.

 

[nafeasonto]

still you did not answer on how many disk, vms, etc...
That would be nice to know.
Than as mentioned by UdoB RAIDZ works the way as it works so sequal speed equals all disks IOPS equal a single disk.
So how did you build your RAIDZ2? For example 1x 10 drives as RAIDZ2 or 2x 5 drives as RAIDZ2, with the second one you would get the IOPS of 2 disks.. still a mirror might be a better setup for your choice. As mirrors scale with IOPS.

Well, RaidZ2 gives you the performance of a single drive. (As you have only a single vdev this is valid for the pool.)

Reading *and* writing 50 MB/s means 100 MB/s traffic for that pseudo-drive.

Read and write happens on different locations in the drive, so head movement is relevant. In my opinion the performance you get is absolutely expected...

Best regards

12 Drives. In RAIDZ2. I did tell you the amount of drives, in the first post.

Second of all to the second quote, that makes no sense. As on my DAS direct RAID 6 arrays, with 6GB interface, i get a VERY stable 250-300MB a sec with no drop. Same drives, same size. Same amount.

There is something else wrong. I'd rather go with a standard RAID 6 array, then a RAIDZ2, if it only uses one drive for write ability.

 

[Herr_Merlin]

There are huge differences between RAIDZ2 and RAID6. ZFS offers a lot more than standard RAID. Standard RAID functions different and thus your RAID6 performance might be correct.

Please read the following: https://www.ixsystems.com/blog/a-co...are-design-part-i-purpose-and-best-practices/
And please also check: https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
RAIDZ: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

This should give you ideas how to setup etc.

 

[nafeasonto]

There are huge differences between RAIDZ2 and RAID6. ZFS offers a lot more than standard RAID. Standard RAID functions different and thus your RAID6 performance might be correct.

Please read the following: https://www.ixsystems.com/blog/a-co...are-design-part-i-purpose-and-best-practices/
And please also check: https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
RAIDZ: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

This should give you ideas how to setup etc.

So what I basically read is that iSCSI works like crap in ZFS.

 

[Herr_Merlin]

no and yes depends on your business needs and how you set it up.
Plus you can't compare a RAIDZ2 with a RAID6.. RAID Z offers a lot more and works different than a RAID5 or RAID6