Vdev Layout

[Fastline]

You might want to read some fundamentals on ZFS ...

ZFS Primer

Background information about the Zettabyte File System (ZFS).

www.truenas.com

Interesting.

But i just don't understand even when i try to use NVMe like PC801, the speed is same as my 5xRAID Z1 single vdev. I even added 2xNVMe and created a separate pool in striped to have the maximum performance, the speed is same as my 5xRAID Z1 single vdev. I just don't get it. Should i test with 2 vdevs of 1 NVMe each and then see the results?

The NVMe is SK Hynix PC801 512GB. Even if i put it in PCIe 3.0 slot, it should do 1Gb/s as it can go up to 3GB/s in bare metal. I do get slow HDD so slow write and the connection saturates but when having an NVMe pool only, either single disk or two in stripe, it should do 1Gb/s isn't it? Or am i missing something here?

 

[Patrick M. Hausen]

But i just don't understand even when i try to use NVMe like PC801, the speed is same as my 5xRAID Z1 single vdev.

Are these measured values or assumptions? What disk devices is the 5-wide RAIDZ1 built from?

What amount of data are you writing or expecting to write continuously? Most SSDs meet the advertised write rate only for small bursts, then they hit their internal cache size and writing drops to much lower rates ...

 

[Fastline]

Are these measured values or assumptions? What disk devices is the 5-wide RAIDZ1 built from?

Measured in real world environment and have tested it multiple times. Those are 5x16TB Seagate Exos, those have like 400-600MB/s speed after 10GB file is copied and then no matter even if i use another pool with NVMe either single disk or multiple.

 

[Patrick M. Hausen]

So copying to that RAIDZ1 spinning disk pool gives you 400-600 MB (bytes? bits?) per second and performing the same operation to a different all NVMe pool gives the same speed?

What is the speed of your network? What type of network interface cards is in use on both TrueNAS and the clients? Have you measured the network througput?

Similarly have you tried creating a dataset without compression and then measuring the pool speed on the TrueNAS host with bonnie++, thus eliminating potential network bottlenecks?