All NVMe build ?

[atakacs]

Hello

Is there any advice / best practice setting up an all NVMe build ? We are mostly interested in speed vs capacity (we don't need more than 8Tb but we want the best speed possible & obviously reliability).

I have reviewed the various FAQ / Guides but they seem to be mostly oriented toward HDD based setups.

 

[Patrick M. Hausen]

Mirrored pairs if you want as many IOPS as the drives can handle. See e.g. the specs of the hypervisor systems in my signature.

 

[atakacs]

thanks - interesting.
Do you really need that many cores ?

 

[morganL]

To build a cost-effective system, it's best to start with a realistic set of goals with respect to Capacity, IOPS and Bandwidth. How are the clients connected and what do they need?

If you want "max performance", its easy to end up building a Ferrari and then complaining about the cost when there is a speed limit on the highway anyway.

 

[Patrick M. Hausen]

Do you really need that many cores?

The job of this system is virtualisation, so, yes, that's the point. We have VMs with up to 16 cores.

 

[atakacs]

The job of this system is virtualisation, so, yes, that's the point. We have VMs with up to 16 cores.

Oh ok - In my case we would not run VM on that server - just storage.

 

[Patrick M. Hausen]

In that case the most important question is "what kind of storage?".
If this is going to be a file server you can go with RAIDZn and lower your cost. If it is going to be block storage for a separate hypervisor system, you should pick mirrors.

 

[atakacs]

Backend storage for hypervisor (ESX). At the moment we would have 2 hosts - FC if we can find a reasonable solution, 10gb Ethernet otherwise.

 

[Patrick M. Hausen]

In that case - here's your article by @jgreco:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.ixsystems.com

 

[morganL]

2 host, 100VMs each, typically requires less than 10,000 IOPS

That can be built with Mirrors or even a RAIDZ1 vdev. using NVME SSDs.

The formula would be different with 10 hosts.

 

[atakacs]

The formula would be different with 10 hosts.

Sorry not sure to understand your implication

 

[morganL]

Sorry not sure to understand your implication

With more hosts, you may need more IOPS and hence Mirrors rather than RAIDZ1 might be preferred. Pools are designed for a performance target.