Should I choose all flash nas?

[leonfox28]

Hi guys,
My group is trying to build a new NAS for following needs:
1. 10GBE network(maybe BCM57412 for RDMA);
2. iscsi for esxi to support about 10 to 15 people working on there own virtual desktop;
3. nfs for those virtual machines;
4. For our daily work, it's mostly will be coding, PCB layout, simulation and etc. They are almost all small files so maybe the random 4K performance is most important?
We've decided to choose TrueNas Scale for host OS because our limited budget but still not sure what we should choose for storage devices: All U.2 NVMe SSD or HDD data vdev + SSD special vdev for metadata or even HDD only is good enough? If we choose the second choice, should we add slog devices?

Thanks a lot!

 

[Ericloewe]

10GBE network(maybe BCM57412 for RDMA)

Broadcom NICs are pretty crappy, and RDMA is a non-starter because it's not supported. It's a lot of effort for not that much gain, in most realistic systems. Intel or Chelsio are the way to go.

iscsi for esxi to support about 10 to 15 people working on there own virtual desktop;

How much actual storage?

3. nfs for those virtual machines;

You mean for standard network file shares to be used from the (virtual) workstations? Assuming you do the usual thing and all relevant data gets stored this way, why do you want to separate out the storage that's backing the actual VMs? It sounds like it would just be a standard image you deploy, and thus little benefit would come from having TrueNAS back those volumes, when you could use storage local to the VM host.

 

[leonfox28]

Broadcom NICs are pretty crappy, and RDMA is a non-starter because it's not supported. It's a lot of effort for not that much gain, in most realistic systems. Intel or Chelsio are the way to go.

Thanks, got it

How much actual storage?

50T~100T

You mean for standard network file shares to be used from the (virtual) workstations? Assuming you do the usual thing and all relevant data gets stored this way, why do you want to separate out the storage that's backing the actual VMs? It sounds like it would just be a standard image you deploy, and thus little benefit would come from having TrueNAS back those volumes, when you could use storage local to the VM host.

Yes, but we also have a small lsf cluster and some other machines, they need to access same files.

 

[Ericloewe]

Yes, but we also have a small lsf cluster and some other machines, they need to access same files.

Sure, but that's the file sharing part, that's easy. It's the block storage part, backing the VMs' virtual disks, that would benefit little from being outside the VM host.

 

[leonfox28]

Sure, but that's the file sharing part, that's easy. It's the block storage part, backing the VMs' virtual disks, that would benefit little from being outside the VM host.

Understand what you mean, yes you are right. We handled both /home and app installation directory on the NAS through NFS and the VMs are just quick clone from the template, so it maybe ok if the esxi local storage blow up. But put them together on NAS maybe better for managent?

 

[Ericloewe]

It is, but it drives up the cost for equivalent performance. You could just keep the VM template backed up on the NAS and deploy as needed to host-local storage.