TrueNAS CORE - High End Design

[Simon Sparks]

The Hardware Specification is as follows:

DELL PowerEdge R730xd - 24 x 2.5" Bay Chassis with Optional 2 x Rear 2.5" Bays.

3 x DELL PowerEdge 12Gbps SAS Host Bus Adapter - External Dual Port PCI-e
1 x DELL PowerEdge 12Gbps SAS Host Bus Adapter - Internal Mini Mono
1 x DELL PowerEdge Broadcom 57840S 10GbE Quad Port SFP+ Network Daughter Card
1 x DELL PowerEdge QLogic QLE2694-SR Quad Port 16Gbps FC Host Bus Adapter
1 x DELL PowerEdge iDRAC Enterprise Module with 16GB vFlash Card
1 x DELL PowerEdge Dual SD Card Module with 2 x 16GB SanDisk Extreme Plus SD Card ( for running TrueNAS CORE on )
1 x DELL PowerEdge PCIe NVMe Adapter Card
2 x Intel(R) Xeon(R) CPU E5-2683 v4 - 16 Hypter Threaded Cores @ 2.10GHz CPUs ( Willing to Replace if there is a better option e.g. less cores and higher speed )
16 x Micron 32GB 2Rx4 PC4-2400T DDR4 PC4-19200 ECC Server Memory MTA36ASF4G72PZ-2G3B1 = 512GB

Information

I have built a High End TrueNAS CORE system to serve an extensive home lab used by several VMware IT Consultants who all work from home.

The storage solution will only be used as a storage solution for iSCSI, Fibre Channel, SMB/CIFS and NFS.

I am planning to enable deduplication on the 4 x 7.86TB NVMe SSDs For Virtual Machines
I am planning to enable deduplication on the 22 x 480GB SATA SSDs For Virtual Machines
I am planning to enable deduplication on the 24 x 1TB SATA SSDs For Virtual Machines

Storage Installed

DELL PowerEdge R730xd - 4 x 7.86TB NVMe SSDs For Virtual Machines
DELL PowerEdge R730xd - 22 x 480GB SATA SSDs For Virtual Machines
DELL PowerVault MD 1420 - 24 x 1TB SATA SSDs For Virtual Machines
DELL PowerVault MD 1420 - 24 x 5TB SATA HDDs For SMB/CIFS or NFS File Sharing
DELL PowerVault MD 1400 - 12 x 16TB SATA HDDs For Veeam Backups

Here are the questions:

1. Should I swap the CPUs for different ones?
2. Are there any Tunables that I should configure to improve the performance.

 

[Dice]

1 x DELL PowerEdge iDRAC Enterprise Module with 16GB vFlash Card

I'd love to see 2x SATA drives for OS.
That SDcard would be the equivalent of putting fake plastic hub caps on a Lambo.
(Makes it all look stupid as well as may fall of during actual use ;) )

2 x Intel(R) Xeon(R) CPU E5-2683 v4 - 16 Hypter Threaded Cores @ 2.10GHz CPUs ( Willing to Replace if there is a better option e.g. less cores and higher speed )

1. You're on the track there, particularly SMB is single core speed dependent.
If possible due to finances (ie, maybe you can only spend once and need it to be right?), I'd not stress about the CPUs until putting the system to use and finding out what load really is put on, particularly through SMB.

At this point there you've not dipped into pool configurations. At a high caliber build, there are a some things that needs to be considered.
What are your thoughts at this point?

 

[jgreco]

1 x DELL PowerEdge Broadcom 57840S 10GbE Quad Port SFP+ Network Daughter Card

The Broadcom cards tend to be problematic. You might want to pick a card from the 10 Gig Networking Primer available in Resources. I've been reasonably happy with the Intel X710-DA4 or variations such as the Supermicro AOC-STG-i4s if you need a half height.

1 x DELL PowerEdge QLogic QLE2694-SR Quad Port 16Gbps FC Host Bus Adapter

Be aware that you're sort of on your own here. Are you planning to use this for target operations from VMware hosts? FC isn't officially supported but it also isn't intentionally hobbled. If you know that you can make this work, well, great. It is STRONGLY DISRECOMMENDED for you to use FC to attach disk shelves for backend ZFS storage. Just want to make sure you hear those words so you don't end up with a costly misadventure.

2 x Intel(R) Xeon(R) CPU E5-2683 v4 - 16 Hypter Threaded Cores @ 2.10GHz CPUs ( Willing to Replace if there is a better option e.g. less cores and higher speed )

Dedup is CPU intensive and it isn't clear to me where the best balance would be. Higher speed lower core count would probably be the classic E5-2643 v4 (6 cores/3.4GHz) that I normally recommend for singlethreaded task optimization, but for dedup you might find some other intermediate CPU better.

Be certain you've read the Resources section article by Stilez on dedup. Expect it to inform some changes in your platform design.

Be certain to read my article on design for block storage.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

2. Are there any Tunables that I should configure to improve the performance.

See my article on 10G network tuning.

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[Simon Sparks]

The Broadcom cards tend to be problematic. You might want to pick a card from the 10 Gig Networking Primer available in Resources. I've been reasonably happy with the Intel X710-DA4 or variations such as the Supermicro AOC-STG-i4s if you need a half height.



Be aware that you're sort of on your own here. Are you planning to use this for target operations from VMware hosts? FC isn't officially supported but it also isn't intentionally hobbled. If you know that you can make this work, well, great. It is STRONGLY DISRECOMMENDED for you to use FC to attach disk shelves for backend ZFS storage. Just want to make sure you hear those words so you don't end up with a costly misadventure.



Dedup is CPU intensive and it isn't clear to me where the best balance would be. Higher speed lower core count would probably be the classic E5-2643 v4 (6 cores/3.4GHz) that I normally recommend for singlethreaded task optimization, but for dedup you might find some other intermediate CPU better.

Be certain you've read the Resources section article by Stilez on dedup. Expect it to inform some changes in your platform design.

Be certain to read my article on design for block storage.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

See my article on 10G network tuning.

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

Hi jgreco thanks for your reply, the FC ports will be in target mode and the ESXi hosts will connect via FC purely for LAB purposes.

I have done it before see this post of mine from FreeNAS 9.3, it worked on all FreeNAS and TrueNAS releases since then also https://www.truenas.com/community/t...-target-mode-das-san.27653/page-7#post-394237

Thanks for the 10Gb tuning that was interesting reading.

 

[firesyde424]

We have exactly this kind of setup, using Dell PE R720xd, 730xd and 740xd servers, and Dell MD3060e and MD3260 JBODs. Some suggestions based on your build:

1 x DELL PowerEdge 12Gbps SAS Host Bus Adapter - Internal Mini Mono

I'm going to assume this is the HBA330. There's a mini and PCIE version. It is effectively a Dell rebrand of the LSI 9300-8i, with Dell firmware. You can flash this to the standard LSI firmware, but you'll lose a considerable amount of integration with the server's iDRAC and it's not necessary. The HBA330 in both versions, runs perfectly with TrueNAS Core and Scale, out of the box.

1 x DELL PowerEdge Broadcom 57840S 10GbE Quad Port SFP+ Network Daughter Card

We have had all kinds of hell over the years with Broadcom network cards, in TrueNAS applications. In our case, we stick to Intel cards and make use of the Intel X520, X710, and X810 for 10Gbe ports, and the Intel XXV710 for 25Gbe ports. Even the TrueNAS enterprise stuff we have uses either Intel or Chelsio as opposed to Broadcom. I would suggest a different card from the TrueNAS hardware guide.

2 x Intel(R) Xeon(R) CPU E5-2683 v4 - 16 Hypter Threaded Cores @ 2.10GHz CPUs ( Willing to Replace if there is a better option e.g. less cores and higher speed )

You are fairly close to the ceiling in terms of Broadwell CPUs, which is the highest model of Xeon the R730xd will support. 32 cores total should be more than enough for most but, depending on your workloads and IO, this may not be enough for deduplication. I can't stress enough how much deduplication can drag a server into the ground in terms of performance. If you absolutely must have it here, I'd suggest a change to the E5-2697v4 which is the same number of cores as the E5-2683v4, but another 500Mhz base clock, per core.

Hi @jgreco thanks for your reply, the FC ports will be in target mode and the ESXi hosts will connect via FC purely for LAB purposes.

If the FC cards are for for the ESXi hosts to connect to this server, I'd suggest just adding additional 10Gbe NICs or, if your switching supports it, 25Gbe. It's far easier to set up NFS shares in TrueNAS as well as ESXi, never mind possible compatibility and additional complexity.

3 x DELL PowerEdge 12Gbps SAS Host Bus Adapter - External Dual Port PCI-e
1 x DELL PowerEdge 12Gbps SAS Host Bus Adapter - Internal Mini Mono
1 x DELL PowerEdge Broadcom 57840S 10GbE Quad Port SFP+ Network Daughter Card
1 x DELL PowerEdge QLogic QLE2694-SR Quad Port 16Gbps FC Host Bus Adapter

This is a lot of PCIE cards, in addition to the NVME adapter card I saw in your build, and the four NVME drives. I can't remember exactly, but think you are only getting 80 or 96 lanes total with two CPUs. You'll want the latest BIOS for the R730xd(DO NOT INSTALL 2.13 or 2.14). Make sure your chassis configuration supports this many cards, not counting the mini card that installs on the motherboard.

1 x DELL PowerEdge Dual SD Card Module with 2 x 16GB SanDisk Extreme Plus SD Card ( for running TrueNAS CORE on )

We've had so many of these fail over the years. The write endurance on most SD cards isn't the best. The PE R730xd has mother board SATA ports that will accept a SATA DOM. You can then mirror these and then you'll not only have far better reliability over the long term, but they will perform far better than SD cards.

 

[HoneyBadger]

Hello Simon!

Several other users have provided some very good advice with regards to configuration needed to run block storage with high performance, as well as notes regarding the added pressure from inline deduplication. The resource from Stilez mentioned by @jgreco is linked below:

My experiments in building a home server capable of handling fast + consistent deduplication

AIM: To help people looking at deduplication on TrueNAS 12+, what I've found on the way making it work on mine. On sustained mixed loads, such as 50GB+ file copies and multiple transfers, using TrueNAS 12 with a deduped pool and default config...

www.truenas.com

Note that this user is using a mix of file storage, VMs, and backup workloads. Deduplication against pure VM workloads, especially those that generate random I/O and do "in-place updates" is likely to be significantly more impactful on system resources versus deduplication of files that are read and modified as wholes, or backup jobs that represent only moderately changing or growing datasets. If you have an existing FreeNAS or TrueNAS setup where this data resides, you might want to perform a dedup simulation using the following command:

(Warning - this will take a long amount of time, and cause a good amount of disk I/O!)

zdb -SSS -U /data/zfs/zpool.cache YourPoolName

This will give you a simulated histogram and a summary result line similar to:

dedup = 1.13, compress = 1.46, copies = 1.06, dedup * compress / copies = 1.55

For the sample pool I analyzed, dedup only gives 1.13x space savings - not worth it, especially when compression alone gives me 1.46x.

Where VMware is in play, separate log devices (SLOG) will likely be beneficial even on all-SSD pools, in order to get optimized write patterns to your data vdevs; even SSDs prefer to be written to in large chunks. Check in the SLOG benchmarking thread - but given your proposed connectivity of multiple 10GbE interfaces and/or FC (manually configured) you're well into the point where you'll be wanting to look at higher-end devices such as Intel Optane DC series or Radian's RMS-series NVRAM cards.

I'd also like to call attention to one potential pain point:

DELL PowerVault MD 1420 - 24 x 5TB SATA HDDs For SMB/CIFS or NFS File Sharing

These drives are very likely Shingled Magnetic Recording (SMR) - you may experience poor performance especially under sustained write pressure. Do you have an exact model number? I am suspecting Seagate ST5000LM000 - which is SMR.

 

[Dice]

If you have an existing FreeNAS or TrueNAS setup where this data resides, you might want to perform a dedup simulation using the following command:

(Warning - this will take a long amount of time, and cause a good amount of disk I/O!)

zdb -SSS -U /data/zfs/zpool.cache YourPoolName

This will give you a simulated histogram and a summary result line similar to:

dedup = 1.13, compress = 1.46, copies = 1.06, dedup * compress / copies = 1.55

For the sample pool I analyzed, dedup only gives 1.13x space savings - not worth it, especially when compression alone gives me 1.46x.

This would make a lovely resource!

 

[pricemc1]

One suggestion I would add is to not go bare-metal for this. Instead load ESXi on the server and create a single VM on the server for TrueNAS. Don't use the server for ANY other VMs (unless you want to impact performance). Allocate essentially all the server's memory and CPU capacity to the TrueNAS VM. In ESXi, passthrough any hardware you need TrueNas to have direct access to (usually your HBAs and NVMe drives) . This gives you several advantages:

1. Your TrueNas system is now portable. By that I mean that the TrueNas VM itself can potentially be moved to another system as needed. Should you decide to replace the physical server in the future, simply shutdown the VM, remove the pass-throughs from it, and copy it over to a new ESXi host using standard VM copy procedures in vSphere. Move any PCIe-based storage, HBAs, and/or external storage enclosures over to the new system and re-add the pass-throughs and your done (assuming your port groups match between hosts).

2. Using ESXi under TrueNAS eliminates most compatibility issues that people have mentioned (i.e. Broadcom cards) on these forums. ESXi works great with the Broadcom adapters. TrueNas has no issues with the VMware VMXNET-3 paravirtual and SCSI adapters that you would use on the TrueNAS VM. If you virtualize TrueNAS then its not directly exposed to the problematic Broadcom hardware so compatibility no longer is a concern and you still get to use the hardware you have. You would build port groups in vSphere for your required TrueNas networks and add then them as different virtual NICs to your TrueNAS VM. This allows you to use the Broadcoms to back TrueNAS networking without invoking any compatibility issues. It also keeps the networking configuration very simple from a TrueNas perspective.

3. ESXi has an extremely low overhead. You're not going to see any performance impact to TrueNas from using it under the TrueNAS VM. You're looking at a maximum 1-2% tax of system resources (CPU/RAM) by running ESXi underneath your TrueNAS VM. To me, this is a very small penalty to pay for the benefits you get from running a virtualized TrueNas compared to the hassles of running bare metal. Of course if you start running other VMs on the same system then that's a different story but my recommendation is to only run that single TrueNAS VM on the host.

I'm sure I can come up with some other benefits as well but my brain is a bit fried today. Now let me play devil's advocate for a minute though...

One could also argue that inserting ESXi also adds an additional layer of undesired management tasks because assuming you patch your environment you now have to patch ESXi and TrueNAS both and that would result in more down time potentially. To me though this is far outweighed by the benefits gained from virtualization. ESXi only takes a few minutes to patch typically so it only will add a very short amount of time to any maintenance window you might have to plan for patching.

The other argument against this approach might be costs (if you care about licensing). You can always run the free version of ESXi to offset that issue. In the overall scheme of things if you have an application that is requiring you to dedicate an entire server to it then its likely that the ESXi licensing costs aren't going to amount to a very large part of the overall system cost anyway because the hardware and application costs are going to be most of the costs.

Lastly...

IMHO there is almost NO reason to run anything (i.e. databases, VOIP servers, software defined storage) bare-metal now a days. Even if you have an extremely resource/performance intensive application you'll always benefit from using the model of putting ESXi under it rather than going bare metal.

Before anyone flames me consider the following. I know alot of people on here will disagree with me about running TrueNas on ESXi but then how many of them have been doing virtualization for almost 20 years like I have? Many of the people on these forums know a good bit about storage but most of them aren't virtualization experts (yes, some are... I just said MOST). I've been doing VMware ESXi / vSphere since 2.x days (~15 years). I've also done other hypervisors such as Hyper-V, Xen, Virtual Iron, and KVM so I have a pretty good idea of what I'm talking about when it comes to the topic of server virtualization. In addition, I have an extensive background in storage that goes back even further than my virtualization experience. I've worked for EMC in the past doing professional services that combined virtualization and storage platform solutions for their customers. I've also worked with most of the major storage vendor's products in my time (i.e. NetApp, HPE, Dell (non-EMC), etc...). I currently do storage engineering and architecture as my day job for large federal enterprise environments. Admitedly, I'm not much of a pro when it comes to open-source storage products so if you want to beat me up about something in storage that's the place to do it... There are a lot of statements out there about why you can't use virtualization with resource/performance intensive systems but the reality is that the kind of things people say either come from ignorance with virtualization or experience using hypervisors back many years ago before they were as fully mature as they are now.

Regards...
pricemc1

 

[rvassar]

I suspect the OP is trying to utilize the network daughter card format, center slot in the back next to the iDRAC board, and keep the PCIe slots for HBA's, etc... There is an Intel option for the R730xd, I think it's dp/n 68M95, but you'll have to do some homework. The xd's are kind of odd ducks. Keep your fingers out of the fans...

On the ESXi idea... If I remember correctly, ESXi 6.7u3 is now EOGS and 7.0 might not like the CPU on a 13G.

 

[pricemc1]

I suspect the OP is trying to utilize the network daughter card format, center slot in the back next to the iDRAC board, and keep the PCIe slots for HBA's, etc... There is an Intel option for the R730xd, I think it's dp/n 68M95, but you'll have to do some homework. The xd's are kind of odd ducks. Keep your fingers out of the fans...

On the ESXi idea... If I remember correctly, ESXi 6.7u3 is now EOGS and 7.0 might not like the CPU on a 13G.

I run ESXi 8 on 13G R630 and R730 servers. Works great. They give you a warning that they may drop support for Broadwell at any future version during install but that it doesn't prevent you from installing and running it.

For R730XD usage with TrueNas the best configuration is to use the flex-zone option to split the storage backplane and add a second controller. I have an HBA330 I assign to 20 of the drive bays and then a H730 that gets 4 of the drive bays. The HBA330 and 20 drives gets passed through to the TrueNas VM. The H730 is just there to provide hardware raid for ESXi itself. Only down side is you can't hook up the 2 rear flex-bays if you go with this configuration.

 

[Ericloewe]

1. Your TrueNas system is now portable. By that I mean that the TrueNas VM itself can potentially be moved to another system as needed. Should you decide to replace the physical server in the future, simply shutdown the VM, remove the pass-throughs from it, and copy it over to a new ESXi host using standard VM copy procedures in vSphere. Move any PCIe-based storage, HBAs, and/or external storage enclosures over to the new system and re-add the pass-throughs and your done (assuming your port groups match between hosts).

How is that any better than moving the install media or reinstalling to new media and uploading the config file? Sure, it's a meaningful improvement in some situations (I have the misfortune of having to deal with some of them), but TrueNAS is thankfully not one of those scenarios...

2. Using ESXi under TrueNAS eliminates most compatibility issues that people have mentioned (i.e. Broadcom cards) on these forums. ESXi works great with the Broadcom adapters. TrueNas has no issues with the VMware VMXNET-3 paravirtual and SCSI adapters that you would use on the TrueNAS VM. If you virtualize TrueNAS then its not directly exposed to the problematic Broadcom hardware so compatibility no longer is a concern and you still get to use the hardware you have. You would build port groups in vSphere for your required TrueNas networks and add then them as different virtual NICs to your TrueNAS VM. This allows you to use the Broadcoms to back TrueNAS networking without invoking any compatibility issues. It also keeps the networking configuration very simple from a TrueNas perspective.

That seems like swatting a fly with a flamethrower. Sure, VMs represent a lot less overhead than they used to... but there still is some overhead.

IMHO there is almost NO reason to run anything (i.e. databases, VOIP servers, software defined storage) bare-metal now a days. Even if you have an extremely resource/performance intensive application you'll always benefit from using the model of putting ESXi under it rather than going bare metal.

I'm not saying VMs are bad or anything, but that's a bit hyperbolic...

There is an Intel option for the R730xd, I think it's dp/n 68M95

There are several, in fact:

1. I350 for quad 1GBase-T - 0R1XFC
2. I350+X520 for dual 1GBase-T and dual SFP+ - 0C63DV
3. I350+X540 for dual 1GBase-T and dual 10GBase-T - 099GTM
4. I350+X710 for dual 1GBase-T and dual SFP+ - 06VDPG
5. X710 for quad SFP+ - 068M95
6. X550 for quad 10GBase-T - 064PJ8

Some of these have additional alternative part numbers, but these seem to be the most popular ones. There's also Broadcom, the cheapskate option; as well a Mellanox ComnnectX-4 with two SFP28 ports for 25 GBE, but no 1 GBase-T. Not enough PCIe lanes on the connector, I guess, which must be why they moved to OCP NIC 2.0 and 3.0 on Gen 15 servers.

 

[Etorix]

I know alot of people on here will disagree with me about running TrueNas on ESXi but then how many of them have been doing virtualization for almost 20 years like I have?

Well, that's the point here… Without that twenty years of virtualisation experience of yours, one is certainly safer sticking with bare metal.
Especially if the whole point of the exercise is to run a grand total of ONE VM on the server, taking all the resources (dedup on all pools ) or to make an exotic NIC "work anyway" (if one can afford 0.5 TB RAM, over 50 TB of SSD storage and over 200 TB of HDD storage, what's the extra cost of picking a recommended NIC over the Broadcom thing?)