R720 & TrueNAS Core 12 ESXi 7.0 optimizing

[lucky644]

I have a R720 /w a IT flashed H710 Mini
64GB RAM
6 x 14TB Seagate EXOS
Booting TrueNAS Core 12 via USB
2 x 10GBE NIC

I currently have ZFS2 running, single pool, iSCSI formatted for ESXi 7.0 and I see speeds maxing my 10GBE for write and about 500-600MB/s read.

When I originally setup my NAS, I was in a bit of a rush, but since then I've decided to take a closer look and remake the NAS to something a bit more appropriate.

• I have the ability to expand it to a 8 drive 14TB array, but then I wouldn't have the option of using Raid 0 /w 2x SSD for cache. (should I even bother using cache with 64gb ram?)
• I also have 256gb, 512gb and 1tb NVMe's available using a PCIe adapter, should I use these for cache instead? Or something else?
• Should I use NFS or iSCSI for the ESXi shared storage? I've heard that data loss can occur due to iSCSI not confirming writes for speed?
• Is ZFS2 appropriate for 6 or 8 drives? I have more than enough space I think, capacity really isn't a issue as much as performance.

I've read through a bunch of posts but there's so much info out there I couldn't find a comprehensive 'best practice' setup for a ESXi shared storage setup. I don't really have any performance issues but I probably don't have the most optimal setup either. When I got a couple more 14tb drives I figured I should rebuild things and do it properly this time.

 

[Samuel Tai]

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

 

[HoneyBadger]

Welcome to the forums.

I have a R720 /w a IT flashed H710 Mini
64GB RAM
6 x 14TB Seagate EXOS
Booting TrueNAS Core 12 via USB
2 x 10GBE NIC

R720 is a good platform in general, although slightly limited in the drive bay count as you're finding out.

I currently have ZFS2 running, single pool, iSCSI formatted for ESXi 7.0 and I see speeds maxing my 10GBE for write and about 500-600MB/s read.

The reason you're maxing out your writes and not your reads is because it's presently dumping your writes into RAM. iSCSI is written asynchronously from ESXi by default.

See this resource for details: https://www.truenas.com/community/r...-esxi-nfs-so-slow-and-why-is-iscsi-faster.40/

When I originally setup my NAS, I was in a bit of a rush, but since then I've decided to take a closer look and remake the NAS to something a bit more appropriate.

• I have the ability to expand it to a 8 drive 14TB array, but then I wouldn't have the option of using Raid 0 /w 2x SSD for cache. (should I even bother using cache with 64gb ram?)
• I also have 256gb, 512gb and 1tb NVMe's available using a PCIe adapter, should I use these for cache instead? Or something else?
• Should I use NFS or iSCSI for the ESXi shared storage? I've heard that data loss can occur due to iSCSI not confirming writes for speed?
• Is ZFS2 appropriate for 6 or 8 drives? I have more than enough space I think, capacity really isn't a issue as much as performance.

Responding in list order:

8x14TB in mirrors is better. Don't use "RAID 0" - if you meant "RAID 10" as an analogue to the ZFS mirrors, then yes, this is what you want.

"Should I even bother using cache?"
Read cache, possibly - but max out your regular RAM first. 64GB is considered "a good place to start" when serving up heavy VMFS workloads.
Write cache - well, ZFS doesn't have one, but you probably want a write log which is a different thing. See the next line.

"I have 256/512/1T NVMe available"
Please post the make and model of these SSDs. Consumer SSDs are often drastically unsuitable for use as a write-log device, often referred to as "SLOG" which is what's necessary to give you the combination of both "fast" and "safe" writes. Generally speaking if they aren't "enterprise" or "datacenter" they will fall over in a hurry under write-log duties.

"NFS or iSCSI?"
I prefer iSCSI due to the better VAAI primitive support. With the general availability of 10GbE the speeds are roughly identical. The "iSCSI can lose your data" comment - see the "Why is NFS so slow?" resource above as to what that's talking about, but it's down to what the defaults are on the VMware side, in that they expect a SAN device to provide safe writes without being asked.

"Is ZFS2 appropriate?"
Assuming you mean RAIDZ2, it's fine for the drive count but not appropriate for your performance goals, especially for VMFS. Stick to mirrors as in the articles linked by @Samuel Tai

I've read through a bunch of posts but there's so much info out there I couldn't find a comprehensive 'best practice' setup for a ESXi shared storage setup. I don't really have any performance issues but I probably don't have the most optimal setup either. When I got a couple more 14tb drives I figured I should rebuild things and do it properly this time.

I'd suggest a read of the resources above, especially re: sync writes, so that you can understand the "why" behind some of these recommendations. But for a TL;DR on best practices

1. Get as much RAM as you can afford (seriously, max out the chassis)
2. You will want a fast write-log device, look at Optane DC cards, there are other options such as NVRAM too, most consumer cards don't cut it here
3. Use mirrors instead of RAIDZ
4. Set sync=always on your iSCSI ZVOLs unless you like data loss, NFS is sync by default
5. In a related note, RAID isn't a backup - have a recovery plan!
6. iSCSI vs NFS is mostly up to you - iSCSI has multipath support through MPIO a lot more easily than parallel NFS - change your path-selection policies in ESXi to VMW_PSP_RR and iops=1 for the cycle rate
7. If you're using iSCSI, make sparse ZVOLs (don't oversubscribe your pool though!) and use VMFS6 to get automatic space reclamation in-guest through TRIM/UNMAP

That's a "good place to start" - apologies for the length of this whole post as even the TL;DR is long.