SOLVED New build planned, feedback welcome!

NobleKangaroo

Dabbler
Joined
Apr 18, 2017
Messages
18
Looking to make a TrueNAS server to replace the current server. Plans are as follows:

R730xd 2.5" SFF
CPU:
2x E5-2690v4 or 2x E5-2640v3(4x or 2x, respectively, more compute power than current NAS)
RAM: 256GB ECC DDR4 - 8x 32GB 2Rx4 PC4-2400T-R
HBA: H730P or HBA330
NICs: 2x MCX312A-XCBT 10Gbps dual SFP+ (4 links in LACP) (I've used these for years and they're solid)
Disks:
- Front plane: 24x 1.2TB 10k 2.5 Samsung SAS (fast VM storage)
- Mid plane: 4x 18TB 5400 3.5 Ironwolf Pro SATA (slow NAS and media storage)
- Rear bays: 2x 200GB 2.5 Kingston SSD (mirrored boot drives)
Pools:
- fast (VMware VM files): 3x RAID-Z2 8x 1.2TB 10k 2.5 Samsung SAS​
- slow (backups / media): RAID-Z1 4x 18TB 5400 3.5 Seagate Ironwolf Pro SATA​
- boot (TrueNAS boot): mirror 2x 200GB 2.5 Kingston SSD​
Capacity: (before file systems)
- 21.6TB (18x 1.2TB) of 10k SAS storage, double-disk parity on each volume​
- 54TB (3x 18TB) of 7200 SATA storage, single-disk parity​

Additional info:
The pool setup currently in use is:​
- (red) 6x 10TB WD Elements SATA in RAID-Z2 (backups / media) / 36TiB usable​
- (gold) 4x 4TB WD Gold SATA in RAID-Z1 (VMware VM files) / 10TiB usable​
- (purple) 2x 8TB WD Purple SATA in RAID-1 (Blue Iris NVR) / 8TiB usable​
After moving to a new server, this frees up the old server. The plan is to repurpose it for offsite backups:​
- RAID-Z3 12x 10TB WD Elements SATA in RAID-Z2 (offsite backups) / 81TiB usable​
- The offsite server will have a little more than enough space to host a full 1:1 copy of the primary server.

Some questions I have:
1. Thoughts on virtualizing TrueNAS, and passing the disks through directly to the VM?
I'm kind of leaning towards just putting it on bare metal, but it would be nice to be able to virtualize TrueNAS (and run other VMs) if the performance is relatively comparable (within maybe a 10% penalty loss) to that of running it physically. Whether I go with 2x E5-2690v4 or E5-2640v3 depends on whether or not I virtualize TrueNAS. Originally I was thinking about passing the disks through to VMware but if that causes issues, I'm going to back off and just install TrueNAS on bare metal. If the consensus isn't one way or the other, I'll absolutely resort to doing testing of my own before a full deployment.​
2. Is an H730P fine, or should I really try to get an HBA330 instead?
I've heard mixed results ("H730P is fine" / "you should get an HBA330") and honestly don't have much of a preference, outside of perhaps resell value. This is such a small non-issue, but comments would be appreciated if there's a clear difference in performance.​
3. Any insights into VM performance with the above proposed setup? (pool architecture, HBAs, etc)
VMs will be served to three ESXi servers via NFS, over 4x LACP'd 10GBps connections. What I have is sufficient but during brief periods of activity, I can see noticeable performance hiccups. I think it's partially because I'm reaching higher levels of storage saturation (using 66% of the 4-drive RAID-Z1 pool currently), but also partially because I have so few drives in that pool (4 drives in RAID-Z1) that are able to share the load. Upgrading to a larger amount of smaller drives should help with this. Surely the new setup will be better than just 4x 7200 RPM SATA drives in a single RAID-Z1, but I'm looking to see some real use-case if anyone has that info.​
4. I'm not necessarily averse to switching to a mirrored setup for even faster VM storage, but...
I don't have too many reservations about switching to a mirrored setup such as:​
- fast (VMware VM files): 8x mirror - 3x 1.2TB 10k 2.5 Samsung SAS​
- slow (backups / media): RAID-Z1 - 4x 18TB 5400 3.5 Seagate Ironwolf Pro SATA​
However I've really enjoyed the additional capacity that RAID-Z2 provides, compared to mirrors, and don't know if the risk of losing both drives in the same mirror (and thus the whole pool) is worth the speed bonus. Sure, I'll have offsite backups, but that could still represent hours - if not days - of data recovery, and short-term "not yet backed up" data loss. Feel free to share your use-case and the speeds you've seen. Despite being slightly biased, I'm open to persuasion.​
5. Anything else that is glaringly obvious, feel free to comment.
Please provide feedback! I'm here to learn and to build a safe, yet performant setup.​

Thank you!
 
Last edited:

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
fast (VMware VM files): 3x RAID-Z2 8x 1.2TB 10k 2.5 Samsung SAS
That 3 disks worth of IOPS available to you
VM's need IOPS
You could have 12 disks worth of IOPS with Mirrors or 8 IOPS worth in 8 * 3 way mirrors if you want more resiliancy

Are these disks new? Have you considered SSD's instead?

slow (backups / media): RAID-Z1 4x 18TB 5400 3.5 Seagate Ironwolf Pro SA
Z1 not reccomended for 18TB disks. You should use Z2 or better

H730P - does not appear to be flashable to IT mode from the only stuff I can find
H330 - DOES have IT mode firmware available - so use this

Lastly as you are using NFS consider the use of a SLOG on the fast array. Use something like Optane (or a pair of them) or an RMS-300 or two would be even better - this will help the NFS write side of things
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,112
slow (backups / media): RAID-Z1 4x 18TB 5400 3.5 Seagate Ironwolf Pro SA
Z1 not reccomended for 18TB disks. You should use Z2 or better
Putting some figures to back up the recommendation:

Nominally, these Ironwolf Pro (7200 rpm) are rated for an URE rate of u=1E-15, per bit. 18 TB = 1.44E14 bits, not so far from 1E15…

Let's follow the "RAID 5 is dead" reasoning and assume that one drive has failed; the remaining 3 members must be read to resilver.
Probability of NOT getting an URE probability for reading N bytes (i.e. successful rebuild): p(N) = exp(-8Nu)
Traditional RAID5 reads everything, p(3*18T) = 65%, or 35% chance of failure (and losing everything).
ZFS reads only what it needs, so if the disks were 2/3 full, p(2/3*3*8*18T) = 75%, so "only" a 25% chance of ZFS reporting a corrupted file which has to be manually restored from backup.

If the choice for the R730xd is not set in stone, I'd go for something with more 3.5" bays, to have a wider raid-Z2 (or Z3) as sufficiently resilient slow storage.
 

NobleKangaroo

Dabbler
Joined
Apr 18, 2017
Messages
18
fast (VMware VM files): 3x RAID-Z2 8x 1.2TB 10k 2.5 Samsung SAS
That 3 disks worth of IOPS available to you
VM's need IOPS
You could have 12 disks worth of IOPS with Mirrors or 8 IOPS worth in 8 * 3 way mirrors if you want more resiliancy
Sorry for the confusion. The chassis has 24 front bays. 24 disks, divided into 3 RAID-Z2, is 8 disks per RAID-Z2 (6 available per RAID-Z2), 18 total. I didn't know a better way to denote that outside of "3x RAID-Z2 8x 1.2TB 10k 2.5 Samsung SAS" in the pools section and "Front plane: 24x 1.2TB 10k 2.5 Samsung SAS (fast VM storage)" in the disks section.

Are these disks new? Have you considered SSD's instead?
Yes they're new. I am not against SSDs. Do you have any recommendations? SSDs will love a shorter lifetime, and it's just not something I thought to look into. Something striking a balance between lifetime and performance would be great. I was mostly concerned with uptime and "leave it alone"-ness.

slow (backups / media): RAID-Z1 4x 18TB 5400 3.5 Seagate Ironwolf Pro SA
Z1 not reccomended for 18TB disks. You should use Z2 or better
Duly noted. Thanks!

H730P - does not appear to be flashable to IT mode from the only stuff I can find
H330 - DOES have IT mode firmware available - so use this
Yeah the only reason I mentioned it is prior to making this post, I had found a few places that mentioned it was possible with newer firmware. Here's a post from the TrueNAS forums even. However just because "new firmware allows it" doesn't nullify the need to make sure the performance is at least comparable to an H330.

Lastly as you are using NFS consider the use of a SLOG on the fast array. Use something like Optane (or a pair of them) or an RMS-300 or two would be even better - this will help the NFS write side of things
Good point. I recently added an Intel 900P (SSDPED1D480GA) configured as a SLOG, and just forgot to mention that as it's new.

Putting some figures to back up the recommendation:

Nominally, these Ironwolf Pro (7200 rpm) are rated for an URE rate of u=1E-15, per bit. 18 TB = 1.44E14 bits, not so far from 1E15…

Let's follow the "RAID 5 is dead" reasoning and assume that one drive has failed; the remaining 3 members must be read to resilver.
Probability of NOT getting an URE probability for reading N bytes (i.e. successful rebuild): p(N) = exp(-8Nu)
Traditional RAID5 reads everything, p(3*18T) = 65%, or 35% chance of failure (and losing everything).
ZFS reads only what it needs, so if the disks were 2/3 full, p(2/3*3*8*18T) = 75%, so "only" a 25% chance of ZFS reporting a corrupted file which has to be manually restored from backup.

If the choice for the R730xd is not set in stone, I'd go for something with more 3.5" bays, to have a wider raid-Z2 (or Z3) as sufficiently resilient slow storage.
I did need to clarify details in the original post as I did with NugentS - the R730xd SFF has 24x 2.5" bays on the front, 4x 3.5" bays in the mid, and 2x 2.5" rear sleds. That's 28 disks in total (plus the two for boot), compared to the 12 disks in the current setup (plus the two SATA DOMs I'm booting from).

It's going to be between these two:
R730xd SFF (24x 2.5 front bays + 4x 3.5 mid bays + 2x 2.5 rear sleds)​
R730xd LFF (12x 3.5 front bays + 4x 3.5 mid bays + 2x 2.5 rear sleds)​

Both options are fine, but ideally better performance on the "fast" storage would be desired. I think the former would provide that, compared to using more 3.5" disks. The current setup is starting to fill up, and VM performance is starting to become slightly, yet noticeably, slower. My running theory is that the number of disks (12 right now) was one of the issues, and also moving the VMs off of the same storage that media and ISOs and other less-often-written data would help the performance. I may be wrong, but it just seems like another 12-bay drop-in (ignoring the fact that the R720xd also has 4x mid bays) wouldn't help much. Would be interested in hearing what you think though.

An alternate pool configuration is to use 2.5" to 3.5" converters in an R730xd LFF:
Disks:
- Front plane: 24x 1.2TB 10k 2.5"​
- Mid plane: 4x 18TB 5400 3.5"​
Two problems with this setup though:
The 2.5" bays are greatly reduced, down to less than half, in the name of adding more 3.5" drives.​
There's now a weird number of 2.5" drives (10) to be split into zvols - either 2x RAID-Z2, or 1x RAID-Z3.​
I don't know that either of those pool configurations is optimal.



Appreciate all of the feedback so far.
 
Last edited:

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
The chassis has 24 front bays. 24 disks, divided into 3 RAID-Z2, is 8 disks per RAID-Z2

As @NugentS told you, that is the equivalent of 3 disks of IOPS because you have 3 vdev. Whenever something is required from one vdev, all drives in it are required to get it. As such, the request is as slow as the slowest drive took to answer. Also, they are all busy answering that request so can not do anything else.

If you do 12x mirrors, you will have 12 vdev, so the basically the speed from 12 drives. For VMs, the more IOPS the better. As such, do not expect great performance from your 3x RaidZ2 for VMs.

Also, about the 2.5 format, many of these drives are SMR or slow performance. SSD will be much faster. Be sure not to have any SMR drives in all cases.
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,112
Sorry for the confusion. The chassis has 24 front bays. 24 disks, divided into 3 RAID-Z2, is 8 disks per RAID-Z2 (6 available per RAID-Z2), 18 total. I didn't know a better way to denote that outside of "3x RAID-Z2 8x 1.2TB 10k 2.5 Samsung SAS".
I think that @NugentS understood you, but a raidz2 has the IOPS performance of a single drive, so your proposed geometry has the IOPS of 3 drives. VMs do better on (lots of) mirrors, and SSDs have more IOPS than hard drives.
If performance on the fast storage is a key driver, mirrors of (data centre grade) SSDs would certainly do better, even with less than 24 drives, and especially if those could be NVMe drives rather than SATA/SAS. Budget allowing, of course.

I forgot to mention that I do have an SLOG, it's recently new so forgot to mention it. I've got a Intel 900P (SSDPED1D480GA) configured as an SLOG.
Good point.

Both options are fine, but ideally better performance on the "fast" storage would be desired. I think the former would provide that, compared to using more 3.5" disks. The current setup is starting to fill up, and VM performance is starting to become slightly, yet noticeably, slower. My running theory is that the number of disks (12 right now) was one of the issues, and also moving the VMs off of the same storage that media and ISOs and other less-often-written data would help the performance.
That would certainly help, but raidz1 is not a recommended setting for VMs and from the first post you're at 66%, which is above the recommended threshold of 50%.
 

NobleKangaroo

Dabbler
Joined
Apr 18, 2017
Messages
18
As @NugentS told you, that is the equivalent of 3 disks of IOPS because you have 3 vdev. Whenever something is required from one vdev, all drives in it are required to get it. As such, the request is as slow as the slowest drive took to answer. Also, they are all busy answering that request so can not do anything else.
Oh, I thought he misunderstood what I meant. Thank you for explaining! That's a very important detail that I overlooked.

I think that @NugentS understood you, but a raidz2 has the IOPS performance of a single drive, so your proposed geometry has the IOPS of 3 drives. VMs do better on (lots of) mirrors, and SSDs have more IOPS than hard drives.
Thank you as well. I see what you mean, how it's limited to the slowest drive in each zvol.

If you do 12x mirrors, you will have 12 vdev, so the basically the speed from 12 drives. For VMs, the more IOPS the better. As such, do not expect great performance from your 3x RaidZ2 for VMs.
Got it. So basically, I should go with mirror vdevs (12x 2-drive mirrors) for best performance? Is triple mirrors (8x 3-drive mirrors) a decent compromise, despite having to write to all three drives? I would gander to think it might actually be even faster for reads, since data can be read from any of the three drives in each mirror. So either 12x 2-drive mirrors (14.4TB, or 12*1.2TB), or 8x 3-drive mirrors (9.6TB, or 8*1.2TB). Either of these are more than enough for the ~2TB worth of VM disks.

Also, about the 2.5 format, many of these drives are SMR or slow performance. SSD will be much faster. Be sure not to have any SMR drives in all cases.
Yeah it's becoming (sadly) common for CMR to be replaced with SMR. I did some searching to find the largest drives that are still CMR, and the option I was looking at was the Seagate ST1200MM0088. These 1.2TB drives can be had for around $80 (new). Another option was the Spindisk 2TB from jgreco's post, and they can be had for around $115 (also new). Either of these two will provide more than enough space for the VMs I run, and allow for adequate breathing room for ZFS to work its wonders.

If performance on the fast storage is a key driver, mirrors of (data centre grade) SSDs would certainly do better, even with less than 24 drives, and especially if those could be NVMe drives rather than SATA/SAS. Budget allowing, of course.

That would certainly help, but raidz1 is not a recommended setting for VMs and from the first post you're at 66%, which is above the recommended threshold of 50%.
Understood. So I need research recommended SSDs.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
One other point to consider. Lets assume 1TB Disks (SSD/HDD doesn't matter for this calculation) (and I am rounding up for simplicity)
24 Disks in mirrors = 12TB Useable space.
Keep at 50% occupancy = 6TB Useable space
Or 3 way mirrors = 8TB Useable Space = 4TB actually useable space at 50% occupancy

DC SSD's are gonna have to work hard to match $100 (ish) - they typically cost a lot more.

Question - how much space do you actually need for VM's? AND what sort of loads are on the VM's?
What is your use case?
 

NobleKangaroo

Dabbler
Joined
Apr 18, 2017
Messages
18
One other point to consider. Lets assume 1TB Disks (SSD/HDD doesn't matter for this calculation) (and I am rounding up for simplicity)
24 Disks in mirrors = 12TB Useable space.
Keep at 50% occupancy = 6TB Useable space
Or 3 way mirrors = 8TB Useable Space = 4TB actually useable space at 50% occupancy

DC SSD's are gonna have to work hard to match $100 (ish) - they typically cost a lot more.

Question - how much space do you actually need for VM's? AND what sort of loads are on the VM's?
What is your use case?
Servers and hosted services:
- TrueNAS server - see my signature for exact specs​
- 3 ESXi servers, R630​
- Running about 20 VMs, mostly Debian and one Windows 10 VM (Blue Iris)
- Docker containers (on 3 Debian VMs, configured to run on different ESXi hosts):
- Reverse proxy: (3) NGINX​
- Keepalived: (3) keepalived (floating IP for reverse proxies)​
- Authentication: (3) FreeIPA, Bitwarden​
- Media storage: Nextcloud​
- Documentation: Dokuwiki, Netbox, PHPIPAM, Openproject​
- Monitoring / scanning: Zabbix, OpenVAS, Logstash, Elasticsearch, Influxdb, Grafana, Kibana​
- Usenet and torrents: NZBGet, Radarr, Sonarr, Lidarr, Bazarr, Transmission
- Media requests: Ombi, Tautulli, Requestrr​
- Media consumption: Emby, Plex
- Game servers here and there (Valheim, etc)
- Other various containers from time to time
Much of what I store is media accessed by family and friends, Nextcloud backups of photos and videos from the family's phones, and 8TB worth of NVR recordings. This is the bulk of the "red" pool currently, and represents (at time of writing) ~15TB of data. I'm in the process of removing old media that is no longer wanted, but that's what is currently being used. I could also move the NVR to a dedicated server, eliminating 8TB of disk usage.

I am contemplating going with an R730XD LFF (12x 3.5") like:
- front: 2x 3.5" 18TB + 10x 1.2TB​
- mid: 4x 3.5" 18TB​
- rear: 2x 1.2TB​
- boot: 2x SATA DOMs (or PCIe-connected drives)​

The addition of SATA DOMs (or PCIe-connected drives) frees up the 2 rear bays, allowing for 2 more 2.5" bays. After subtracting 2x 2.5" bays from the front (to provide a total of 6x 3.5" drives for "slow" storage) and adding 2 to the back, that brings the total for 2.5" drives back up to 12.
- fast: 12x 2.5" (10 in front bays, 2 in rear bays), in either 6x mirrors (6x 1.2TB, 7.2TB) or 4x triple mirrors (4x 1.2TB, 4.8TB)​
- slow: 6x 3.5" (4 in mid bays, 2 in front bays), in RAID-Z2 (4x18TB, 72TB)​
- boot: 2x SATA DOMs (or PCIe-connected drives)

This configuration would be a mid-ground between enough mirrored drives for VMs (either 28% or 42% depending on mirrors) and enough space for storage that can be slower (around 25-30%) after migrating. I might also have to get over my mirror concerns and just make absolutely certain that solid backups exist (good practice anyhow). :wink:
 
Last edited:

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
I have no idea what a R730XD costs - or what your budget is
Would it not be better to DIY with a 24/36 bay case and appropriate HBA's, M/B, CPU, Memory etc using second hand kit where available?

It is a fairly impressive kit list though
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'm kind of leaning towards just putting it on bare metal, but it would be nice to be able to virtualize TrueNAS (and run other VMs) if the performance is relatively comparable (within maybe a 10% penalty loss) to that of running it physically. Whether I go with 2x E5-2690v4 or E5-2640v3 depends on whether or not I virtualize TrueNAS. Originally I was thinking about passing the disks through to VMware but if that causes issues, I'm going to back off and just install TrueNAS on bare metal. If the consensus isn't one way or the other, I'll absolutely resort to doing testing of my own before a full deployment.

Follow the guidance at


I've heard mixed results ("H730P is fine" / "you should get an HBA330") and honestly don't have much of a preference, outside of perhaps resell value. This is such a small non-issue, but comments would be appreciated if there's a clear difference in performance.

Well, you can discount the sources where you've heard "H730P is fine" as sources of bad advice. Please see

 

NobleKangaroo

Dabbler
Joined
Apr 18, 2017
Messages
18
I have no idea what a R730XD costs - or what your budget is
Would it not be better to DIY with a 24/36 bay case and appropriate HBA's, M/B, CPU, Memory etc using second hand kit where available?

It is a fairly impressive kit list though
The cost is around $2,200 as configured, minus the cost of drives, and it'll have iDRAC and match the other servers (outside of my current TrueNAS and pfSense router) in the fleet. Plus these Dell servers retain their resell value pretty well (I moved from R710s about 6 months ago and they sold for a pretty good portion of what they cost initially).

Follow the guidance at

I've been doing a lot of reading and actually came across that thread a few days ago. It's an excellent read, and I think I'm going to give it a shot. I'm not dead set on virtualizing TrueNAS, but it's something I'd like to experiment with at least. I will have plenty of time to A/B test things and can easily swap back and forth between OSes by installing them on a single drive while testing. I will share whatever results come of testing, good or bad. I think the best thing to do would be starting out installing TrueNAS on baremetal then switching over to ESXi to test. At least if something isn't right with how I'm trying to configure ESXi, it won't won't be the cause of any issues and instead it would just fail to import the volume that was created by TrueNAS.

Cheers for another awesome guide!

Well, you can discount the sources where you've heard "H730P is fine" as sources of bad advice. Please see

Thanks. Point 5 "A RAID controller that supports "JBOD" or "HBA mode" isn't the same" sums it up pretty well. I'll stick with an H330 for safety. There's not much benefit to "trying to make it work" with an H730P.
 
Last edited:
Top