TrueNAS Mini XL+ Recommendation

[Cloudified]

Hello,

This is my first post on this forum. I am about to purchase a TrueNAS Mini XL+ with 64GB of RAM along with the TrueNAS Mini 10Gb Dual-Port SFP+ Ports, TrueNAS Mini Write Cache (ZIL), and TrueNAS Mini 480GB Read Cache (L2ARC) Upgrades. I would like some recommendations on my HDD configuration options. My primary use for this NAS is storage and to host VMs and Docker images using TrueNAS SCALE and would ideally like 32TB of usable space.

After reading countless posts on all of the possible ZFS configurations, I have determined that a single pool with mirrored vdevs is the best way for me to go. I am not on an unlimited budget, so I want to make the best decision based on my future storage needs and reliability during a recovery situation. Since this chassis has 8 hot-swappable 3.5" drive bays, am I better off going with (8) 8TB disks or (4) 16TB disks? Both will give be 32TB of usable storage, but I am concerned about re-silvering time during failed disk situation to to mention the cost of an individual failed disk. If I go with the larger configuration, I will still have 4 hot-swappable bays for future expansion although 32TB of usable storage is way more than I need for the foreseeable future. If I go with the more of the smaller disks, will performance be better because of 4 mirrored vdevs instead of 2 larger ones?

I am pretty sold on the WD Red Plus HDDs or WD Red Pro HDDs, but I'm not sure if I am better off with 4 larger disks or 8 smaller disks. The WD Red Pro HDDs are cheaper than the WD Red Plus HDDs for whatever reason despite the higher-end specs.

I also understand that there is an M.2 slot the motherboard. Does this mean I can use any M.2 2280 SSD in addition to 8 SATA drives? I am a little unclear on how many SATA drives the backplane can handle. I believe the BIOS is limited to 8 HDDs, but not sure sure if the M.2 slot counts as one of those 8. I could use this slot for a single device pool for some future need when SSD performance is needed.

This is overkill for home use, but I no longer want to keep a bunch of USB drives laying around with duped data all over the place. This NAS will be used for system backups (Time Machine), huge movie libraries, and all personal and business data. I will also sync the important data to Azure. Please share your thoughts on all of this. Thank you so much!

/David

 

[Evertb1]

First of all: good for you that you try to start wel informed before you buy. That's the reasonable way to go.

Choosing a configuration for your storage pool is depending on your use case. But be aware of the fact that mirrors are not the most economical way to handle your need for storage space. perosnally I like the economy of RAID-Z2 with at least 6 drives the best. I am aware that I only achieve single disk speed that way but it's more then sufficent for my use case.

You should also be aware of the fact that if you go for a mirror setup when you need the speed, you will end up with less then 50% of your raw disk space as usable space. You will have some overhead in the form of some slop space and a 20 % free space limit needed to let ZFS function properly when your usable space is getting filled up. That 20 % is not a concreet figure but it would be wise to caluclate it in anyway. For example: 4 x 8TB disks in 2 mirrored pairs will give you only around 11.23 Tib or 12.34 TB practical usable storage capacity. That's only 38% of your raw capacity. If you want to play with some figures your self I suggest that you visit this ZFS/RAIDZ capacity calculator.

 

[Etorix]

Reading your post and use case, I'm not convinced you need a ZIL (which is NOT a write cache, no matter how many people make the mistake) and even a L2ARC.
Bulk storage does best on Z2/Z3 (with ca. 80% occupancy limit) but VMs do best on mirrors with ca. 50% occupancy limit.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

My suggestion would be to drop ZIL and L2ARC options, buy two 2.5" SATA SSDs and make them a mirror for your VMs.
Then 6-8 3.5" HDD raidz2 for storage. Whatever combination of size (8-16 TB) and drive number comes out best for your planned capacity. If Red Pro are cheaper than Plus, go for Pro—but first check whether you can find WD Gold, Seagate Ironwolf, Seagate Exos or Toshiba N300/MG for an even better price!

 

[Cloudified]

First of all: good for you that you try to start wel informed before you buy. That's the reasonable way to go.

Choosing a configuration for your storage pool is depending on your use case. But be aware of the fact that mirrors are not the most economical way to handle your need for storage space. perosnally I like the economy of RAID-Z2 with at least 6 drives the best. I am aware that I only achieve single disk speed that way but it's more then sufficent for my use case.

You should also be aware of the fact that if you go for a mirror setup when you need the speed, you will end up with less then 50% of your raw disk space as usable space. You will have some overhead in the form of some slop space and a 20 % free space limit needed to let ZFS function properly when your usable space is getting filled up. That 20 % is not a concreet figure but it would be wise to caluclate it in anyway. For example: 4 x 8TB disks in 2 mirrored pairs will give you only around 11.23 Tib or 12.34 TB practical usable storage capacity. That's only 38% of your raw capacity. If you want to play with some figures your self I suggest that you visit this ZFS/RAIDZ capacity calculator.

Thanks for the response. I am good with choosing the the speed and reliability parts of the triangle since I am will be running VMs. In other words, I’m okay with losing 50% of space in favor of the other two. I have played around a lot with that same calculator you provided. My main concern is the stress and resilvering time if I want to start replacing disks with bigger drives using a RAIDZ2 configuration. I haven’t upgraded my UniFi switches to 10Gb yet, but I’m planning ahead and getting all of the upgrades at once so they are already there for the initial TrueNAS SCALE install (unfortunately I have an all-or-nothing mentality).

 

[Cloudified]

Reading your post and use case, I'm not convinced you need a ZIL (which is NOT a write cache, no matter how many people make the mistake) and even a L2ARC.
Bulk storage does best on Z2/Z3 (with ca. 80% occupancy limit) but VMs do best on mirrors with ca. 50% occupancy limit.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

My suggestion would be to drop ZIL and L2ARC options, buy two 2.5" SATA SSDs and make as mirror for your VMs.
Then 6-8 3.5" HDD raidz2 for storage. Whatever combination of size (8-16 TB) and drive number comes out best for your planned capacity. If Red Pro are cheaper than Plus, go for Pro—but first check whether you can find WD Gold, Seagate Ironwolf, Seagate Exos or Toshiba N300/MG for an even better price!

All good suggestions and something for me to think about. I can go ahead and order all of the TrueNAS hardware as I contemplate which disks to go with. My plan is have it all built and ready for the official release of TrueNAS SCALE next in a few weeks. Thanks for the response!

 

[Cloudified]

Thank you guys for your suggestions. I'm glad I did more research and figured out why the WD Red 8TB drives (Plus or Pro) are not on the Mini-series compatibility list. Like their 10TB cousins, of which there is a footnote, they draw significantly more power (over 8w) than the rest of the drives which could cause power issues and overheating - especially when connecting 8 drives in the Mini XL+. Now I'm leaning towards just getting the Mini X+ with (5) 14TB drives. Two mirrors and one hot-spare or possible RAIDZ2.

 

[ChrisRJ]

Can you be more specific what VMs and Docker containers you will be using? I am running a small DMS (document management system) on a RAIDZ2. But that only works because I am the only person using it. Likewise a Unifi Controller would work on this configuration. But a shared database server would certainly go to its knees.

 

[Cloudified]

Can you be more specific what VMs and Docker containers you will be using? I am running a small DMS (document management system) on a RAIDZ2. But that only works because I am the only person using it. Likewise a Unifi Controller would work on this configuration. But a shared database server would certainly go to its knees.

Nothing too power intensive. I will use Docker containers for Pi-hole, Homebridge, and a few others. I am currently running all of those applications on Raspberry Pi's now, but I would like to migrate everything to the TrueNAS that I'm building. As far as VMs are concerned, I will use them on an as-needed basis for testing new software and OS builds. I will also have a dedicated Windows 11 VM running for doing PowerShell with O365. Like you, I will be the primary user for everything.

It's worth noting that I have not upgraded my UniFi network to 10Gb yet, so using non-SSDs should be fine for my use case. I'm going to go ahead and throw the 10Gb card into the build just so it's there when I need it later.

 

[jgreco]

Be aware that WD Red 8TB non-plus/non-pro are likely to be SMR drives, so make sure you don't fall into that trap.

 

[Cloudified]

Be aware that WD Red 8TB non-plus/non-pro are likely to be SMR drives, so make sure you don't fall into that trap.

I would only use the Plus or the Pros. I can't figure out why, but some of the same size drive offerings are cheaper with the Pro than the Plus. For example, the WD Red Plus 14TB is $350 and the WD Red Pro 14TB is $330 on Amazon and WD.com right now. Slightly faster with a 5-year warranty instead of a 3-year for less money? Sign me up!

I've also decided against the 8TB versions. Like the 10TB offerings, they use over 8W which is too much draw for the XL+ with 8 drives.

 

[ChrisRJ]

I can't figure out why, but some of the same size drive offerings are cheaper with the Pro than the Plus.

As @Etorix mentioned, you should definitely check the enterprise drives as well. Those tend to be even cheaper, at least when I checked here in Germany.

 

[Cloudified]

OK, I have decided to order the XL+ with 64GB and the ZIL/L2ARC upgrades.

Usable space side, if I end up going with one of the following configurations, will I really be able to tell the difference performance between RAIDZ2 vs mirrored vdevs? As I mentioned earlier, this is for home use for some occasional VMs and full-time Docker containers for Pi-hole, Homebridge, and UniFi. I will pretty much be the only user and have never used ZFS before, so I am just trying to make sure I make the right decision on my setup before I order the disks and configure TrueNAS SCALE.

All of these disk configuration options will give me the usual apace I need. Just trying to my the right choice on the ZFS options. Not sure if I will really notice the difference in performance. Using 4 larger disks vs 8 smaller disks gives me some room to grow. If 8 smaller drives performs better than 4 larger ones, then that's what I'll probably do. I'm trying to stick with drives on the TrueNAS Mini compatibility list. Welcome any thoughts. Thanks.

Option 1: (8) WD Red Pro 6TB drives with 4 mirrored vdevs (24TB usable)
Option 2: (8) WD Red Pro 6TB drives with RAIDZ2 (32TB usable)
Option 3: (4) WD Red Pro 12TB drives with 2 mirrored vdevs (24TB usable)
Option 4: (4) WD Red Pro 12TB drives with RAIDZ2 (32TB usable)

 

[Evertb1]

Option 1: (8) WD Red Pro 6TB drives with 4 mirrored vdevs (24TB usable)
Option 2: (8) WD Red Pro 6TB drives with RAIDZ2 (32TB usable)
Option 3: (4) WD Red Pro 12TB drives with 2 mirrored vdevs (24TB usable)
Option 4: (4) WD Red Pro 12TB drives with RAIDZ2 (32TB usable)

You stated before that you looked at the ZFS calculator. You should know then that you don't end up with 50% of the raw space as usable space. There is always some overhead. You can't avoid that. But more important is the fact that ZFS needs some free space to function properly. The rule of thumb is 20% but you can play it closer then that if you want to take the risk. That 20 % will effect your usable storage, leaving you with the practical usable storage of around 38 - 39 % of the raw space. So if you really want to end up with at least 24TB practical usable space you should take that in account. You would not be the first one to ask for help when your storage has gone out of sufficient free space and problems arise with your ZFS system, manifesting itself as a big drop in performance for example. In terms of usable space mirrors are expensive. In order to have 24 TB as practical usable space you should plan for a vdev with around 64 TB raw space.

 

[Cloudified]

You stated before that you looked at the ZFS calculator. You should know then that you don't end up with 50% of the raw space as usable space. There is always some overhead. You can't avoid that. But more important is the fact that ZFS needs some free space to function properly. The rule of thumb is 20% but you can play it closer then that if you want to take the risk. That 20 % will effect your usable storage, leaving you with the practical usable storage of around 38 - 39 % of the raw space. So if you really want to end up with at least 24TB practical usable space you should take that in account. You would not be the first one to ask for help when your storage has gone out of sufficient free space and problems arise with your ZFS system, manifesting itself as a big drop in performance for example. In terms of usable space mirrors are expensive. In order to have 24 TB as practical usable space you should plan for a vdev with around 64 TB raw space.

Thanks for the response. I was just estimating and which why I said “usage space aide” in bold. I’m just trying to determine the performance difference between the 4 options. Any of these options give me more than enough usable space to meet my needs even with the overhead factored in. Thanks again.

 

[Etorix]

Options 3 and 4 (4 drives) give you the opportunity to add a second vdev to extend the pool—or a mirror vdev for VMs.
Options 1 and then 3 will have higher performance, especially for IOPS, but if there's one single user, it may not be noticable. Options 2 and 4 (raidz2) are arguably more secure—with such large drives, option 3 should consider 6*12 TB in two 3-way mirrors.

 

[Cloudified]

I would like to thank all of you that chimed in. This give me a lot of different options that I hadn’t considered. I am very excited that my Mini XL+ will be here next week. I also ordered the additional TrueNAS 32GB of RAM, TrueNAS Mini 10Gb Dual-Port SFP+ Ports, TrueNAS Mini Write Cache (ZIL), and TrueNAS Mini 480GB Read Cache (L2ARC) upgrades.

I will order the HDDs after I figure out what my final TrueNAS SCALE pool strategy is going to be. Thanks again for all of your suggestions!

 

[Etorix]

We may not have weighted enough on this one: The SLOG (hardware ZIL) is not a write cache. It does nothing for asynchronous writes. It does speed up synchronous writes, but asynchronous writes are always faster than synchronous writes (even with a SLOG).
Synchronous writes are for NFS, iSCSI and mission critical VMs. If none of these apply, there is no use for a SLOG.

 

[mauirixxx]

I just got my Mini XL+ last week. The m.2 slot is ued as the boot drive, just an fyi @Cloudified

 

[Cloudified]

I just got my Mini XL+ last week. The m.2 slot is ued as the boot drive, just an fyi @Cloudified

Thanks. I ended up going with a different NAS solution.

 

[mauirixxx]

Thanks. I ended up going with a different NAS solution.

did you roll your own or bought another off the shelf solution?