TrueNAS Mini R max HDD performance configuration (22TB WD Red Pro)

[mb17]

We are building out the cheapest iSCSI target we can build using quality hardware for our backup server. We chose the Mini R for this as it will be going in a data center and price was the main concern. Yes I know this could have been done many ways but without getting into all the reasons, this is what we chose. It is also just a ~1yr stop gap until we replace with a larger long term solution. Depending on how well this project goes, TrueNAS will be chosen for that solution when the time comes.

Chosen Hardware:

• TrueNAS Mini R w/ 64gb RAM and the 10gbase-t default connections.
• 12x 22tb WD RED Pro drives. - I know these are not on the verified list and accept the risk.

Goals I am trying to achieve:

• ~120tb usable
• Highest IOPS possible
• >4gbit sustained during fairly random 128/256k block size reads.
  • This is to meet restore performance goals.
  • I am not worried about write performance as our current, slower solution, has a backup window that is sub 8 hours on a bad day.

Config as I am thinking so far:

• Single 10gbase-t direct connection to port on the backup server with iSCSI.
• 6x Mirror Vdev's
• lz4 compression enabled
• 128 or 256 record size
• Sync - set to "Standard" which I think for this use case would end up being "Always".
• Scrub - 5th of every month
• Short SMART Test - 1st, 11th, 21st
• LONG SMART Test - 15th
• Backup server will be formatting the iSCSI target in ReFS format file system. The current data gets over a 5x reduction but this does come at the expense of increased data fragmentation.

I am looking for any recommendations someone might have on this configuration. The main area I am worried about is the Scrub and SMART schedules. I have searched many forum posts and see configurations with more CPU/RAM with large drives or smaller drives with this type of CPU/RAM configuration. Just looking for a little feedback from anyone willing to spend the time to help.

I do plan to run many tests using Crystal DiskMark once setup. Then again once I have migrated the data over. I don't want to promise but I will try and post all my results for anyone else to benefit from. If anyone has any specific tests they would like ran on this configuration before it goes into production, I will do my best to run them.

Note: I am still learning a lot about TrueNAS so bear with me. I do have 2 custom boxes at home for learning with and a new M30 all flash at work, just so you know where my current knowledge is coming from.

 

[HoneyBadger]

Migrated to the Performance forum.

Can I ask what the backup software being used is? Depending on the software it may be advantageous to store files directly on the NAS over SMB/NFS vs mounting it as an iSCSI target.

 

[mb17]

Veeam - latest release.

 

[HoneyBadger]

Veeam - latest release.

I'll link back to the documentation on Veeam integration here:

Veeam

Guide for deploying TrueNAS CORE systems as a Veeam backup solution.

www.truenas.com

Note that in this example, they've configured for capacity with RAIDZ2 - for your case driving "maximum IOPS" you may wish to stick with mirror vdevs, but this will give you approximately 6x22T = 132T of space before any overhead, meaning storing 120T puts you at over 90% of space used.

Sync - set to "Standard" which I think for this use case would end up being "Always".

sync=always will likely cause a significant reduction in write throughput, even with the larger volblocksize for your zvols, but this might be mitigated if you use ReFS with a 64K cluster size. It's also suggested to make multiple zvols and iSCSI extents to allow Veeam to scale out across them.

 

[mb17]

I have read that article a few times and again just now. Here are a few clarifications.

My raw backup with Veeam is ~32TB. After the multiple backups using full synthetics and allowing for our on-prem retention plan, we sit around 80TB used due to ReFS. With the 132TB of raw disks showing in the 120-125TB range after formatting (TB vs TiB, etc), I will be more like 70-75% usage.

sync=always - I am perfectly fine with the reduced write performance. Our current solution completes the write part of the on prem incremental backups in about 3 hours. This solution even with sync=always should at least be equal in performance.

Our current 2 storage targets running:

• 8x 10tb Raid10 on Server 2016 - Basic LSI cacheless raid card on board. 6 years old.
• 8x 10tb Raid6 on Server 2016 - Basic LSI cacheless add in raid card. 5 years old.

I would love to use RaidZ2 but I really want the IO to be as high as I can for Veeam performance during restore testing and restores themselves. Again this is temporary for 1-2 years until I can get budget for a proper solution. My hope is something like a 36 HDD system or if I get lucky an all flash solution.


My last questions are on the Scrub and SMART tests. Is the schedule in my original post adequate?

 

[HoneyBadger]

You can always benchmark with sync=always vs sync=standard to determine the impact on your backup workload.

Monthly scrubs and long SMART tests with short SMART tests 3x a month is more than enough - ensure that email or other alerts are in place within TrueNAS, and processes are in place to react properly to these alerts in a timely fashion by replacing drives. I'd recommend having at least one spare HDD on-site (ideally in an extra Mini R drive sled) with it having received a burn-in test separately.

 

[mb17]

You can always benchmark with sync=always vs sync=standard to determine the impact on your backup workload.

Monthly scrubs and long SMART tests with short SMART tests 3x a month is more than enough - ensure that email or other alerts are in place within TrueNAS, and processes are in place to react properly to these alerts in a timely fashion by replacing drives. I'd recommend having at least one spare HDD on-site (ideally in an extra Mini R drive sled) with it having received a burn-in test separately.

I setup email alerts everywhere that I can. I also plan on purchasing 1-2 spares, I just haven't because the vendor only had 12 at the time of purchase. I will have it before this is deployed.

And thank you for all the help and validation on the config. I really appreciate it.

 

[mb17]

Here is the first update.

Got the hardware Friday and filled with 22TB HDDs. Got 1 link setup LAN DHCP and 1 link setup iSCSI to desktop via 10gbase-t.

I started a SMART Long test on all HDD's for the weekend. It took somewhere around 30 hours to complete.

My tests were initially done with NTFS, 1500 & 9000 MTU, and CrystalDiskMark 8.0.4 using a variety of configurations. Pool setup with 128k and 256k options landing with 256k for final. I did this mainly for testing sync enable/disable for iSCSI. Lets just say that I am going to accept the risk of using sync=disabled for this project because the writes are just that bad. I've been spoiled with SSD's that I've forgotten how horrible HDD's speed is when doing anything other than 1 simple task at a time. I really wish this unit had an x8 PCIe slot so that we could put in a dual m.2 card with dual Optane drives for write cache. This thing would fly then.


Now for my real tests. I first attached the iSCSI to a server 2022 machine long enough to format with ReFS 64k blocks since Windows Pro doesn't have the ability to format with this file system. It can read however so I immediately moved the unit back to my PC via direct connect 10gbase-t for the tests. (See the end where I talk about a networking issue with the TrueNAS OS.)

NOTE: Sorry I didn't use the CrystalDiskMark label at the bottom of the window when taking the screenshot. I know better but forgot.

Screenshot configuration: MTU 1500, Sync=Enable, 256k test


Screenshot configuration: MTU 1500, Sync=Disable, 256k test


Screenshot configuration: MTU 1500, Sync=Disable, 512k test


Screenshot configuration: MTU 9014 (My NIC supports 1500/9014), Sync=Disable, 512k test


I did 3 screenshots for the last one to show variations in the speeds I am getting.


Things of note.

• Tests were all done using 64GiB sample size and the unit has 64GB RAM so these are inconclusive for sustained large workload.
• 256/512k blocks are what Veeam uses when reading and writing data after its internal compression. This is why I only reported these values.
• Veeam recommends storage target to be 256k block formatted with ReFS 64k for the filesystem
• IO is much better for 256k chunks of data because it aligns the best with the pool size of 256k. When using 512k data blocks, IO is cut in half.
• CPU during tests would get up to about 75%
• The unit is amazingly quiet. I have it just 2-3 feet from my head. It is no louder than the background talking in my open concept office floor.
• This unit is going into a datacenter and why I am not as concerned with using sync=Disabled. Again I still think having dual m.2 for write cache would make this thing sing.
• Price for what this unit delivers is a really good value.
• I am curious what it could do with a full load of 7.68tb SSD's. From what I have seen, the CPU would probably bottleneck the throughput.
• I wish TrueNAS would publish speed results for this unit in a few different configurations and give some use case information. This would help get it's name out there. Not everyone is willing to drop 5-20k loading this up without some idea of what it is capable of.

In conclusion, I think this is going to work great as a Veeam backup target for our short term needs. It isn't going to be the fastest thing out there but for the price, it can't be beat. Our video editor stopped by and was drooling over the size of drives and how quiet it is.

If anyone has any questions or tests they would like me to run, the server won't be in place until end of week so ask away. I would like to limit my testing to iSCSI as that is what I have the most experience with.

Now for the downside. I am going to post this here but I may start another forum post specifically about this as well.

This unit has two 10gbase-t connections. ix1 I have connected to our lan using DHCP just for management. ix0 is what I have configured for the iSCSI using a static IP. It however shows DHCP yes on the interfaces screen, along with the static IP. When you expand this it shows MTU N/A. When I click on edit, the screen goes back to default. Also when I moved the connection from desktop to server and back to desktop, each time it lost the static IP I set. The only way I found it to work permanently and display all the settings correctly is to first set everything on ix0 for my iSCSI, then go into ix1 which now shows disabled DHCP and correct this. Then when you test the settings it all works and saves as expected. This system just doesn't allow me to change 1 interface at a time. To me this seems like a bug.

 

[mb17]

I forgot to mention, running latest TrueNAS-13.0-U5.1

 

[joeschmuck]

This system just doesn't allow me to change 1 interface at a time. To me this seems like a bug.

Use 'Report a Bug'.

 

[mb17]

Update 2

I got all my data moved over to the unit using the Veeam data mover tools. (make sure you are using Veeam v12 or newer.) My old storage was pulling from two separate servers and only able to peak to 6gbit, 3gbit average throughput to the TrueNAS. I know this is a limitation with my old storage as it was the same speed I got when restoring to our SAN. Also the TrueNAS never went above 65% CPU except for an occasional spike during this data move.

Last night was the first run based on my standard backup window. I had jobs that would normally take upwards of 50 minutes to run but nothing took over 15 last night. I know this is only one day of examples but considering that is already a ~70% reduction, I am very pleased.

Final configuration:

• Veeam server:
  • 10gbase-t connection directly connected to the TrueNAS
  • MTU 9014
  • Formatted the drive using ReFS 64k block size
• TrueNAS Mini R:
  • 6x mirrored 22TB drives
    • 120TB usable but iSCSI volume set to 90TB. I did this to keep it under the 80% usage recommendation.(Yes I am using TB not TiB)
  • 256k record size
  • Sync=disabled
  • lz4 compression
  • iSCSI presented as a single drive
  • No throttling or iSCSI security turned on. I did this because I am using direct connection and my backups are stored encrypted.

While I know this is a very basic setup, it is proving to be very capable. Time will tell if the ReFS fragmentation, that comes with Veeam synthetic full backups, will cause this to slow down any but so far I am impressed. I think this solution will fit many organizations needs and shouldn't be ruled out. If you are the type of company that doesn't "need" fully redundant storage controllers, PSU's, etc, then I suggest considering this as a great performant option for the price.

I will try and post again once I have completed a few weeks of backups and ran through some real world restore performance tests. The whole reason I did a stripped mirror configuration was for maximum restore performance.

Note: The whole reason I am posting these details is to hopefully help someone else.

 

[mb17]

Update 3

I have good news and bad news.

Good News... It is much faster than my old storage.

Bad News... I can't say how much faster in real world. My server running Veeam is now the bottleneck. I can say that I am seeing 1-4x performance on the same jobs I had setup. When it starts hitting the 3-4x speeds, the server CPU and especially RAM is maxed out. Until I get funds to replace it I will not be able to tell how fast it really can go.

So my final thoughts are that more companies should consider the TrueNAS Mini R for use with their Veeam backup solution. The obvious limitations that some may not be willing to accept is single PSU, single OS drive, and not HA controllers. If this is acceptable to you and all flash prices are out of your budget then I don't think you can beat this option.

Two recommendations to TrueNAS/iXsystems

• Come up with an addon card to add caching drives. https://www.amazon.com/Sedna-Controller-HyperDuo-Technology-Connector/dp/B07SGQLTQS/ I know it is just sata but paired with your caching drives might allow a workload like mine to be improved by having a pair of mirrored SSD's as write cache. This could allow me to switch to Sync=Enable and still maintain enough performance while gaining back power loss protection.
• If I am not mistaken, the motherboard in this unit is capable of 256gb RAM according to the board manufacturer. Some customers might like the option of purchasing with this ammount.

 

[NugentS]

There is no write cache - a SLOG is not a write cache. You would do much better with Optane or an RMS300/16G than that Sedna Controller acting as a SLOG.

 

[mb17]

Update 4

I know this is short and sweet but so far this solution is working well for us. It will not be our long term solution as we definitely want more restore speed but for what it is, it works great. If you are a company looking to do just a few TB for a full backup, this might just be the best solution out there.

I have fallen in love with what this does offer though and have already purchased 2 more units loaded with 6TB from them for projects at remote sites.

I've also setup two old Dell desktops at home to act as primary and backup (Located in detached garage) for my home data. And a third that I am setting up now to play around with the newest Scale changes.

So yeah, "I drank the cool aid" for TrueNAS ecosystem.