Check out my TrueNAS Scale Build

[Ebedorian]

So, I've been on here since May 2023, and I have had varying success with several iterations of a TrueNas Scale build that I've been working on. I have had issues with fan speed control (see forum post here) and drive errors (posted here), but I haven't gotten much help. I've just been doing a lot of reading.

I could use some guidance on this. This build is intended to provide archive storage for large files and run a few virtual machines. Most of the gear I used is linked in case anyone is interested in using any of it. If most of it is garbage, I'd still like to know, as this has been a learning process for me.


My Current Build includes the following:

Supermicro X11SSL-F
Intel Xeon E3-1270 v5 @ 3.6 GHz
SST-NT07-115X-USA (CPU Cooler)
64GB Micron Crucial DDR4 (4X16GB) 2133MHz UDIMM RAM
Noctua NF-A6x25 PWM x 2
Rosewill RSV-Z3100U 3U Server Chassis
Silicon Power 128GB SSD 3D NAND A55 SLC x 2 (Boot Pool)
SAMSUNG 845DC 800GB PRO MLC x 4 (Storage Pool)
WL4000GSA6472E (4 x OOS4000G and 1 x ST4000NM0033) (Storage Pool)
MCX311A-XCAT 10GB
9200-8I IT (HBA)
DUAL NVMe to PCIe
Intel OPTANE SSD P1600X Series 118GB x 2 (SLOG)
WD 1TB WD Red SA500 NAS 3D NAND Internal SSD (Cache)

My current design. The support VDEVs are intended for use with the spinning disks, not the SSDs. I'm not even sure if ZIL is necessary. This is where I could use some advice. Running Crystal Disk Mark on an iSCSI drive had putrid results, hence the reason for adding the support VDEVS. But this is where I need the advice.

Thanks again for any assistance given.

 

[chuck32]

nd drive errors (posted here), but I haven't gotten much help.

You just posted your log to an old thread, usually you will receive good help here if you provide good information.

I didn't check everything in detail, it's just at first glance to give you a starting point.

WL4000GSA6472E (4 x OOS4000G and 1 x ST4000NM0033) (Storage Pool)
MCX311A-XCAT 10GB

For the hard drives it isn't easy to come up with information, I may be totally off track here but the OOS4000G seem to be seagate barracudas and it seems they may be SMR drives.
I wouldn't put high hopes on that 10G NIC either. networking primer

Running Crystal Disk Mark on an iSCSI drive had putrid results,

Can you elaborate on that please?

Could be the NIC, could be SMR drives, could be false expectations, ...

hence the reason for adding the support VDEVS. But this is where I need the advice.

For pure archival storage, what performance do you expect?

SLOG use case you can determine for yourself if you need a SLOG device. The ZIL exists on your pool anyway, that's not a separate device. This also looks useful. I have no experience with iscsi, but for archival storage I do not think this is needed.

From the ZFS primer:

As a general rule, L2ARC should not be added to a system with less than 64 GB of RAM, and the size of an L2ARC should not exceed five times the amount of RAM. In some cases, it may be more efficient to have two separate pools: one on SSDs for active data, and another on hard drives for rarely used content.

You would not benefit from a cache vdev in my opinion.

What I would consider, depending on what you are planning to do with your VMs. If you go for striped mirrors on that SSD pool you can increase the performance there (if needed). 3x mirror vdevs is "only" 800 GB more for redudancy over your current raudz2 setup. If you are content with your performance though, don't touch that. It may be less resilient than a raidz2 setup on top of needing more parity.

 

[Davvo]

• Your use case (archive storage for large files) doesn't apparently justify the use of L2ARC (what you called "cache") VDEV.
• The SLOG VDEV might be useful depending on the type of VM you are using (iSCSI/block storage).
• For iSCSI and block storage you want to use MIRROR VDEVs.
  

  Resource - The path to success for block storage

  ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

  www.truenas.com

  Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

  ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

  www.truenas.com
• You do not want to use those HDDs: you want NAS/enterprise drives designed to run 24/7: examples are WD's RED PLUS/PRO and Seagate's Ironwolf/Ironwolf PRO models. If you want to take the risk in home use, look for refurbished drives but DO NOT buy USED DRIVES.
• You do not require a boot pool in a mirrored VDEV: slap just one drive, then have a config backup regularly... if something happens, you install TN to the new drive, import your pool and your config back, and you are done; if you require HA, please read the following resource.
  

  Highly Available Boot Pool Strategy

  I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

  www.truenas.com
• SSDs are more reliable than HDDs, you do not require the same amount of redundancy... a RAIDZ1 would serve just fine, IF that's the proper config you need. Please read the following resource.
  
  

  ZFS Storage Pool Layout

  This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

  www.truenas.com
• It appears to me that the fan speed issue has been resolved... it's a pretty common issue, and there is a resource about it.
  

  How To: Change IPMI Sensor Thresholds using ipmitool

  Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

  www.truenas.com
• Your desired M.2 adapter card is really overpriced, you can find similar ones at 1/3 of the price.
• If you tell us more about your use case and your expected performance we can help you. You should test your pool's performance using fio, and you can use jgreco's brun-in script to get a sample of your drives' maximum performance.
  

  solnet-array-test

  Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

  www.truenas.com
• I strongly suggest the use of joe's multi_report script for SMART monitoring and automated config backup.
  
  

  Multi_report.sh version for Core and Scale

  This resource was originally created by user: @joeschmuck on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/multi_report-sh-version-for-core-and-scale.179/download Multi-Report Tool to chart and record statistical data on your drives. This will email the...

  www.truenas.com
• A reading of the following resource about 10G is also suggested.
  

  Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

  Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

  www.truenas.com

 

[Ebedorian]

Wow, these responses are great. Thanks for the feedback and the links to other resources. I'll try to share some more useful information once I've got TrueNAS Scale up and running again.

I guess I did something else wrong,

I can't make heads or tails of this.

 

[Davvo]

You have boot pool issues. Try connecting only one drive.

 

[Ebedorian]

You have boot pool issues. Try connecting only one drive.

Tried that, no change.

 

[Davvo]

Tried that, no change.

Try the other? If that doesn't work, reinstall.

 

[Ebedorian]

Thanks, Davvo. I've reinstalled TrueNAS Scale on an older Crucial MX100 256GB SSD. It doesn't appear that either one of the pair of
Silicon Power 128GB SSD 3D NAND A55 SLC SSDs I have been using are any good. Thanks for the help guys. I'll post updates on this build here.

 

[Ebedorian]

So, I am somewhat shocked by the SMB Share performance. I am using

My Current Build includes the following:

Supermicro X11SSL-F
Intel Xeon E3-1270 v5 @ 3.6 GHz
SST-NT07-115X-USA (CPU Cooler)
64GB Micron Crucial DDR4 (4X16GB) 2133MHz UDIMM RAM
Noctua NF-A6x25 PWM x 2
Rosewill RSV-Z3100U 3U Server Chassis
Crucial MX100 256GB SSD (boot drive, with 16GB swap)
SAMSUNG 845DC 800GB PRO MLC x 4 (SSD-Pool)
WL4000GSA6472E (4 x OOS4000G and 1 x ST4000NM0033) (OOS-Pool)
MCX311A-XCAT 10GB
9200-8I IT (HBA)

Is the performance due to the type of share, 10G NIC, or the fact that it was a short file transfer? I'd love to understand this a little better. Let me know if you need additional information to furnish an explanation.

P.S. The PC I ran this from is also connected to my network via a 10G NIC.
I am definitely not using swap.