Resources List including Detailed Hardware and System Build Notes (plus new user advice / help)

Resources List including Detailed Hardware and System Build Notes (plus new user advice / help) v1.24

CISCO Datacenter.jpg

TrueNAS is not as simple as videos make it look, though it is fairly easy to get a system running if you read the documentation and don't color outside the lines.

Briefly:
  • Use TrueNAS, on bare metal server equipment, with as few modifications as possible, if your data is really valuable.
  • Use something other than TrueNAS that is more suited to your needs if you want a custom solution.
TrueNAS has a solid reputation for being a data serving powerhouse for a reason: It's designed to be in a Datacenter for the purpose of attaching a crap-ton of storage to the network. Using it for something other than what it is designed for can be a recipe for disaster. Using it for home storage is...well, you should really have a need first....and I think that's why people are trying to shoehorn it into all these "also" situations, like "also gaming" and "also a Virtual Machine server" and "also works on hardware designed to run Windows 3.11."

Seriously, if you don't have an actual need, [used] server-grade hardware, and way more time to invest than you're expecting, load up Ubuntu Server and work on a great data backup strategy--you'll be way farther ahead, have easy-to-understand guides, and your electricity bill will be far less. If you have to run Proxmox: set up file shares, Proxmox can do that. If you really have to have a lab in your home because like me you live in an arctic climate where it's dark 23.75 hours of the day and the high is -12°F and you need the heat so your coffee cup isn't simply holding a solid block of ice, set up TrueNAS on one box (with ECC memory and a real NIC of course), and Proxmox on another box (with ECC memory and a real NIC of course), and a workstation or two (with ECC memory and a real NIC--of course!), and a crap router so nothing works the way it should.*

----
* (I have a good router, I'm kidding you.)

If you want to save yourself lots of money and loads of time, the top of the page has Documentation and Resources tabs which contain extremely helpful information like the TrueNAS manuals. There's also a Search tool at the upper-right that works extremely well. If you understand the system and requirements before jumping in you'll save money, time, and frustration.

As general advice, if you run TrueNAS as a NAS and don't add things to it it tends to run fine and without issue, though I suggest using an Uninterruptible Power Supply and NUT.

Precautions: (dreaming big is fine, but let's nail down a few things that go badly for new-to-TrueNAS users)
  • Adding Virtual Machines, Jails, Containers, TrueCharts, scripts, and especially running TrueNAS in a virtual environment (Proxmox) is where people get themselves into trouble and start losing data. It appears most initial problems occur due to network bridging not being configured properly, with drive access perhaps being the second biggest issue. (keep reading)
  • If you saw a video with TrueNAS 1) in a VM, 2) being used as a gaming server, or 3) Plex/Jellyfin server: You're probably in for more work than you expect. First build TrueNAS as a NAS on server-grade hardware, and when that's tested and stable incrementally expand it, or better yet use a different platform completely as TrueNAS doesn't work spectacularly for any of those purposes (though it does work well as NAS, not surprisingly). (keep reading)
  • TrueNAS CORE (BSD) is faster and more stable than SCALE (Linux), so consider running CORE. There's almost no interaction with the operating system anyway.
  • DO NOT use gaming hardware for a NAS; TrueNAS is a Datacenter-grade platform and will not run well on gaming hardware just like games don't run well on server hardware.
  • If you're trying to repurpose an old desktop system use Ubuntu Server (possibly with ZFS though that's often not the best choice for older systems, use LVM instead), not TrueNAS. Videos showing TrueNAS working for a day of filming before tearing down the system for another build is entertaining, but this isn't good for long-term stability (or even short-term stability). TrueNAS uses hardware in ways desktops were not designed for and they'll eventually break and your data will be lost forever. Use Ubuntu, --or Gentoo if you want to wander into the deep end of the pool. Repurposing a desktop can get real expensive real quick and you're better off buying used server equipment--you'll save money and time.
    (Note that Ubuntu Server and TrueNAS have a text-only terminal attached, unlike Windows or Ubuntu which are graphical and use a mouse.)
  • Is TrueNAS "a lot cheaper and as easy to use as a Synology NAS?" Well, cheaper...that depends on Total Cost of Ownership, including the time you invest in learning any system. It's hard to beat Synology if you want an easy-to-use home system that you just plug in and click around a bit, though Synology gets expensive when storing a lot of data.
  • TrueNAS as a VM Under Proxmox or VMWare: Don't do this, Proxmox can operate as a SMB Server for Windows or NSF Server for Linux. VMWare sharing methods: (link1, link2)
  • Minecraft: This isn't a good idea, Minecraft is best run on an old desktop where it will run far faster and smoother--and without the cost of server hardware TrueNAS requres. The ZFS filesystem TrueNAS uses can be used there but you don't want to do it as it will wreck performance.
  • Steam Games and Deduplication: If you have the money, it's doable. Otherwise...
  • Plex/Jellyfin: This isn't a great idea for many reasons, though it can be done. Transcoding is somewhat pointless as even cheap phones can display 4K data streams, so a Raspberry Pi [alternative] with Kodi is enough to stream video and will use far less power and be much quieter. Both work great under Proxmox (which is designed to run containers), however be prepared for a lot of hardware and configuration headaches you'll most likely face if trying to do this with TrueNAS.
If the above suggestions don't meet your needs because you have valuable data and truckloads of it, and you're convinced you need an industrial storage solution on server-grade equipment to handle it (without breaking the bank), then TrueNAS is probably what you want.


(If you're reading this on a phone, turn it sideways and read in Landscape Mode, longer lines will make more sense)

Fundamentals

Basic Understanding​

- Search Before Posting (get immediate answers!)
- About YouTube Videos and TrueNAS
- Terminology and Abbreviations
- File System Types
- The 'Hidden' Cost of Using ZFS for Your Home NAS
- Slideshow explaining ZFS, VDev, zpool, ZIL and L2ARC for noobs
- ZFS Introduction
- ZFS Pool Layout
- ZFS Capacity Calculator
- RAID-Z1 is Dangerous (well, -Z0 is worse)
- Universal Serial Bus of Doom (why USB drives aren't used in a NAS)
- ECC Memory: Do I have to??? (explained in a build thread)
- Ryzen "may-be ECC" vs Xenon 4-bit ECC (normal Intel ECC is 1-bit)
- RAID Controllers won't work; Use a Host Bus Adapter (HBA) (realllly important, even though...)
- Understanding Serial Attached SCSI (SAS)
- Serial AT Attachment (SATA) Port Multipliers [of Trouble]
- System Manuals (each version's manual is under Documentation at the top of the page)

Hardware​

Range of Builds:
- Under 30W Systems (The most economical systems / Die sparsamsten Systeme)
- Building a Power Efficient Home Server (23W D-I-Y full-size server, YouTube - Wolfgang's Channel)
- General Overview (UNRAID, though applies to TrueNAS as well)
- Super Powerful Micro Server (Supermicro Superserver E300-9D-8CN8TP about $2,200 USD 04/2023)
- LattePanda Sigma - Hackable x86 Windows/Linux Single Board Computer/Server (Dave's Garage video, $580 05/2023)
Required Hardware:
- Hardware Guide (READ THIS BEFORE BUYING ANYTHING!!!)
Storage:
- LSI HBA Card Insight (and 'how to frugal')
- Fake LSI Cards (eBay is full of them)​
- Westerm Digital Red / SMR Drive Incompatibility with ZFS (ZFS eats them for lunch)
- Hard Disk Drive Price/Performance Analysis
- 5,400 RPM Drive Speed (a good thing)
- Serial Attatched SCSI (SAS) Informational Videos
- Fibre Channel (playin' with the big boys) QLogic QLE2564, QLogic QLE2694-SR.
Power:
- Power Supply Unit (PSU) Selection and Sizing (also important)
- Current Ripple and Life Calculation (triple your ripple and say goodbye to your power supply) (explained, filter considerations)
- [Official] CPU Heatsink Recommendations
Network:
- Realtek Network Adapters ARE Bad (why to avoid cheap Ethernet)
- 10GBASE-T Over Cat6 vs Cat6a vs Cat7 Cabling
CPU:
- Intel Atom vs CORE
- Intel processor comparison/specifications
- Note: The Intel Quark SoC, lower-power chips than Atom, were discontinued in 2019.
Low Power Mainboard:
- Low power Mainboard & CPU
- 'On the frugal': Mainboard: Supermicro X9SCM-F, CPU: Intel Xeon E3-1230
- More Power: Supermicro A2SDi-2C-HLN4F / A2SDi-TP8F, X10SDV-TLN4F (note)
Insight:
- How to Buy Used Hardware (eBay hard drives)
- Zebras All the Way Down (why simplicity is important)

Example Systems​

- ESXi and Plex on a Node 304
- Pheran's 80TB build (with photos)
- beard-strokers

Building​

- Choosing a TrueNAS Version
- Building, Burn-In, and Reliability Testing
- Hard Drive Reliability Testing
- Hard Drive Troubleshooting Guide
- SFF-8644 External HD MiniSAS Connectors: Not necessarily difficult to plug them in wrong.
- Solnet Array Test Script

Configuration​

- Leave the Boot Drive Alone (do not partition it for data storage)
- Setting the Time Correctly (NPT drift)
- Structuring Datasets
- Useful Commands
- Useful Settings
- Supermicro's Fans Fix via IPMI Sensor Thresholds
- Removing SMB1, NetBIOS, and WINS
- rsync fails when copying Windows datasets

Maintenance​

- Backups
- Drive Life Expectancy (about 50,000 hours for Enterprise drives)
- Hard Disk Drive Study (Backblaze)
- Solid State Drive (Study (Backblaze)) (Article)​
- Replacing [Failed] Drives
- System Update Procedure - !!! READ THIS FIRST !!!
- Managing Pools
- How do files *ONLY IN A SNAPSHOT* get corrupted?
- How to diagnose strange crashes
- How Thermal Paste Works and How to Apply It (Intel)

FEA9ZMYGE05CTLG[1].jpg FLMJI3DGE05CTLS[1].jpg FX7ARH1GE04Z4IV[1].jpg


DO NOT add the following "stuff" to your system until you actually need to as it will unnecessarily complicate things and possibly break other stuff:

Ease of Use

- S.M.A.R.T. Reporting Script (multi_report.sh for CORE and SCALE)
- HDD Spindown Timer Script
- Pool Rebalancing Script
- FreeNAS Scripts
- ZFS transfers with netcat on trusted networks
* bc the basic calculator is in CORE and the script will work there out of the box; SCALE however does not include bc in the base system (yet...this may change) but can be added if you want to use the script there.

Advanced

Storage​

LSI/Avago/Broadcom Host Bus Adapters:
- LSIutil (fixing your LSI card)
- Crossflashing LSI 9211/9300/9311 HBA (and variants)
- Erasing the LSI 9207-8i BIOS (not the firmware) for Faster Boot
Data Deduplication:
- How to Kill Performance (and raid your wallet) With Deduplication (why you shouldn't use de-dup)
Bottleneck Reduction:
- What are IOPS ???? (un-bogging drive-bound systems)
- Simple SLOG analogy
- ZIL & SLOG (additional info) (sync write backup & using a separate backup device)
- SLOG Benchmarking (additional info)
- Fusion Pools (ZFS allocation classes, ZFS special vdevs, and metadata vdevs)
- Nope: Boot, SLOG and Fusion Pool cannot be on the same mirror (how to break everything)
- Hierarchical Storage Management (also known as Tiered storage)
- Open Source: Online Hierarchical Storage Manager (under development, whitepaper)
Boot Resiliency:
- Highly Available Boot Pool (3 SSDs for redundancy)
Large-Scale RAID:
- dRAID (CLI currently in SCALE, GUI in upcoming SCALE Cobia)

iSCSI / Block Storage​

- How to Configure Block Storage (not file shares)
- Why iSCSI often requires more resources for the same result
- Why Use Mirrors for Block Storage

Network​

SAMBA Performance:
- Hardware
- Tuning
- SMB Multichannel
Bandwidth:
- 10 Gbps Ethernet Primer (a diet for your wallet)
- 40 Gbps Mellanox InfiniBand (about, notes, also note the switches are different than Ethernet and a bit spendy)
Switch, Firewall:
- Ultimate CHEAP 2.5 & 10GbE Switch Guide (generally under $200 in 1st quarter 2023)
- Even Cheaper!!! (generally under $125 in 2nd quarter 2023, see notes)
- CHEAP powerhouse 2.5GbE Mini PC Router/Firewall ($300 in 1st quarter 2023)
- MikroTik CRS518 (16) 25GbE & (2) 100GbE port Low Power Switch ($600 in 4th quarter 2022)
Special-Use Cases (Fast Networking by Breaking Convention):
- Understanding LACP (Link Aggregation)
- Remote Direct Memory Access (RDMA)
- Overview (Slideshow) (Outline)​
- How RDMA Works and Bypassing RDMA Security (USENIX Security 2021 video)

UPS​

- Network UPS Tools (NUT support)
- Energenie UPS

Dangerous Territory​

- Nextcloud-iocage-FreeNAS (maybe TrueNAS)
- Plex (I think...I didn't try it, your mileage may vary)
- Virtualize TrueNAS (iXsystems)

Applications​

TrueNAS is an excellent NAS appliance, and there are sound reasons to stretch that further, though doing so is not without inherent risk. While the linked threads contain some "bold exaggerations" they also contain valuable information on what to expect and how to keep your system out of trouble while capitolizing on the resources.
- TrueCharts: Things may not go as expected (so plan ahead)
- Community App Repository (similar situation to TrueCharts, plan ahead)



CREDIT: This list was originally based on @Chris Moore's Links to useful threads and @Davvo's list (in his signature).
DISCLAIMER: What you do is up to you, opinions and comments herein non-withstanding. Comments are based on enterprise-level hardware reliability statistics and critical data retention, which to different people means different things and holds different values. You could conceivably run TrueNAS in a VM without ECC memory on generic consumer hardware for years without problems, buuut that's not really "best practice." Sometimes though, "you gotta do what you gotta do and throw caution to the wind." I'll just mention you might want to keep on top of your backups... :wink:
Author
WI_Hedgehog
Downloads
656
Views
6,677
First release
Last update
Rating
5.00 star(s) 3 ratings

More resources from WI_Hedgehog

Latest updates

  1. v1.20

    Added: more TrueNAS CORE links more Raspberry Pi alternatives Fundamentals Hardware low-power...
  2. v1.10

    Added: more Proxmox information Steam Games Fibre Channel more Power Supply information...
  3. v1.0 Major Overhaul

    @Davvo's signature was rolled into Hedgehog's signature and expanded upon, when that grew too...

Latest reviews

When I first opened this resource up a few weeks ago I thought to myself, "self, a lot of thought and effort went into this posting, this must have took many hours to type in." and I felt a tingle at my feet, it was the dog wanting attention. And I came to the conclusion that I really like the huge list of references you have here. The are well laid out and make sense.

This is not only great for a beginner, but yes, for an experienced user as well.

I totally agree with your first statement. "TrueNAS is not as simple as the videos make it look." But it's not that difficult if you put in some level of effort.

Thanks for this great list and I hope you are able to support and update the resource for many years to come.

And yes, maybe I'm biased as well but we do have a great community willing to share knowledge and help others who asking simple or very difficult and challenging questions.
Another example of why the TrueNAS Community is one of the best. ( I might be biased)

Thanks for taking the time to write this up, this is a must-read for any new or experienced TrueNAS users.
TrueNAS 101, a must read.
Top