Can't install TrueNAS (Scale OR Core)

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
I used a case rear slot fan controller (not PCIe plugin) with a variable resistor so I can easily adjust it for temp and noise. No automated adjustment though. But I graph the drive temps in Graphana so I can see if I need to turn things up a bit.

The current hottest drive is 27
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
@chuck32 Have you looked into fan control scripts?
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
Chelsio T520: OK
If you go for Solarflare, try to find a 7000-series (PCIe 3.0) rather than 5000/6000 (PCIe 2.0). PCIe 2.0 is enough bandwidth, but PCIe 3.0 cannot hurt.
If you go for a -16i HBA, the 9305-16i uses less power than the 9300-16i (9300-16i is two 8i controllers on the same cards, 9305 is a single, more eneergu efficient, controller).
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
Chelsio T520: OK
If you go for Solarflare, try to find a 7000-series (PCIe 3.0) rather than 5000/6000 (PCIe 2.0). PCIe 2.0 is enough bandwidth, but PCIe 3.0 cannot hurt.
If you go for a -16i HBA, the 9305-16i uses less power than the 9300-16i (9300-16i is two 8i controllers on the same cards, 9305 is a single, more eneergu efficient, controller).
OK, I'll buy this chelsio then :smile:
And I'll go with the 9300-8i (+ 8 HDD connected on the MB with 4x SATA 3 + 1 mini SAS), do you know if there is any difference between those 3 options ("LSI 9300-8I", "Supermicro 9300-8i IT Mode" or "Fujitsu 9300-8i IT Mode" or this 9302-8i), since I would obviously like to avoid spending money unnecessarily.

Thanks again !
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
A 9300-8i should be a 9300-8i… I suppose you can decide on price or on where the connectors are placed for ease of setup.
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
Hey guys (happy new year) !

I've been able to test the system with 64 Go of (DDR5) ECC RAM, and the installation went flawlessly ! :)
(Also received the LSI and Chelsio card, their installed but not tested yet)

The system isn't done yet, I'm waiting more RAM, since you told me I required at least 128 Go, I'll try this out, and I hope I'll not require more because of the price point (250 € per 64 Go)... and if 2x 64 Go isn't enough, I'll likely have to do 4 x 64 Go, outch.

Now I'd like to have your recommendations, shall I go with 1 vded of 16 HDD in RAIDz3 or 2 vded of 8 HDD in RAIDz2 ?

Thanks you
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
16 wide is too wide.
2 vdevs, RaidZ2 is a much better idea.

Note that I am assuming you are essentiually just storing files on the server. If you want something with high IOPS then Z2, even 2 vdevs is not the way.
 
Last edited:

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
16 wide is too wide.
2 vdevs, RaidZ2 is a much better idea.

Note that I am assuming you are essentiually just storing files on the server. If you want something with high IOPS then Z2, even 2 vdevs is not the way.
I'll store medias (videos).
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
16 wide is too wide.
2 vdevs, RaidZ2 is a much better idea.

Note that I am assuming you are essentiually just storing files on the server. If you want something with high IOPS then Z2, even 2 vdevs is not the way.
So do you think it might require "high IOPS" ? :)
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
To store videos? No.

16 disk wide is just to wide in general for anything. 2x 8 wide is a nice fit.
 

PhilD13

Patron
Joined
Sep 18, 2020
Messages
203
Two Z2's will generally have more performance than a single large one. If you have a number of people accessing things, lots of small files being accessed, then more vdevs is generally better.
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
Two Z2's will generally have more performance than a single large one. If you have a number of people accessing things, lots of small files being accessed, then more vdevs is generally better.
Multiple access at the same time but on video files.
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
The motherboard is too new, and hence too expensive for what it is (PCIe 5.0? no use for that in a NAS). And the last generation Core i9 is way too powerful for what a NAS requires; unless you plan to run CPU-intensive VMs/apps, this is massive overkill.
For a storage plateform with light apps, and with a 600 E budget "for the motherboard", you could have an A2SDi-H-TF instead. Except this one already includes the CPU, 12 SATA ports and a server-grade 10 GbE NIC—and, if you buy second-hand RAM, you can fit it with 128 GB for 120 E (RDIMMs are GREAT).
If you do need an iGPU for transcoding, the recommendation would be an earlier generation: C2x6 chipset and Xeon E-2000, or even a Core i3-8100/9100 going back to Coffee Lake. Less sexy, but powerful enough for a NAS and hopefully lighter on your wallet (although C246 motherboards are now hard to find and so more expensive than they should and than they used to be).

Edit. The above began as generic recommendation. Then I went back up and remembered you want 16*20 TB… 320 TB before taking out redundancy. At these heights, the "1 GB RAM per TB of storage" does not apply strictly, but 128 GB RAM feels like a bare minimum. On UDIMM platforms, a L2ARC would help; on RDIMM platforms, consider 192-256 GB RAM.
You're in a territory where a 1st (or 2nd) generation Xeon Scalable would be quite a reasonable option—put a GPU for transcoding, or just handle that on the CPU.


Urgh! I more or less feared the Aquantia NIC (dubious driver support in TrueNAS; server-grade Chelsio T520 or Solarflare 5122F/6122F/7122F cards go for $50 on eBay) but the "SATA card"… HOLY CRAP! 16 ports on a 4-port controller (ASM 1064, PCIe 3.0x1) and multiple port multipliersDo NOT use that with ZFS! This is litterally guaranteed to destroy your data. Get a SAS HBA instead (LSI 9200/9300 series, or the Dell/HP/IBM equivalents).
Beside, with so many drives, a server enclosure with hot-swap trays on a SAS backplane would make your life easier than cramming 16 drives in a consumer-grade ATX tower (Define 7XL or the like?).

Hi,

I got a problem, you're telling me that I require at least 128 GB of RAM and maybe more but my motherboard is limited to a maximum of 48 gigabyte per slot and I have not been able to find 48 gigabytes RAM stick so I'll have to use either 32 or 64 GB stick.

Now I am wondering if I shouldn't try to go with RAID hardware instead of software, do I require as much as RAM with the hardware RAID ?

It is a problem because of the capacity of my motherboard and also because of the price, if I want to buy 4x 64 gigabyte of RAM (DDR5 ECC) it would cost me more than a thousand euros, and i'm not even sure it will work, if it does, I'll have only 192 GB instead of 256 GB.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
1. Do not go with RAID hardware
2. You have a lot of storage. 128 GB is a very good starting point.

What is your use case - what do you plan on doing with all this storage?
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
1. Do not go with RAID hardware
2. You have a lot of storage. 128 GB is a very good starting point.

What is your use case - what do you plan on doing with all this storage?
2. The problem is... if I go 128 Go, it means I'll get 4x 32 Go, so I couldn't upgrade -easily, if it's not enough- to 192 Go ECC or if I do, I'll have to rebuy 4x 64 Go...

A media center (movies, series, etc.).
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
How many apps (containers) are you planning on running - or this primarily storage?

Cos 128GB will be absolutely fine for a media server providing you don't have vast numbers of users
 

WoisWoi

Dabbler
Joined
Nov 20, 2023
Messages
32
How many apps (containers) are you planning on running - or this primarily storage?

Cos 128GB will be absolutely fine for a media server providing you don't have vast numbers of users

About 20 containers, I think.

The thing is I have quite a vast number of users (family, friends, family friends, etc.), in peak I can have about 30 simultaneous users, and I guess that I could theoretically have about 50 max.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
All sharing the same location and connected via 1G or 10G Ethernet? Or logged in via VPN and limited by your Internet uplink?
 
Top