All SSD TrueNAS build (homelab)

[Archaeopteryx]

Hi there,

I'm since many years a user of FreeNAS and apart from the finicky configuration in the beginning I typically have uptimes>1 year (and then planned shutdowns). Thanks for offering such a nice and powerful solution to the home user.

___
History:

So far I was using the following system:
- CPU: Intel Xeon E3-1230 v3, 4C/8T, 3.30-3.70GHz, boxed (BX80646E31230V3)
- 2 x 8 GB ECC RAM, Kingston ValueRAM DIMM 8GB, DDR3-1333, CL9, ECC (KVR1333D3E9S/8G)
- Mainboard: Supermicro X10SLM+-F bulk (MBD-X10SLM+-F-B)
- Power supply: be quiet! Straight Power E9 400W ATX 2.4 (E9-400W/BN190)
- Case: Fractal Design Define Mini

The HDDs were mainly WD green (with deactivated power saving), 6 x 3 TB and my boot usb stick were two SanDisk Extreme 16 GB in mirrored configuration. Recently one of the boot USB Sticks failed and also the system reported that it could fix 8K of data after a scrub, so I think I don't have the full year after one after another disk will fail. Thus, I am planning my new build.
___
Goals:

The NAS was so far very helpful to be a central hub for media (video, music, pictures) and backups of all the systems in the household.
A new goal would be to offload my raw pictures onto the NAS and process them on the NAS, too. The pictures can be quite big (~40 MB) and needs to be processed by a special software (So many times reading/writing quite big chunks of data), so I want to keep myself the option to install a 10 GbE card.
___
Current plan:

Since the rest of the hardware works flawless, I want to keep the mainboard, CPU and RAM and case. Since the NAS will be placed in the living room, I want to exchange the HDDs by SSDs, exactly the Crucial MX500 2TB, SATA (CT2000MX500SSD1 / CT2000MX500SSD1Z)
I already got 4 of them for a good price and plan to but at least 2 more for a RAID-Z2 configuration.

Since the mainboard is limited to 6 SATA ports (2 of them even SATA 3 GB/s ) this will be the max. amount of disks I can attach, but if I need more space I could some SAS PCI cards.

I have another SSD (Samsung 860 Evo, 250 GB) which I want to use as a boot medium instead of the not so optimal USB sticks, but I want to use the internal USB port and a USB to SATA adapter (supports UDMA) since I am running out of SATA ports.

As software I want to use the new TrueNAS 12.
___
Questions:

1. Makes a RAID-Z2 configuration for 6 SSDs (later maybe 8 SSDs) sense? So far, the argumentation was that for HDDs above 2 TB or so the rebuilding is very stressful and takes a while / the speed of them is quite slow, so RAID-Z1 was not recommended, but is that still the case for the fast SSDs? Or is it rather limited by the processor speed which needs to check all the checksums?

2. Does TrueNAS support TRIM or do I have to make any special parameter in tunable that the system gets the best performance / runs smoothly with an all-SSD pool? I found this video from SpaceRex where he sets a tunable for the 4K sectors(if the SSD supports that), any others needed?

3. With my current HDD build I can already saturate the 1 GBit/s, but with all the SSD speed available I would like to go the 10 GbE route, which card can be recommended to play along nicely with TrueNAS? I plan to make a direct connection to my windows PC (where I also need to buy a 10 GbE card)

Thanks for reading and your help

 

[ornias]

Don't forget:
Raidz1 also means you can not recover any blocks if a block happens to be corrupted during rebuild. It isn't "just" about total pool failure.

 

[Archaeopteryx]

Yes, the "sweet sirens" of more available storage space are always calling, but I will go for Raidz2. Especially when I go from 6 SSDs to 8 SSDs I will have a better storage ratio for Raidz2.

 

[Etorix]

1. The reason to avoid RAIDZ1 above 1 TB is the URE (unrecoverable error rate) of 10E-14 or 10E-15 of HDDs. SDDs typically have URE of 10E-17, so RAIDZ1 should be fine. You have a backup on another NAS, haven't you?

3. Chelsio 520 cards work nicely with TrueNAS and have official drivers for many OSs. The only issue is to find a second hand bargain on them.

 

[Archaeopteryx]

Yes, I have an extra backup on a single external HDD. At the moment I transfer it over the network to my work PC and then over USB3, but I'm also planning to do a more automatized backup off-location (at a friends place) with a Raspberry PI to just sync file changes, but this is project on its own

 

[Archaeopteryx]

Quick update:

Server is running since a few weeks. I tested all SSDs (6 x Crucial M500, 2 TB new and 1 x Samsung 860 EVO, 250 GB) and also exchanged a broken case fan with BeQuiet Pure Wings 2 (120 mm) in the server. Additionally, the stock CPU cooler got exchanged.

The typical hiccups and configurations errors are like always, for example IPMI from Supermicro still uses a very old Java Applet and somehow the system got quite loud with pulsating fans in the case - kind of the opposite of my goal. It turned out that they were so low-rpm running that the Supermicro mainboard thinks they are failing and puts all other fans to 100 %. Then even the "failing" fan picks up speed and all fans are, according to the mainboard, back online and it settles down to a lower rpm speed, where it thinks that a fan is failing - rinse and repeat.

I first had some ipmi program for windows which was not successful, in the end I changed the fan threshold values via the console in the TrueNAS webgui, which worked. Since then, the server is really noise free, can hear a needle drop next to it. So from this standpoint full success.

Concerning the speed, the 1GbE connection is clearly limiting, but more than sufficient at the moment. If necessary, I will upgrade it to 10 GbE.


Another thing was that the server somehow lost the DNS after the 12.1 update, but setting the nameserver to 8.8.8.8 solved this issue, too. (Normally the nameserver from the router should be sufficient to resolve all internet address, but somehow it did not work anymore)

So, the typical configuration effort in the first week, but this was expected because afterward I know from experience it is very smooth sailing with FreeNAS/TrueNAS

 

[Etorix]

Well done! You may want to have a look at this script for the fans: PID fan controller Perl script