Home NAS server

[darkness]

Hi, some time ago I've started thinking about NAS server for home puprose (backups, data, photos, watching photos/movies, maybe vm). I want to synchronize pc's and androids for constant upload when f.e. new photo's arrived. Firstly I think about synology/qnap but that will be to easy :D I hope that truenas learn me something new :)

To the point.. Firstly I'll buy mobo and cpu, from my reasearch i choose (I was guided by the availability in my country and the price):

MOBO: SuperMicro X11SSL-F ( new [~300$] or used [~150$] in my country)
CPU: Intel Pentium G4560 (used, for start and if will be not enough, then buy I'll buy f.e. xeon e3-1230v6)

I just have some doubt's about age of MOBO and cpu - it isn't to old in 2023/2024?

 

[darkness]

OK, so I decide to get another build and change my mind about usage:

MOBO: MBD-X10SRM-TF-O
CPU: Xeon e5 2699v4
RAM: 2x32GB RDIMM ECC
STORAGE: 3x4tb HDD Iron Wolf + 2x1tb SSD WD Green + 1x256gb NVME
PSU: Be quiet! Pure Power 11 600W Gold
CASE: Fractal Design Node 804

I'll use it as homelab with ProxMox HV. Truenas will be as VM and run 24/7.

Storage config:

SSD -> Proxmox + VMs with RAID1
HDD -> Data with 3-way mirror
NVME -> Truenas cache

What do you think about this config? Now it's time to study about Truenas - if you have any tips, I'm open to learn ;)

 

[NugentS]

[Not sure if I am correct here - as I haven't tried]
You need to pass through the HDD controller to TrueNAS. This will likley contain all the HDD's which means that ProxMox won't have any disks to boot from, or in your case for VM's

I would have thought you would need an HBA so that you can pass the HBA and all its disks through to TrueNAS leaving any motherboard disks to proxmox - or I guess the otherway around.

Oh and the cache won't do anything as it won't have enough memory to work with. Leave it out - it might even slow things down

 

[Davvo]

Efficency wise the X10 platforms are really starting to show their age.
If you want to virtualize please read the following resources. (spoiler, @NugentS is right)

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Regarding your cache... If you mean L2ARC, you should start considering using it only after you have 64GB of ARC (dedicated to TrueNAS; 128GB since we talk about SCALE); if you mean a SLOG, which could be vital looking at your intended use, please read the following resources.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you...

www.truenas.com

 

[darkness]

Thanks for resources :) I'm still in 'getting knowlege' process and I'm considering new build (with new parts):

MOBO: ASRock X570D4U
CPU: Ryzen 5 3600 (I use it in PC so just swap)
RAM: 128GB RAM ECC UDIMM
DISKS: 3x4TB (3way mirror) + 2xnvme pci-e 4.0 ( for VM/boot in mirror ) / 120gb ssd ( if truenas baremetal boot )
PSU: Be quiet! Pure Power 11 500W Gold
GPU: nvidia gt 710 (19W TDP, f.e. to with plex)

These fit me in all ways (budget, average power consumption, power of machine, still 'new' parts and I like ryzen).

This mobo is up to 105W TDP so I can swap to ryzen with 16c/24t in furture.

Puprpose of this machine:
Mainly NAS (work always)
VM win 10 (work always)
few VMs/services like plex, pfsense, IDS/IPS, logserver, home assistans, security testing etc. (all of these are linux)... (work not always, change dynamicly, testing)

I think it's good build for home purpose lab but still consider about proxmox/trunas baremetal. For only NAS home pupropse I think it's overkill.
After when I understood that truenas fells better on bare metal - I consider that.... I'll create VMs in truenas - why not :) How do you think about this build and how it'll work on truenas?

 

[darkness]

Efficency wise the X10 platforms are really starting to show their age.
If you want to virtualize please read the following resources. (spoiler, @NugentS is right)

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Regarding your cache... If you mean L2ARC, you should start considering using it only after you have 64GB of ARC (dedicated to TrueNAS; 128GB since we talk about SCALE); if you mean a SLOG, which could be vital looking at your intended use, please read the following resources.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you...

www.truenas.com

Assume that I'll use hardware above only for TrueNas and media stuff like plex, nextcloud etc.. not for testing / labs purpose as I mention in post above.. I tested proxmox/truenas few days and I've comming to conclusion that better not mixed these two worlds.. SCALE it's not stable with vms as I suposed for lab purpose i think.

What is the best way to use this 2x nvme pci-e 4.0 in my setup above and what capacity for speed up write/read? I've started reading your resorces about cache but for now is black magic, need some time to understand it.

 

[Davvo]

What is the best way to use this 2x nvme pci-e 4.0 in my setup above and what capacity for speed up write/read? I've started reading your resorces about cache but for now is black magic, need some time to understand it.

I would use them for the app/vm pool (which usually is also made the system's dataset pool) in a 2-way mirror configuration. Capacity is up to you, and as long as it is nvme you won't encounter any particular bottleneck with plex or nextcloud (since the data storage part of such apps, like your plex's library, should be put in the HDD pool). I don't think there is really any merit in going bigger than 1TB, you could even just use 500GB ones depending on your apps' needs.

Since your motherboard of choice has an internal USB, you could use a high quality USB flash as your boot drive, leaving the 3rd SSD for L2ARC if you want to boost your reading performance: in that case, use a drive that's 250GB for maximum performance or up to 500GB (best ratio of ARC to L2ARC is 1:4 iirc, and you can go up to 1:8 with growing performance penalties) or partition a bigger drive and use only that partition. Ideally you should do this if you see your ARC's hits to be at the very least lower than 90%.​

 

[NugentS]

Efficency wise the X10 platforms are really starting to show their age.

Sniff - all of my servers (TrueNAS & ESX) are X10, apart from the single Atom 3758

 

[darkness]

I see, thanks Davvo!

Now it's time to choose something.. Firstly I want go with one machine to everything (so I consider about X10 build). Now, after tests I know that better is have NAS separate. As I wrote in first post I have posibility to get X11SSL from second hand - @Davoo - your build will be enough these days for only NAS purpose (with 32/64gb ram)?

 

[NugentS]

For just "normal" NAS, 32/64 is great.
Adding virtual machines, containers, jails and massive storage may want a bit more

 

[Davvo]

64GB will take you a long way.

 

[darkness]

Thanks, so I think I stay with X11 build ( less expensive and do job ). I'll learn on it and if needed upgrade for something powerful. This ram will be ok: KSM26ED8/16HD ??

 

[Etorix]

I would have thought you would need an HBA so that you can pass the HBA and all its disks through to TrueNAS leaving any motherboard disks to proxmox - or I guess the otherway around.

It should be possible to pass the chipset SATA controller and boot from NVMe—just forget L2ARC, whose utility is not yet established.

I would bump the PSU to accomodate 8-10 HDDs, as the case allows, and put 4 HDDs as a pair of mirrors. The cages just want to be filled with four drives.

As I wrote in first post I have posibility to get X11SSL from second hand - @Davoo - your build will be enough these days for only NAS purpose (with 32/64gb ram)?

Yes, but go straight for a Xeon E3v5/6. These should be dirt cheap by now so there's no reason to hang out with a Pentium or even a Core i3.

 

[darkness]

I would bump the PSU to accomodate 8-10 HDDs, as the case allows, and put 4 HDDs as a pair of mirrors. The cages just want to be filled with four drives.

Be quiet 500w have 6x sata and 3x molex which gives next 3x sata ports (with adapters). 500W Will be enough?
With pairs you mean 2x(2x4tb)?

Yes, but go straight for a Xeon E3v5/6.

Thanks!

 

[Etorix]

Be quiet 500w have 6x sata and 3x molex which gives next 3x sata ports (with adapters). 500W Will be enough?

By the PSU sizing guidance, 650 W would be safer. 750-850 for a Xeon E5 (W-2000 if you find a X11SRM-(V)F).
Better spend a little more now than find out later that filling the case with drives pushed the PSU on the edge.

With pairs you mean 2x(2x4tb)?

Yes, striped mirrors. These cages are best handled as a single entity (extra bonus when using 4-way breakout cables rather than bundles of regular SATA cables).

 

[darkness]

OK, so I take 650W, you're right :)

Xeon E3-1230v6 and ram Kingston UDIMM ECC 16GB DDR4 2Rx8 Hynix D 2666MHz ( KSM26ED8/16HD ) - will be okey ?

 

[Davvo]

They sell this as the 16GB 2666MHz RAM compatible with the one in QVL.

https://www.amazon.com/dp/B09G88HYN5

 

[Etorix]

I've yet to find an ECC module from Micron/Samsung/SK Hynix that didn't work in just any compatible server motherboard. But if one can actually find a module on the QVL for a non-eye-watering price…

 

[Davvo]

I've yet to find an ECC module from Micron/Samsung/SK Hynix that didn't work in just any compatible server motherboard.

I can't use the second channel in my system... likely due to a slot being broken

 

[darkness]

They sell this as the 16GB 2666MHz RAM compatible with the one in QVL.

https://www.amazon.com/dp/B09G88HYN5

This looks really fancy ;D

I see that many 8+ sata PSU are modular. It's a problem or better search no modular for server? going further.. it's possible to expand/change cables in modular? f.e PSU have 9 SATA -> 1x4 sata / 3xsata + molex / 2xsata + something.. what if buy 4xsata cables and replace that sh!t?:)

In X11ssl-f is possible to expand sata port from motherboard? Or only PCI-e comes in game?