My first FreeNAS build, Ryzen1700x

[ezra]

Hello,

Finally saved enough to build myself a nice FreeNAS box. It'll be running at home as a file server/media server/backup station. Build costs around 2000+ euro's.

• Case: Fractal Design Arc Midi R2
• Motherboard: ASrock x370 TaiChi
• CPU: ADM Ryzen 1700x
• CPU FAN: Cooler Master Hyper 212 LED Turbo
• MEM: 2x Crucial 16 GB 2400Mhz ECC non-buff (CT16G4WFD824A, only the 8gb version is in the QVL as ECC but I've read that the 16GB also works)
• PCIE: LSI 9240-8i 8-port SAS SATA LSI00200 IBM M1015
• PSU: CoolerMaster 500W
• Storage: 6x WD RED 3TB with 1x Transcend 32GB SSD370S SLOG (RAIDZ2 All bought from different shops to get different batches)
• 2x Samsung 850 EVO SSD 250GB with 1x Transcend 32GB SSD370S SLOG (ZFS mirror for Jails/VM's)
• 2x Transcend 32GB SSD370S (ZFS mirror. Freenas install with 16GB swapfile)
• Extra:
• Bitfenix recon ( 5x fancontroller with 5x temp sensor),
• 2x 5 inch 6x 2.5 hotswap bay delock
• 4x case fan on top of the 3 stock ones.
• UPS: Eaton Protection Station 800VA 500W

* Default 2GB swap on each disk turned off, enabled 16GB swapfile on mirrored boot SSD
* Added SLOG's are there when the system is using sync writes, I'll leave it at sync=standard. So thats why I'm not planning on using PLP and mirrored devices for SLOG on both stores. Also have a UPS, i know its not a replacement for PLP...


Services that i'll be using:

• SMB
• NFS
• Rsync
• Plex
• B2 backblaze backup
• VM - OMV - SAB, CP, Sonarr, Deluge, Radarr, Duplicati, LazyLibrarian, Headphones, jackett, Ombi, PlexPy etc

Remarks:

• Router = OPNsense 4x gigabit lan, 1 NIC dedicated to FreeNAS to control access in and out (VM's will also fall under this NIC)

Questions:

1. Airflow setup with 7x fan?
2. Set my entire LAN + FreeNAS NIC to MTU 9000 via DHCP and set WAN to default 1500? (of course I'll have static entries in my firewall)

Thank you, have a nice day!

Ezra

 

[LTCM]

Will that board post without a video card? I don't see a way to install FreeNAS with your current hardware list.

 

[ezra]

Good one, I'd bet it will after install... I have a spare Geforce 210 around that I'll pop in for the installation and figure out if it works. Also i'd like to create a windows VM and passthrough a GTX680 and do NVIDEA or Steam streaming to a RPI + PS3 controller in the living room.

 

[tvsjr]

That seems like an awful large list of stuff to run on a single 8-core, 32GB box and expect decent performance.

 

[ezra]

I'm running a AMD A8 FX-5500 with 8 GB of ram with 3 VMS, OPNsense, Ubuntu with all the serrvices mentioned above just not ISCSi, and a windows VM with veeam and it saturates my gigabit LAN and is pretty fast in doing its job.

You're telling me that FreeNAS won't pull this off? Leaving off the mentioned windows VM?

 

[tvsjr]

It will - but how much performance do you want? ZFS is RAM-hungry, especially if you're going to use it for iSCSI. Think about your memory consumption with all of this:
Plex - 2GB minimum
SAB/CP/Sonarr/etc. - 4GB minimum
Nextcloud - 2GB minimum
Bhyve VM - 2GB minimum
Backup, everything else - another 2GB maybe?
Your 32 is now 20GB. And I'm being kind and not considering *all* of your services. Throw a few more VMs, some Docker containers, etc., and you're even lower.

You should absolutely NOT run dedup. Dedup needs massive quantities of RAM and CPU. Jumbo frames is a big MEH these days... wouldn't waste my time on that.

My FN box, providing SMB to Windows clients plus NFS/iSCSI to VMware and Linux, has 128GB on board and it's all in use for caching. More RAM = more cache performance.

You haven't included a SLOG device, which will be important for iSCSI performance.

I would swap the two USB boot devices out for a single small SSD. Much better performance and massively better lifespan.

I would also give real thought to encryption. Why are you doing it? It's computationally expensive, even with AES-NI, it dramatically increases your chances for pool loss, and lots of people turn it on because "hey, it's cool". Systems like NASs are typically compromised over the network... where encryption at-rest doesn't help you. Encryption at-rest primarily addresses the loss or theft of the physical device. If you think encryption will keep the "gub'mint" out of your data, please review:
https://xkcd.com/538/

 

[ezra]

True, though those estimates are quite high. I was aiming to keep 26 GB free for FreeNAS itself and 6 GB for services MAX. I could drop the ISCSi don't really need it, more for the home lab...

• Dedup, i wont use, thanks
• Jumbo frames a lot of hassle?
• Only SLOG for iSCSI and NFS right? (Board is capable of 32Gb/s NVMe)
• FreeNAS is loaded into RAM at boot right? why waste a SATA slot on an SSD that way?
• Encryption because i'm paranoid, if my system gets in the hands of anyone... I'm aware a running system is not encrypted. Other security ill manage via Firewall + VPN to masq my traffic.

Thanks for your reply!

 

[wblock]

FreeNAS is loaded into RAM at boot right?

Not for several years now, no.

Dedup is useful in certain situations which are rarely encountered in home environments. For VMs, cloning is at least as effective and far more efficient.

SLOG is for when you have large sync writes. I would start without it and only add it if performance requires it. Also, for SLOG to be effective, it must be a fast, special-purpose SSD, and one SSD can only be a SLOG device for one pool. (People sometimes try to partition SSDs and use them for multiple devices, but it defeats the purpose.)

Please understand what you are getting into before using encrypted pools. The major point of that encryption is that when a drive is returned to the vendor for a warranty replacement, they will not be able to read the data. But if the encryption keys are not carefully stored and backed up, that data can become inaccessible to the user. And in home environments, people tend to not be as careful as they might in a business use.

An alternate approach is to just burn in new drives for a while, and after they appear to be trustworthy, use them unencrypted. If one fails, discard it, and consider replacing it cheaper than trying to recover lost encryption keys.

 

[ezra]

Not for several years now, no.

Dedup is useful in certain situations which are rarely encountered in home environments. For VMs, cloning is at least as effective and far more efficient.

SLOG is for when you have large sync writes. I would start without it and only add it if performance requires it. Also, for SLOG to be effective, it must be a fast, special-purpose SSD, and one SSD can only be a SLOG device for one pool. (People sometimes try to partition SSDs and use them for multiple devices, but it defeats the purpose.)

Please understand what you are getting into before using encrypted pools. The major point of that encryption is that when a drive is returned to the vendor for a warranty replacement, they will not be able to read the data. But if the encryption keys are not carefully stored and backed up, that data can become inaccessible to the user. And in home environments, people tend to not be as careful as they might in a business use.

An alternate approach is to just burn in new drives for a while, and after they appear to be trustworthy, use them unencrypted. If one fails, discard it, and consider replacing it cheaper than trying to recover lost encryption keys.

• I mistakenly took some info on the net about OS in RAM for granted, sorry...
• Yes I've read that on SLOG, will try without first.
• I fully understand the risk with encryption. And I still want to encrypt my /root with a password, my pools with a random key...

Thank you for your post!

 

[nightshade00013]

Ryzen still has some growing pains in FreeBSD so be aware of that and the possibility that you will probably have to tweak things to make it work and it may always have problems.

 

[Ericloewe]

2000 bucks for this build? You can do much better value for money with Supermicro and an Intel processor.
Specific concerns:

• Motherboard: ASrock x370 TaiChi

• CPU: ADM Ryzen 1700x

The platform is unproven with FreeBSD. 11.1 may improve things, but it's still desktop hardware for a server, which is rarely a good idea.

PCIE: Conceptronic CSATA600EXI (2x Sata + 2x eSata 600gb/s)

Expect that thing to be a paperweight.

PSU: EVGA 500W

Model? They sell a lot of stuff from low-end crap to high-end supplies.

Storage: 6x WD RED 3TB (Encrypted RAIDZ2 or Mirrors, not sure yet. All bought from different shops to get different batches) - 2x Samsung 850 EVO SSD 500GB (Encrypted ZFS mirror for Jails/VM's) - 2x 1TB 2.5 inch

Encryption isn't going to do you much good in 99% of situations. Realistic threats will operate via the clients or via the 5$ wrench decryption process.

VM's

ISCSi (ESXi datastore)

VMs on the server and on a separate VM host? That seems convoluted.

To dedup or to not dedup, I don't think so right now...

The answer is always no. Dedup is pretty horrible and useful only in very specific situations.

 

[moosethemucha]

Just a query you wouldn't have a link to a tutorial or POC for running a windows VM as a gamestation with a raspberry pi as the client ? I think I want to try this.

 

[Ericloewe]

Just a query you wouldn't have a link to a tutorial or POC for running a windows VM as a gamestation with a raspberry pi as the client ? I think I want to try this.

That sounds needlessly complicated, passing through a GPU and all.

 

[moosethemucha]

your point ? I think its clever; sustainable, useable or reliable - I doubt very much. but heck it sounds fun.

 

[ezra]

He back again... did some tweaking and the part list is now updated...

@moosethemucha
Its more for fun. I've setup my OdroidXU4 with android, that launches moonlight (Nvidea gamestream) on boot (and sends a wake on LAN to windows) which on turn brings me to my Steam library. PS3 controller connected.
This is now done with a less powerfull machine. I'd like to use that one as a Desktop and move the GTX over to the FreeNAS box.

Now there is an option to passthrough my GTX680 via CLI but I'd rather wait for the option to be added in the GUI.
Also I'll add another 32 GB ECC RAM once I'll create the windows VM

Ref: https://github.com/moonlight-stream/moonlight-android (use with the XU4 odroid)
RetroPIE moonlight (only for the Raspberry) https://github.com/TechWizTime/moonlight-retropie

Passthrough: https://wiki.freebsd.org/bhyve/pci_passthru

edit: other option would be to just run ESXi and 2 VM's: FreeNAS and Windows, passthrough all the storage to FreeNAS and only the GPU to Windows.

 

[ezra]

Updated, ordered everything now!
Components added: LSI 9240-8i 8-port SAS SATA LSI00200 IBM M1015
Still have to set it to IT mode...

FreeNAS 11.1 comes with Ryzen support or at least fixes for Ryzen.

 

[ezra]

Anyone else?

 

[wackymole]

Ryzen is not stable on FreeNAS 11.1. Crashes randomly... every 3-15 days for me.