Help with first FreeNAS build for a medium-sized home server?

[antesilvam]

Dear all,

in the current Corona lock-down I finally find the time to start a project which I have in mind since quite a while. I would like to finally build my own FreeNAS server.

The requirements for the system are rather 'standard':

• I want to use it mostly for backups and for storing/accessing/streaming photos and music.
• The data should be natively encrypted.
• 2-3 people might need to access data in parallel.
• Due to my current flat situation the server will have to share with me the living room, so the build should be at an acceptable noise level (ideally quiet). ;-)

I went through the excellent hardware guides and came up with the following components:

Code:

CPU:                Intel Core i3-6100
Motherboard:        Supermicro X11SSL-CF (https://www.supermicro.com/en/products/motherboard/X11SSL-CF)
RAM:                2x16GB DDR4 2400MHz ECC
Storage:            1x64GB SDD (boot device)
                    5x WD Red 5TB 3.5" 5400RPM (planning to use RAID-Z1)
Storage Controller: SAS 3008 SAS3 controller (on board)

Do you see problems with that hardware selection or do you have better recommendations for my use case?

The Motherboard comes directly with 6 SATA3 (via C232) and 8 SAS3 (via Broadcom 3008 SW controller) ports. Is there any preference which ports to use to connect the HDDs?

While looking around, I stumbled over the IMHO neat idea of running the FreeNAS server as a VM inside proxmox. This solution would help me to combine a few more services in one machine (which I run on different raspies at the moment), like:
* PLEX
* PiHole
* Unify controller
* VPN server

I found this interesting post in this forum where all the NAS storage is behind a PCI SAS controller which is then passed through directly to the FreeNAS VM: https://www.ixsystems.com/community/threads/standalone-freenas-migration-to-proxmox-freenas.81621/

What is your opinion on that option? Would you recommend this setup or should I better keep a storage server and other server tasks in seperated physical machines.
If you think it is a good idea to combine: Do you think the mentioned CPU is sufficient?

I would be very grateful for any recommendation or thoughts!

Thanks a lot!

Benedikt

 

[Inxsible]

Due to my current flat situation

"flat" -- indicates to me that you are probably not in the US -- which is why I won't even bother telling you to go the used components route.

CPU: Intel Core i3-6100 Motherboard: Supermicro X11SSL-CF (https://www.supermicro.com/en/products/motherboard/X11SSL-CF) RAM: 2x16GB DDR4 2400MHz ECC

Your chosen CPU doesn't seem to support 2400. You might want to upgrade it to i3-8100 unless your RAM supports lower clock speed of 2133

(planning to use RAID-Z1)

Don't !!! Use RAID-Z2

The Motherboard comes directly with 6 SATA3 (via C232) and 8 SAS3 (via Broadcom 3008 SW controller) ports. Is there any preference which ports to use to connect the HDDs?

Nope. You can choose whichever ports you want in order to connect the drives.

FreeNAS server as a VM inside proxmox.

It's doable. Many users here run FreeNAS as a VM under ESXi too. Your selected cpu and i3-8100 support VT-d, so passthrough should not be a problem. Just use ZFS for the storage and while you are at it, you can use ZFS for proxmox as well. I would keep the proxmox pool and the FreeNAS pool separate though.

 

[antesilvam]

Thanks for your reply!

Your chosen CPU doesn't seem to support 2400. You might want to upgrade it to i3-8100 unless your RAM supports lower clock speed of 2133

i3-8100 is a coffee lake CPU, no? I thought it is not supported by the chipset on the motherboard (C232). I was further looking around and was wondering if it makes sense instead to upgrade to an Xeon E3 processor, like e.g.

• E3-1225v6 [cores/threads: 4/4, 3.3GHz, internal GPU] ~250EUR
• E3-1230v6 [cores/threads: 4/8, 3.5GHz, no GPU] ~320EUR
• E3-1275v6 [cores/threads: 4/8, 3.8GHz, internal GPU] ~390EUR

If I go for the virtualized solution, it might be nice to have some CPU headroom...? Please correct me if I am wrong, but an internal GPU is quite beneficial even if the server is going to run headless in order to have hardware support for transcoding and therefore I would give preference to the models which feature already a GPU.

Don't !!! Use RAID-Z2

Okya. Could you elaborate why you would prefer RAID-Z2 over RAID-Z1 in this case?

Nope. You can choose whichever ports you want in order to connect the drives.

Ok, but I understand correctly that the on-board SAS controller of the mother board needs to be flashed such that it is in IT mode, right?

Thanks again for the help!

 

[Inxsible]

i3-8100 is a coffee lake CPU, no? I thought it is not supported by the chipset on the motherboard (C232).

You are right. I forgot about that. In fact, I think going with a Xeon would be better. If this were going to be a pure FreeNAS build then I would not use a xxx5 series processor as FreeNAS just doesn't make use of the GPU at all. But if you know you need it for a VM etc via Proxmox, then go for it.

Okya. Could you elaborate why you would prefer RAID-Z2 over RAID-Z1 in this case?

Large drives have a higher chance of failure during resilvering. RAIDZ1 provides a tolerance for 1drive failure, whereas RAIDZ2 provides tolerance of 2 drives. Save your data. Use RAIDZ2.

Ok, but I understand correctly that the on-board SAS controller of the mother board needs to be flashed such that it is in IT mode, right?

Yes, whether on board or a PCI HBA card, you have to flash it to IT mode before using it with FreeNAS. It's not too difficult to flash it if it hasn't been flashed already.

 

[antesilvam]

Dear all,

please find here the updated/final hardware collection and some corresponding thoughts:

• I decided for the moment to go with i3-7300 for budget reasons. If it turns out to be under-powered, I still can upgrade to a Xeon E3-1225v6 or E3-1275v6 later rather easily. Instead, I decided to invest the saved 150EUR into...
• SuperMicro X11SSH-CTF: I found this board only after my initial posting and find I really nice! It is a better version of X11SSL-CF featuring an on board SAS3 controller (like the X11SSL-CF), but in addition has 2x10GbE. I figured that it might be a good investment into the future as at some point I will certainly upgrade my home network to 10GbE and especially a NAS will profit from the larger bandwidth.
• The RAM speed is faster than supported by the CPU/motherboard, but it was a good offer so I took it anyways.

CPU	1	Intel	Core i3-7300 (2c/4t; 3.9GHz; iGPU HD P630; 51W)
motherboard	1	SuperMicro	X11SSH-CTF (8XSAS3 via onboard Broadcom 3008; 8x SATA via PCH; 2x10GbE)
RAM	2	Crucial	CT16G4WFD8266 (16GB; DDR4; 2666MHz; ECC; UDIMM)
HDD​	7	Western Digital	WD RED WD40EFRX (4 TB; 5400rpm; 64 MB; 3.50")​
SSD​	2	Transcend	32GB SATA III 6Gb/s SSD370S​
case	1	Fractal Design	define r5
CPU cooler​	1	Noctua	NH-D9L​
additional case fan​	1	Noctua	NF-A15 PWM​
SAS breakout​	2	cablecreation	HD Mini SAS (SFF-8643 Host) zu 4 x SATA​
PSU	1	be quiet!	STRAIGHT POWER E9 (480W)

Now, I started to make the detailed planning of the (virtualized) setup and would be very grateful for some feedback from the community. My current idea is:

• 6 HDD: FreeNAS storage pool connected to the SAS controller and operated in RAID-Z2 as suggested by Inxsible
• 1 SSD: FreeNAS boot connected to one port of the SAS controller
  (I might add another small SSD in the future to mirror the boot device)
• 1 HDD: VM storage 'pool' for proxmox connected to one PCH SATA port
  (I might expand this by a second HDD to mirror the first one at some point, but don't have the money right now)
• 1 SSD: proxmox boot connected to one PCH SATA port

I would 'simply' pass through the entire SAS controller to the FreeNAS VM and should be done or do I miss something? If I understand things correctly than in this configuration the following statements should be true:

• FreeNAS should be running like on bare metal in terms of storage access
• this setup would give me the possibility to de-virtualize again easily
• it should circumvent all the potential pitfalls of a FreeNAS system with virtualized storage pointed out in those many warnings out there

Please let me know if I miss a point or misunderstand something. Do you see any problem with this plan?

Do you maybe also have some advice for the resources I should attribute to the FreeNAS VM in terms of RAM/CPU cores/...? How should the ethernet interfaces be handled?

I am very excited to get the hardware into my hands and start settings things up!

Thanks a lot for your help!

 

[Pitfrr]

Why do you use one SSD for the FreeNAS boot and one for the Proxmox?
I'm not familiar with Proxmox so it might not apply. I'm using ESXi and I have VMware on a SD card to boot then I use SSD to store the VM images (and the FreeNAS), so FreeNAS is installed on the VM storage provided by the SSD.
And then I have the HDD passed through to the FreeNAS VM.

With your configuration, I would use one SSD for Proxmox boot and then use your VM storage HDD to store the FreeNAS boot environment.

Personally I would use all the HDD for FreeNAS and get some "big" SSD for the VM storage (it'll be much faster to boot up VM. That is not necessarily a strong requirement but it's very nice to have! ).
So maybe instead of 2x 32GB SSD, I'd go for one 512GB or 1TB SSD as VM storage.
And then use the 7x 4TB HDD as RAIDz2 storage for FreeNAS.

Depending on how many VM you want to run you might want to increase the RAM as well...
If you're going to run FreeNAS with no plugins then 8GB RAM reserved is good. If you have few users, you'll have to see if you might need to increase the amount of RAM eventually. The nice thing with virtualization is that you can easily add resources (i.e. CPU or RAM) so you'll see how it performs.
My understanding of virtualization is that you'd want as much cores as possible. Again I don't know how this is handle in Proxmox... And depending on your Plex usage it can turn out to be very CPU hungry...


Did you also consider you backup options for such server?

 

[antesilvam]

Hi Pitfrr, thanks for your comments!

Why do you use one SSD for the FreeNAS boot and one for the Proxmox?

My idea behind that was to be able to de-virtualize easily in case I decide at some point in the future that the NAS should run bare metal and stand alone. Is there an easy way to do this, if I use the VM pool also for FreeNAS boot? I found the idea neat to have all storage devices related to FreeNAS (boot and storage pool) behind the SAS controller which I simply pass through. By doing that I keep the storage non-virtual, but can virtualize CPU/RAM/... . But I am not at all experienced - so I am more than willing to listen to your advice!

Did you also consider you backup options for such server?

Not too much yet, I have to confess. I have a 2-bay qnap NAS currently with 2x6TB WD Red installed. Maybe I could do something with that? Do you have recommendations/ideas/pathways for a good backup strategy?

 

[Inxsible]

@antesilvam, with your original config, are you planning on booting either FreeNAS or Proxmox at any given time? That would not be useful at all in my opinion.

What @Pitfrr said is correct, you need to install Proxmox on the bare metal and then create a VM for FreeNAS. You have to choose between virtualization or bare metal and they are not easily changeable without a re-install of FreeNAS at least. You can still re-utilize the zpool, if you want to -- provided you take the right measures.


My take would be

1. Install Proxmox on bare metal
2. Use ZFS or LVM for Proxmox VM storage -- SSDs would be better than HDDs but they are not necessary
3. Create a VM for FreeNAS
4. Create a separate pool for FreeNAS storage. Pass this to the FreeNAS VM
5. Create additional VMs/CTs on Proxmox for whatever you want
6. Profit !

Or build 2 separate systems -- 1 for bare-metal FreeNAS and another for Proxmox.

Onboard SAS and 10GBe is great to have until it craps out, then you gotta change the board. I think -- and I could be wrong -- you would probably save money if you buy a board without SAS and 10GBe and add those if and when you want. That way you are not spending money now for something that you "might" need 4 years down the line. HBA cards are super cheap to add. and if and when you do put in 10GBe in your network, you can always add a Mellanox or Intel 550 card to your FreeNAS system. Just make sure you buy a board with enough PCI lanes for now.

 

[Pitfrr]

make sure they are not SMR drives

My understanding was that WD red at least were never SMR drives (as opposed to the Seagate ones where you would never be sure what you get)?

My idea behind that was to be able to de-virtualize easily

You still can do that easily... Basically for FreeNAS, you just need to save the config file and then you're good to go almost anywhere! So as long as you keep your drives and your config file, you can restart a FreeNAS server virtualized or not with the same hardware or not.
Therefore, for me this doesn't necessarily require to have the system on a dedicated drive. Well I don't see that as a big deal if I have to reinstall and restore the config (and down time is not an issue for now).
I'd say the most important are your drives (FreeNAS pool) but the boot device itself doesn't matter much. You could even go for a USB stick if you want to start you system bare metal (ok USB sticks are not so reliable so mirror USB sticks... :) ).


As for your backup, if you have a 2 bay NAS lying around it might be good to use it as a backup device (with rsync for example). The good thing is that the HW is available and you can use it directly, that's perfect!
Depending on the criticality of the data, I'd also suggest an off-site backup (like external HDD synchronize every now and then).
Also, even though it is not a backup so to speak, I would encourage the use of snapshots.

Regarding the backup (2x6TB): you might also give it some thoughts to check if the backup space is enough...
You are aiming let's say for 7x4TB in RAIDz2 that is 5x4TB = 20TB so with the don't-fill-your-pool-over-80%-capacity rule that would be around 16TB maximum usable.
And on the other side you have your QNAP with 2x6TB so either mirror with 6TB usable or stripped with 12TB (but then no redundancy).
Is this ok for you or not...?? For example are you fine with a backup that has no redundancy?
But I wouldn't bother too much on that, at least to begin with, I'd say you have an easy solution with you QNAP to start with! :)

 

[Inxsible]

y understanding was that WD red at least were never SMR drives (as opposed to the Seagate ones where you would never be sure what you get)?

The newer EFAX ones seem to be SMR. See: https://www.ixsystems.com/community/threads/first-home-nas.83070/#post-576287

 

[Pitfrr]

Hmmm quite annoying I though WD was more consistent and, at least for the RED series, didn't propose SMR drives. It seems not to be the case and worst, they don't even disclose the information! :-((

 

[antesilvam]

The newer EFAX ones seem to be SMR. See: https://www.ixsystems.com/community/threads/first-home-nas.83070/#post-576287

I got WD40EFRX, which according to some googling is PMR, so seems like I was lucky. I was totally unaware of this spec...