First FreeNAS Build

[CelticDubstep]

Motherboard: Gigabyte GA-990FXA-UD3 (rev. 4.0)
CPU: AMD FX-8350
RAM: 4x 4GB Crucial Callistix Tactical 1600MHz 8-8-8-24
SSD: 2x 256GB Samsung Pro's (going to use them in a boot pool should one fail)
HDD: 8x Hitachi HGST 8TB 7200RPM 3.5" 12Gb SAS HUH728080AL4200 4KN
HDD Controller: LSI 6Gbps SAS HBA LSI 9211-8i (Unconfigured IR Mode. FreeNAS is able to read SMART Data, Make, Model, Serial, etc)
NIC: Intel Pro/1000 PT Quad Port/Onboard Realtek GbE
Note: I have a bunch of SSD's laying around... perhaps I should use one for cache?

This is a long post so please bare with me here. I'm long past due for a file server at home as I have various random drives laying around all over the place with data that either have no backups, or is duplicated on multiple drives. It's a mess and has been for well over 20 years now. A little about myself... I've been in IT for the better part of 15+ years now and I am currently the sole IT person at my company which is an engineering firm. I manage two office locations and 5 physical servers and 2 Synology (one RackStation and one DiskStation). All the physical servers are using hardware RAID controllers with cache, battery backup, etc. This is pretty much how I've always done things throughout my IT career managing servers.

Synology has been the only NAS experience I've had which has left a very bad taste in my mouth and what once served as our file servers at the two offices, now serve as a place to store backups. Users started to intermittently get errors in the middle of working on engineering projects such as "The handle is invalid", "Location is not available", "P:\ is not accessible", and "The system cannot open the file" which often time caused them to lose, at times, hours worth of work depending on the last time they saved their work. The issue was a nightmare to track down since it was very intermittent and I spent weeks working with Synology support (which is terrible) as well as troubleshooting myself before I finally discovered the issue. I was eventually finally able to pull Wireshark logs from a users workstation when the issue happened and match that with the logs from Samba on the Synology and in short... Wireshark logged a response from the Synology "STATUS_TOO_MANY_OPENED_FILES" and "STATUS_INVALID_CLASS_INFO". Meanwhile at the same time, the Samba logs showed the following:
../source3/smbd/open.c:785: [2019/12/04 16:06:42.803727, all 0, pid=6931] fd_open
Too many open files, unable to open more! smbd's max open files = 16424
../source3/passdb/pdb_smbpasswd.c:272: [2019/12/04 16:06:44.006566, passdb 0, pid=6931] startsmbfilepwent
startsmbfilepwent_internal: unable to open file /etc/samba/private/smbpasswd. Error was: Too many open files
../source3/passdb/pdb_smbpasswd.c:1333: [2019/12/04 16:06:44.006634, passdb 0, pid=6931] smbpasswd_getsampwnam
Unable to open passdb database.

Due to how locked down the Synology ecosystem is, it wasn't possible to increase that limit and at the end, they simply said we were out of luck and needed to find another solution. Ultimately in the end, I purchased Windows Server 2019 (to replace Server 2012 R2) for our server, as well as upgraded it from 32 GB Ram to 128 GB Ram and I also purchased 4 8 TB SAS drives and put them in RAID 10 and haven't had a single problem since.

This has made me cautious of solutions such as FreeNAS and Unraid, but I've heard nothing but great things, especially FreeNAS due to the ZFS file system. There were also many features of the Synology I loved, such as the snapshots (so much better than shadow copies), and apps you could get such as cloudsync to sync with various cloud providers.

For my home setup, at the moment, I'm strongly considering FreeNAS with a RAIDZ2 or RAIDZ3 setup. My biggest concern is that if my hardware fails, how difficult is it to get my data off these drives since they are in a ZFS RAID configuration? My other concern is that I've read you should have 1 GB ram per 1 TB of storage and my server will only have 16 GB. Now, I do have two "lab" i7-4790 systems, each running 32 GB Ram that I could "swap" out so the FreeNAS box will have more RAM, but I'd really prefer not to do that. My FreeNAS Box will be strictly for storage and nothing more (except cloudsync maybe). My Plex Server will run on the i7-4790 system and simply pull the files from FreeNAS to transcode.

I also have 8 2 TB drives laying around (and 8 300GB 15K RPM SAS drives, but we won't go there) that I thought about using as a 2nd FreeNAS box, but don't see much of a purpose in that?

Also, before anyone asks... I can't do anything with the i7-4790 systems. They are 100% proprietary other than the CPU, RAM, and Hard Drives. Everything else (PSU, Motherboard, Case, etc) are completely useless, and I don't have any spare motherboards for 4th gen CPU's.

Thanks in advance for any feedback.

 

[Yorick]

My biggest concern is that if my hardware fails, how difficult is it to get my data off these drives since they are in a ZFS RAID configuration? My other concern is that I've read you should have 1 GB ram per 1 TB of storage and my server will only have 16 GB

Easy to get off, just pop them into any system that can understand the feature flags. Linux, another FreeNAS, whatever you like.

1GiB per TB is outdated guidance. If you have a lot of files, you will wish for more than 16GiB, though, just because accessing metadata will be slow.

Motherboard: Gigabyte GA-990FXA-UD3 (rev. 4.0)
CPU: AMD FX-8350

I'm assuming that's what you have lying around? With the Intel Quad NIC, you'll be okay.

Note: I have a bunch of SSD's laying around... perhaps I should use one for cache?

Cross that bridge if you come to it. L2ARC eats ARC, and, if you have a lot of files, running out of a place to stick your metadata ain't fun. You could use a small SSD (128GB or less) as L2ARC, if you find metadata won't fit into ARC and it's slowing you down. Only then. arc_summary.py will guide you.

 

[CelticDubstep]

Easy to get off, just pop them into any system that can understand the feature flags. Linux, another FreeNAS, whatever you like.

1GiB per TB is outdated guidance. If you have a lot of files, you will wish for more than 16GiB, though, just because accessing metadata will be slow.




I'm assuming that's what you have lying around? With the Intel Quad NIC, you'll be okay.




Cross that bridge if you come to it. L2ARC eats ARC, and, if you have a lot of files, running out of a place to stick your metadata ain't fun. You could use a small SSD (128GB or less) as L2ARC, if you find metadata won't fit into ARC and it's slowing you down. Only then. arc_summary.py will guide you.

Thanks for your reply, I appreciate the response and information. I actually have several other systems laying around that are unused if you think they'd be better suited? This is what I have... All these are paired with Intel Desktop Motherboards:
At least one other FX-8350 system with an ASUS motherboard, one of them having a 970 chipset.
One i7-2600K System
One i7-950 System
Two i5-750 Systems

I have a bunch of other systems laying around, but they'd be too old as they are Core 2 Quad's, Duo's, Athlon XP's, Athlon TBirds, Pentium III, Pentium II, Pentium MMX, Cyrix.... etc.

In case you're wondering, yes I'm a hoarder. :) However, I use these systems for various things, but mostly retro gaming (legacy Windows XP, 9x, and DOS games that don't work under emulation such as DOSBox). My daily driver is an aging i7-6700K with 32GB Ram and a 980 Ti.

 

[Yorick]

More RAM will help you, mostly as a read cache. If you have something with a supported Intel NIC and 32GiB of RAM, that'd be a good choice.

"The pool is king" - how you structure your ZFS pool is far more important than what hardware you run FreeNAS on. With the caveat that yes, 9 out of 10 issues with FreeNAS are hardware-related - but that's mostly stuff like "I tried to use a RAID controller" (don't) or "this NIC won't work right" (easily fixed by popping an i210 or such in). As long as the hardware is supported and stable, you're fine. You can always move the pool to other hardware, as long as FreeNAS has direct access to the disks. AHCI or HBA, and you're golden.

 

[Yorick]

LSI 6Gbps SAS HBA LSI 9211-8i

Only supported firmware is P20, take that seriously. IR is fine; but if you have to flash anyway to get to P20, why not go IT.

 

[CelticDubstep]

More RAM will help you, mostly as a read cache. If you have something with a supported Intel NIC and 32GiB of RAM, that'd be a good choice.

I could downgrade one of the i7-4790's to 16GB and upgrade the FreeNAS box to 32GB. I could also disable the onboard Realtek NIC on the FreeNAS box and only use the Intel Quad Port NIC. I have 4 of those Quad Port NIC's. I was going to use them for a project a while back, but apparently the specific model I have is extremely picky on the type of motherboard it will work with. They wouldn't work on my i7-6700K, nor the two i7-4790's. It gets rather technical on the Intel forums, but this version of card I have is two/two ports linked via a bridge of some sort which only work with CPU's and motherboards that aren't PCIe Gen3. I dunno, It's dumb. The i7-4790's I have, the motherboard's PCIe slots are Gen2, but because the CPU supports Gen3, the NIC's won't work in them. It works great on the soon to be FreeNAS box though since the CPU and Motherboard are Gen2.

Only supported firmware is P20, take that seriously. IR is fine; but if you have to flash anyway to get to P20, why not go IT.

I actually flashed it from IT mode to IR mode. It's running P20. I wasn't 100% sure if I wanted to go the FreeNAS route which is why I flashed it to IR mode so I could use the RAID functions of the card, but it's a very "weak" card for RAID as it doesn't support 6, let alone 5. I've already done a test install of FreeNAS on the box with two of the 8 TB drives in it and created a share and it all seemed to work well. I had also loaded Windows Server 2019 at one point and tried out Storage Space... That's a horrid mess. I'd rather use software RAID (Intel RSAT) or hardware RAID over that mess. So, at this point its either buy a Hardware RAID card that supports RAID 6, or the FreeNAS route which seems to be better due to ZFS and if the hardware fails, I can swap it over to another system which is much more complicated with hardware RAID.