1st freenas build, please give opinions & advice!

zerohimself

Cadet
Joined
Mar 25, 2019
Messages
5
Ok, so i'm getting ready to build a workstation and a NAS/vm/rendering assisting machine.
I'm currently capturing 1080p60 and lower video for youtube content, and hope to upgrade to 4k over the next year or 2 ( i am going for 4K capture cards, along with having a backup for older analog sources)

I'll list my current thoughts on the workstation as well, as I am always up for knowledgable input!!. This is my first NAS build and my first new PC build since 2007. So that being said, I may have unrealistic expectations and thoughts, and appreciate any help/suggestions.

So my initial thoughts are to build a thread ripper workstation with a RTX 2080 ti to do the majority of the work. I want to set up a cheap 10gbe network to serve the purpose of allowing my first 1st workstation (and maybe a lesser workstation) to access the NAS. The workstation will contain either a NVME or M2 as the boot drive, and prob a 8tb or larger spinning disk for general storage (probably just apps and a little gaming).

The NAS will be hosting my filestore, it will contain my old home network (old apps, my videogame backup library[roms and isos], some personal video backups of bluray rips and such, and be the storage location for the video / footage we are working on). I will be using adobe and devinci for video processing.

I am concerned about data redundancy, along with performance. I prefer reliability/redundancy over performance, however a balance of both would be amazing. I was intending on running raidz3, as i have had 2 drive failures and lost everything on more than one occasion. I am currently working on a cost effective offsite backup solution as well, I may just build a offsite large NAS and backup important folders over the internet as opposed to the cost of backing up this entire setup to a glacier or backblaze plan. (again, open for suggestions)

Cost is a concern, however I would much rather throw a little money at this for reliability/useability/future proofing. I want to build a good system at a decent price that do what I need for at least a couple years. I want to be able to expand the storage space as needed, and upgrade in unit instead of having to build a new server and migrate the data between machines.

I am mainly looking for advice on the server & network equipment, however am open to all/any suggestions.

here are my thoughts on the hardware:
NAS Server
chassis/mobo/cpu see this link: Supermicro 24 Bay 6Gbps FREENAS Storage Server Xeon E5-2650 V2 16 Core 128GB Ram
Chassis: Supermicro 4U 24x3.5" drives CSE-846E16-R1200B
Motherboard: X9DRi-F
CPU: 2x Intel Xeon E5-2650 V2 Octo Core 2.6Ghz
Ram: 128GB DDR3 (16 x 8GB - DDR3 - REG PC3-10600R (1333MHZ) ) -- unspecified brand(I will ask if you have suggestions!!)
HBA controller: 1x LSI 9210-8i HBA FREENAS UNRAID Controller
Capture card: AVerMedia Live Gamer 4K (primary purpose is to capture footage from the workstation #1, however i do have a 2nd workstation that can do this)
GPU: EVGA GeForce RTX 2080 Ti (subject to change for price & advice & quality)
NIC: Please advise (i'm open for 10gbe or 40gbe either ethernet or fiber) - I don't need anything overkill(haha) but i do need to be able to interconnect at least 3 devices
Bootdrive: Please advise!!
Read cache: Intel NVME SSD DC P3700 Series 2TB 1/2 Height PCIe 3.0 SSDPEDMD020T4D Please advise, i'm not sure if i even need in this setup!!
HDDs: 5x low use or new 8TB HGST HE8's (was recommended) (should i go for a SATA interface or SAS?)
UPS: Please advise!!

I am concerned about having enough PCIe lanes available for this setup, as i'll use 32 of 40 not including any chipset, or the HBA controller.
If this is the case, I suppose the capture card could be moved to the 2nd workstation we are using.
Work Station
Mobo: MSI MEG X399 Creation sTR4 AMD X399 Extended ATX Motherboard OR ASUS ROG Zenith Extreme Alpha X399 HEDT Gaming Motherboard AMD Threadripper 2 (TR4) EATX DDR4 M.2 10G LAN USB 3.1 Gen2
GPU: ASUS GeForce RTX 2080 Ti 11G Turbo Edition GDDR6 HDMI DP 1.4 Type-C Graphics Card
CPU: AMD 2nd Gen Ryzen Threadripper 2950X, 16-Core, 32-Thread, 4.4 GHz Max Boost (3.5 GHz Base), Socket sTR4 180W YD295XA8AFWOF

PSU: EVGA SuperNOVA 1200 P2 220-P2-1200-X1 80+ PLATINUM 1200W Fully Modular EVGA ECO Mode Includes FREE Power On Self Tester Power Supply
HDD: WD Red Pro WD8003FFBX 8TB 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
SSD: SAMSUNG 970 EVO M.2 2280 1TB PCIe Gen3. X4, NVMe 1.3 64L V-NAND 3-bit MLC Internal Solid State Drive (SSD) MZ-V7E1T0BW
Ram(will expand if needed): G.SKILL Trident Z RGB (For AMD) 32GB (4 x 8GB) 288-Pin DDR4 SDRAM DDR4 3200 (PC4 25600) AMD X399 Desktop Memory Model F4-3200C14Q-32GTZRX
Capture #1: AVerMedia Game Streaming Capture Card Live Gamer 4K (GC573)
Capture #2: Datapath Vision RGB E1 1080p Video Capture Card PCI-Express Full Profile
Sound card (if needed): to be determined

NIC (if needed): open for advice
UPS: Please advise!!

Network hardware
Switch: Please advise!!
Cabling: Please advise!!
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142
GPU: EVGA GeForce RTX 2080 Ti (subject to change for price & advice & quality)
If you plan on running this as a standalone FreeNAS box, this would be totally wasted. If you plan to use a hypervisor (VMWare, Xen, etc.) then you can pass through the GPU to a VM and have the VM as a rendering machine.

Same problem as above.

Everything else in the Supermicro case should play nicely with FreeNAS, when it comes to the ReadCache (L2ARC) someone else will probably be better at answering this than me.
 
Last edited:

zerohimself

Cadet
Joined
Mar 25, 2019
Messages
5
If you plan on running this as a standalone FreeNAS box, this would be totally wasted. If you plan to use a hypervisor (VMWare, Xen, etc.) then you can pass through the GPU to a VM and have the VM as a rendering machine.

I am still a complete NEWB here. what is my best option here to allow use of this machine as a freenas server, and to VM a moderate use workstation? can I pass the GPU through, and then pass the HBC through to the freenas VM? is this even a good idea/way to do it? what hypervisor would be best?
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142
First off, have a look in the Useful Threads section, especially Slideshow explaining VDev, zpool, ZIL and L2ARC
When it comes to FreeNAS and ZFS, do make sure you read up on "everything", it will save your a** later on :p

I prefer VMWare, and I think that is the "most" supported hypervisor, but again, there are a lot more people here that know a lot more than I do on the subject.
and then pass the HBC through to the freenas VM?
Yes, you pass the HBA through to the FreeNAS VM, this is important.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Ok, so i'm getting ready to build a workstation and a NAS/vm/rendering assisting machine.
I think you are not making a good choice in trying to have your FreeNAS system do video rendering. Even if you setup a ESXi system and make your FreeNAS as a virtual system instead of on bare metal, it just increases complexity and stands to reduce the performance of the NAS.
Wendel at Level1News did a two part video series on the render system they setup. You might want to take a look:
Part 1:
https://www.youtube.com/watch?v=96e9grnOTZE
Part 2:
https://www.youtube.com/watch?v=60QDV_fHQuE[/QUOTE]
That, or something like it, would likely work much better for you and just use the server as a server with FreeNAS on bare metal and that saves you from the additional complexity of managing a hypervisor.
NIC: Please advise (i'm open for 10gbe or 40gbe either ethernet or fiber) - I don't need anything overkill(haha) but i do need to be able to interconnect at least 3 devices
You will need a switch if you want to interconnect multiple devices. Do you want them all at 10Gb speed?
I am not endorsing a specific item of hardware, but SFP+ 10Gb hardware is much less costly than the BASE-T (RJ-45) gear and it has lower latency. If you want a fast network, there is absolutely nothing wrong with this older hardware. It is only being sold because the place that was using it probably upgraded to something even faster. Here is an example:
https://www.ebay.com/itm/Brocade-BR...-Switch-24x-10-Gb-SFP-Uplink-JMW/232231406925
As for the network card, I have used Chelsio 10Gb cards in my FreeNAS systems, but I recently was told that these worked well for another forum member:
https://www.ebay.com/itm/Solarflare...rt-10G-Ethernet-10GbE-PCIe-w-SFP/113540380624
I ordered two of them myself to do some testing with, but I have not had a chance to install them yet.
If you can get Chelsio cards at a good price, like these:
https://www.ebay.com/itm/Chelsio-10...30-2x-10G-SFP-Tranceivers-HIGH-F/113669593010
It might be more about personal preference, but I also understand that these Chelsio cards might be limited to roughly 6Gb of bandwidth, so it would be better to go with a newer model.
Bootdrive: Please advise!!
I suggest using SSD boot drives. That chassis has a place inside where you can mount an adapter like this:

1553616531372.png click to expand

Drive Cage Part for 4U, 24 bay, Supper Chassis:
Supermicro MCP-220-84603-ON 2.5" Fixed HDD Tray
https://www.amazon.com/Supermicro-MCP-220-84603-2-5-Fixed-Tray/dp/B009AF7Z22
Read cache: Intel NVME SSD DC P3700 Series 2TB 1/2 Height PCIe 3.0 SSDPEDMD020T4D Please advise, i'm not sure if i even need in this setup!!
I am not sure why you think you might need it. It depends on how you end up using the NAS.
HDDs: 5x low use or new 8TB HGST HE8's (was recommended) (should i go for a SATA interface or SAS?)
SATA drives can be connected to SAS controllers, no problem, and the server you have selected has a SAS HBA and a SAS Expander backplane for the drives to plug into. No room for or need of an extra SATA controller. There is a SATA controller built into the system board and you can use that to run the boot drives.
I would suggest getting one more drive, so you can have a six drive RAIDz2 vdev in your storage pool, that will leave room in the chassis for three more like vdevs to be added, to expand the pool, as your storage needs increase. Have you familiarized yourself with these terminologies? You might need to review the guides here:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/
UPS: Please advise!!
Where I work, and what I use at home, we have these APC UPS units and we have been very satisfied with them:
https://www.amazon.com/APC-Back-UPS-Battery-Protector-BR1500G/dp/B003Y24DEU
 

zerohimself

Cadet
Joined
Mar 25, 2019
Messages
5
....
Wendel at Level1News did a two part video series on the render system they setup. You might want to take a look:
I actually watched that right after my post, and i def agree at this point!
...
I am not endorsing a specific item of hardware, but SFP+ 10Gb hardware is much less costly than the BASE-T (RJ-45) gear and it has lower latency. If you want a fast network, there is absolutely nothing wrong with this older hardware.
I actually had the same thoughts there
I would suggest getting one more drive, so you can have a six drive RAIDz2 vdev in your storage pool, that will leave room in the chassis for three more like vdevs to be added, to expand the pool, as your storage needs increase. Have you familiarized yourself with these terminologies? You might need to review the guides here:
My initial thoughts were to do a raid z3 - I guess Z3 might be a little overkill? The suggestion to use 6 bays for each vdev makes a lot of sense..
i'm still a little up in the air betrween Z2 and Z3, as i've had 2 drives fail at one time on me before.. I'm really thinking about just populating this with new drives from different mfg date batches, and have a minimal offsite backup instead of paying a monthly cost for cloud storage(everyone says it's cheap, but at the point I would need to backup this much data, it would get pretty expensive..
Also i was refereing to if I needed a L2ARC, but upon reading the info you gave me, I believe it would be a waste of money.
Also I was unclear about the sata vs sas. I was asking which kind of drives i should populate this with, as i imagine mixing them would not be a good move!

Thank you very much for your help!
I feel like I am getting this narrowed down, I may just have a very overpowered server for my NAS, but it will be very expandable.
I will continue reading, however I am still a little puzzled over the best way to install freenas and run a few VM's. Would the best way would be to install freenas on baremetal & run a few VM's under freenas?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
The suggestion to use 6 bays for each vdev makes a lot of sense..
i'm still a little up in the air betrween Z2 and Z3, as i've had 2 drives fail at one time on me before..
I have also. I keep cold spares on hand that are already burned in, tested and ready to go. That way, when a drive fails, I can do a quick swap and the system survives. A little over a year ago, I had two drives in the same vdev fail in my home NAS, but at that point I was using drives that were over five years old. I did my swap on both drives and the vdev resilvered without loss of data, but I have a full backup server that is getting updated every few minutes during the day and a backup pool that is getting updated every night. So, even if I had a pool failure, it would be unlikely for me to loose data. We do have a few forum members that are on RAIDz3. It just depends on your tolerance for risk. I don't think RAIDz2 is that bad a risk, especially if you have a backup.
Also I was unclear about the sata vs sas. I was asking which kind of drives i should populate this with, as i imagine mixing them would not be a good move!
I would not mix SATA drives with SAS drives. It should work but I prefer not to take a chance. I like SATA drives because I feel like the SMART report data they give is better for diagnosing faults. SAS drives are good, but most drives, if you look at the data sheet from the manufacturer, the specs are the same between SAS and SATA, so the only way it makes sense to pay the price premium that SAS usually has attach is if you are buying drives that are able to be multipathed. There is a lot more involved in that, cost and hardware, so most people are not going in for that unless it is a mission critical system.
 
Last edited:

Evertb1

Guru
Joined
May 31, 2016
Messages
700
My initial thoughts were to do a raid z3 - I guess Z3 might be a little overkill? The suggestion to use 6 bays for each vdev makes a lot of sense..
To me the redundancy is for the most part about keeping the data available but not at all cost. I am not running an enterprise. The way I see it a vdev with 6 drives in RAIDZ-2 provides a beautifull balance between security, availability and economy of storage space. At least for a home server. And yes all my data is important to me. Not only private data but I keep some project data as well on my server. I work from my home at least 40% of my working time.

My backups however are all about security. Backup storage, being it my backup server or my cloud storage, is just that: backup. It will not be used for anything else. Loosing my main pool would be no joke and would be time consuming to say the least but it is not supposed to be a disaster either.

I should confess that I also have a lab computer with ESXi on it and a FreeNAS VM with its own HBA, storage disks and datasets. One of them being a always fairly recent copy of my main pool. If the need would be there I could update it from my backup server and switch to that one for the time being. I do realize that not everybody has that luxery.
 
Last edited:

zerohimself

Cadet
Joined
Mar 25, 2019
Messages
5
Ok, so i'm deciding on raidZ2, as this is a home, and once i have this system in place, it will be a good mix of both home/work, But nothing enterprise level.

I've got 2 questions now, I've been pricing online backup solutions, and it looks like i should expect to spend $30-$40 a month for online backups of about my first 10TB. Does anyone have a good suggestion on that? It will be for backup only, and i don't mind waiting a little or paying a little should i need to recover it.

2nd question: can anyone provide some good links on burn in procedures? I did see the one here on ixsystems, but always want a 2nd opinion/method to read up.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I've been pricing online backup solutions, and it looks like i should expect to spend $30-$40 a month for online backups of about my first 10TB.
If you look at $40 a month times just two years, that is $960 and three years of that would be $1440. This is the reason I built my own backup NAS instead of paying someone else to host my data. I setup my first FreeNAS server in 2011, and have been running a pair of them since 2012.
2nd question: can anyone provide some good links on burn in procedures? I did see the one here on ixsystems, but always want a 2nd opinion/method to read up.
I have used Darik's Boot and Nuke ("DBAN") as a way of testing the disks before putting them into service.
https://sourceforge.net/projects/dban/
It is really easy to do a 3 pass wipe and change the setting to do a check between passes. It is quick and easy to setup and you just walk away. I usually do a conveyance test if the disks support it, then a short and an long SMART test on the drives before running DBAN, then do another short, and a long after to make sure everything still looks good.
It is not much more difficult to do the testing within FreeNAS. Here is a good step-by-step guide for setting up a new FreeNAS system.

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

It was written for the old UI, so just sign in to the "Legacy" UI and it will look about the same, which might make following along easier.
Uncle Fester's guide has a section on disk burn-in and here is a script for it and some other good scripts.

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

I have used many methods of disk burn-in over the years and I still think DBAN is the easy-button answer, but it doesn't support some of the latest generation SAS controllers because it has not been updated in years.
 

Evertb1

Guru
Joined
May 31, 2016
Messages
700
I've got 2 questions now, I've been pricing online backup solutions, and it looks like i should expect to spend $30-$40 a month
My storage needs are modest and I don't need 10 TB cloud storage (for now). That's why my Office 365 home account comes in handy. Office 365 home comes with rights for me and five others to install Office and use it on the local PC and/or on the web. Every user has 1 TB of OneDrive cloudstorage.

At the moment I use the cloudstorage of three users for FreeNAS backup. The only other users are my son and my wife. And my wife does not use her cloudstorage. So I still can expand with 2 TB if needed. The official MS price for that account is 99 euro's a year. But I buy yearly extensions at web resellers sometimes for less then 75 euro's. So six times an Office account and 6 times 1 TB cloudstorage for a bit more then 6 euro's a month.

You can imagine that I was pretty happy for the support of OneDrive with Cloud Sync in FreeNAS. I know that there are a lot of people out there that don't trust their data with "evil" Microsoft but to me that is no problem. I use encryption anyways. And with those prices I most certainly don't complain.
 
Last edited:

zerohimself

Cadet
Joined
Mar 25, 2019
Messages
5
You can imagine that I was pretty happy for the support of OneDrive with Cloud Sync in FreeNAS. I know that there are a lot of people out there that don't trust their data with "evil" Microsoft but to me that is no problem. I use encryption anyways. And with those prices I most certainly don't complain.

Yes, I believe I already have a single user license, but it may be worth upgrading to use on my other PC's, and for the storage.. I think i just need to build a 2nd NAS-- however I need to wait a couple months before I can afford to add another build to this configuration..
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142

Evertb1

Guru
Joined
May 31, 2016
Messages
700
What kind of encryption do you use with backups stored in the cloud?
I use rclone crypt as offered when you set up the Cloud Sync task. I made the choice for Filename encryption as well but I think that is a bit overdone and making it a bit harder to recover a single file or folder when needed.
 
Top