Homeserver for Filestorage and Plex

[ude6]

Hi all,
this is my first post to the Forum. I am currently running a Linux Server with ZFS that holds about 25TB of data.
But Capacity is nearing the 80% Level so I am looking to build something new.

The workload is mainly fileserving for 4people and a plex Server with 1-3 streams and maybe a Linux VM.

I tried to read-up on the advice given in the forums and this is what I have come up with also looking at the
availability here in Germany:

1 * Chassis INTER-TECH IPC 4-HU-4424
1 * Motherboard Supermicro X10-SRi-F
1 * CPU Xeon E5-1650 v4
1 * CPU Cooler Noctua NH-U9DX i4
4 * RAM - Hynix - DDR4 - 32 GB - DIMM 288-PIN - 2400 MHz / PC4-19200 - CL17 - 1.2 V - registriert - ECC (HMA84GR7MFR4N-UH) --- the only supported 32G RAM modules I found as
available ---Other Ideas?
1 * PSU Corsair RMx 1000
3 * HBA Dell PERC h310's or IBM ServeRAID M5110, PCIe 3.0 x8 (00D7082) ------ Which is perfered?
3 * HD Fans Noctua NF-F12 PWM
2 * Exhaust Fans Noctua NF-A8 PWM
2 * Boot Samsung SSD 850 PRO 256GB, SATA (MZ-7KE256BW)
8 * HDs Western Digital WD Red 8TB, 3.5", SATA 6Gb/s (WD80EFZX)

The SSDs would be used as a mirrored boot/System. The Datadrives I would be looking to set-up as a single raidz3 (with the Option to 2 to more vdevs later on).

I would be happy for any comments and ideas....


Thanks!

ude6

 

[danb35]

I would be happy for any comments and ideas

Your proposed build really sounds like overkill for your stated use case--both the RAM and the CPU are very excessive, the SSDs are far larger than necessary (unless you just can't find anything suitable cheaper), and mirroring boot SSDs seems unnecessary. Suggest you look into the X11SSL-CF, the current version of the E3-1230, and two 16 GB sticks of RAM.

I'm not at all familiar with your proposed chassis, but if it's available in a version with a SAS expander backplane, that would be worth looking into--then you'd only need a single HBA (which is included on the motherboard I suggested above). Otherwise, used Supermicro gear is always a good option, but may be hard to find in .de.

 

[ude6]

Hi,
thanks for the quick reply.
I agree that the SSDs are probably Overkill, but I have them around and might as well use them. Could I Partition them and use the non-os part as the storage for the Linux VM?

As for the CPU and RAM: Well... :) ... I guess it is overkill. Might think about this again.

The case is the imported/re-branded Version of the NORCO RPC-422. The server will run in the Basement but should be resonably quit. I hear that it is quite easy to use the 120mm fans in the Chassis and that would be good. Also one can use a "normal" ATX power supply - which is not the case for most of the other cases if I am not mistaken. Again my concern there is noise.

 

[nightshade00013]

No you can not partition them. The entire SSD will be used by the FreeNAS installer. You could use one to hold the install and the other to hold VM's though. You just want to make sure you configure everything correctly when you go through and do the setup.

 

[ude6]

ok. Thank you nightshade00013. I have been burned by a dying boot/systemdisk in the past. I wanted to avoid this by mirroring.

 

[nightshade00013]

If you make backups of your config when you makes changes or after an upgrade it really does not matter. You reinstall fresh and import the config. If you REALLY want to have the latest version of the config there is a script that can be used to automatically backup the config as a cron job. Set daily to backup to the pool you will be able to easily recover your install.

 

[Chris Moore]

ok. Thank you nightshade00013. I have been burned by a dying boot/systemdisk in the past. I wanted to avoid this by mirroring.

Welcome to the forum.
If this is your first venture into FreeNAS, there is a lot to learn. FreeNAS takes care of a lot of the things for you that Linux left you to do on your own or even at the command prompt but there are still little nuances to FreeNAS that must be handled, so do take advantage of the forum and read as much as you can. There have been some significant changes from version to version so if you have questions, please ask.

I use the Dell h310 cards in three different servers. One server has three cards to support 24 direct connected drives and the other two servers have two cards to support 16 direct connected drives. These cards cross-flash to the Avago / LSI firmware perfectly and work great. They are often available for less money than the others and the hardware is good. Personal experience as I have been using them for 3 to 4 years now. Full disclosure, I did have one start being unreliable, it stopped showing up at boot sometimes but worked for almost a year before it stopped showing up at all. I replaced it with another of the same model and the server is working great with the replacement card. These cards do get warm and I have replaced the original fans in my servers with slower fans (less noise) and I think the card may have overheated due to not getting enough air.

If you are going to run a few VMs, having the extra CPU power and RAM might be nice. If you can afford it, why not? You could use the SSDs for a dedicated ZFS pool just for VMs, if that is interesting to you. The latest version of FreeNAS has good support for creating VMs and you can even run Windows in a VM, I am told, and I will be setting that up myself as soon as time permits. If it works as well as I hope, I may be able to eliminate my dedicated VM host.

For the boot drives, I suggest using a pair of 40 GB laptop hard drives that you can pickup on ebay for around $10 each (not sure the cost in Europe) but they are cheap here and they are MUCH more reliable. I have been using the HGST brand drives in mirror sets in my servers for years and have had zero problems. You can also use the smartmontools to track their health as they are actual hard drives. I have that configuration in some of the servers at work and in my personal servers at home. I have not had a drive fault take a server down since I started doing that. Still, for safety sake, make backups of the config file for FreeNAS because there are other problems that can cause you to need to reinstall the boot pool and once you reload the config file, it brings the system back to almost exactly where it was when the config backup was made.

 

[Chris Moore]

Another thought crossed my mind. I looked into this for work recently and found that the HGST drives were the best price option up against Seagate and WD. The HGST Deskstar 8 TB Internal HDD - 3.5" is only $269 (here in the US) where the comparable Seagate and WD drives were both more, at the time I did the comparison. I can't afford them myself.
Also, the HGST drives have been the least failure prone drives we have ever run in our data center at work and I have one at home with over 50000 hours (5.7 years) and it still works like new.

 

[lopr]

for bootdisk:
just use 2 or three quality brand usb-sticks. freenas won't really read or write from them unless startup/upgrade. keep the SSDs in to use for jails/vms and the system dataset. or use the SATA slots and bayspace for bigger pool.
mirroring the bootdisk is great for hardware failure, however do backup your config before upgrading. yesterday an upgrade killed my 3-way bootdisk mirror (bootrecord somehow got messed up, reinstall of freenas on the same sticks found the existing config and asked if it should use it; booted up like nothing ever happened)

cpu-ram:
with that amout planned you have nothing to worry about deduping the SSDs and running a few vm's with dedicated ressources

 

[Chris Moore]

for bootdisk:
just use 2 or three quality brand usb-sticks. freenas won't really read or write from them unless startup/upgrade. keep the SSDs in to use for jails/vms and the system dataset. or use the SATA slots and bayspace for bigger pool.
mirroring the bootdisk is great for hardware failure, however do backup your config before upgrading. yesterday an upgrade killed my 3-way bootdisk mirror (bootrecord somehow got messed up, reinstall of freenas on the same sticks found the existing config and asked if it should use it; booted up like nothing ever happened)

This is what I am talking about: http://www.ebay.com/itm/Fujitsu-40G...760216&hash=item25dc3f6b6f:g:Lt0AAOSw3v5Yn6d2
Put a pair of those, don't even need to match brand, and you have a boot pool that will last the life of the system. If one fails, the system will still be up on the other and you can easily replace the failed drive. I have four servers running that way right now and they have been problem free for over two years. Those drives are made to handle the bumps of a mobile environment, when they are sitting still and rarely used, they last for ages. We decommissioned some servers at work that had that setup and the boot drives had lasted the entire life of the server without ever having a problem, 8 to 10 years in this case, because they were cold storage servers.

 

[ude6]

Thanks Chris Moore and lopr for the Feedback.
I guess I will be using the SSDs in a dedicated pool then and pick up some small drives for the boot as you suggest Chris Moore. My Intention is basically also what you describe...one drive dies, just replace and the Server keeps running. The case does have room for 2.5 drives internally so this will not block the 3.5 bays.

Also I saw that People are using Broadcom 9211-8is as HBAs. Which is the preffered one H130s or the Broadcom 9211-8i?

 

[Chris Moore]

Thanks Chris Moore and lopr for the Feedback.
I guess I will be using the SSDs in a dedicated pool then and pick up some small drives for the boot as you suggest Chris Moore. My Intention is basically also what you describe...one drive dies, just replace and the Server keeps running. The case does have room for 2.5 drives internally so this will not block the 3.5 bays.

Also I saw that People are using Broadcom 9211-8is as HBAs. Which is the preffered one H130s or the Broadcom 9211-8i?

Sorry, the name of the company has changed several times in what I think of as being "recent" history. The SAS controller technology that is now owned by Broadcom, was previously owned by a company named Avago and before that it was called LSI. There may have been some small changes along the way but I have been using the Dell H310 controller that can easily be sourced on ebay: http://www.ebay.com/itm/Dell-H310-6...544925?hash=item25dbceacdd:g:3TgAAOSwTf9ZWHPf
These cards can be flashed with the IT mode firmware and used as HBA cards and they work perfectly. If done properly, they even appear to the operating system as actually being the LSI / Avago / Broadcom part instead of being a Dell part. It is the same chip under the heatsync and you load the proper firmware on the card, so it just works. The one I linked to has already been flashed by the seller, so the work is done. This is what I have been using for several years and the only one that has failed on me was (I think) because I let it get too hot. I need to improve the airflow over the heatsync. I have my fans running slow to keep the sound down and that is allowing too much heat to accumulate inside the chassis. I am using five of these cards on a daily basis for over three years and only had one failure and I think that was my fault. It didn't cause any problems with my data, the card just stopped working, once I replaced it everything worked fine again.

Edit: Sorry, I didn't realize I was telling the same story again in the same thread.

 

[ude6]

ok my bad. I did not realize that they are the same thing. Sorry.

 

[ude6]

ok...new developments. I found a SC847 (SC847E16-R1400LPB) for a good Price on Ebay.
So I will have 2 SAS2 Backplanes to connect to: BPN-SAS2-846EL1 / BPN-SAS2-826EL1.
As those are SAS2 and all X10 boards I saw use SAS3 Controllers I assume I should be
using an external SAS2 HBA still?

 

[Chris Moore]

You can run the SAS2 backplane on a SAS3 controller. The problem is that a SAS2 expander will only work with 2TB and smaller drives.
That's my understanding. Anyone else who has used that specific hardware might have more information.

Edit: Sorry, mistakes were made (SAS vs SAS2) I was confused. I upgraded from SAS to SAS2 for the reason above. SAS3 is the new 12Gb/s controller that I have in some of the systems at work, but I have not got at home because it is still too rich for my personal budget.

 

[ude6]

Thanks @ChrisMoore. I think you are worng on this one the Expander is the SAS2 and not SAS. According to https://www.supermicro.com/products/nfo/files/storage/SAS-CompList.pdf (see the Hitachi) and various Forum Posts.
The Limit is correct for the "SAS" (i.e. 1.5G link).
The SAS2 backplane to SAS3 Controller´: Sure about that? I read about asking for Trouble for example here: https://forums.freenas.org/index.php?threads/supermicro-hbas-to-backplane-questions.38238/ but that was 2015...

 

[danb35]

The problem is that a SAS2 expander will only work with 2TB and smaller drives.

No, this is not correct. You're thinking of a SAS1 expander. SAS2 expanders (specifically, the very backplanes @ude6 refers to, which I also have in my server) work perfectly with drives as large as 6 TB (which is the largest I've used so far), and I understand much larger.

 

[Stux]

Your build sounds like it might be a clone of mine:
https://forums.freenas.org/index.ph...24-supermicro-x10-sri-f-xeon-e5-1650v4.46262/

I'd suggest using the Noctua Industrial 120mm 3000rpm version of your specified fans.
http://noctua.at/en/nf-f12-industrialppc-3000-pwm

With the 1500rpm fans you've specified, I found when all 24 bays were loaded, and 8 of the drives were 7200rpm drives, they ran a bit hot. 3000rpm versions of those fans should resolve it.

I actually have them here to replace the NF-F12 1500rpms... just haven't yet.


edit: this doesn't apply to the Supermicro chassis you've found...

 

[ude6]

@danb35 : Thanks for confirming that SAS vs SAS2 issue. Which HBA would you recommend an the X10 series? External SAS2 or using the X10SRH-CF?
@Stux : Yes...the build inspired me I think for cooling I will start with the stock fans and PWM (attached to the MB) and take it from there. In the Initial build there was room for the 120s but this case only has room for 7! 80s....
Also I am looking to replace the PSUs with a single PWS-920P-SQ.

 

[danb35]

Which HBA would you recommend an the X10 series? External SAS2 or using the X10SRH-CF?

AFAIK, the onboard SAS3 HBA should work fine with the SAS2 backplanes, assuming the proper cabling.