BUILD Xeon E3 v5 Supermicro Build Check first FreeNAS

[Chiaki]

Hello everyone.
I'm new to FreeNAS and this community, I hope you go easy on me :). I've read some recommendations concerning hardware, FreeBSD and FreeNAS in general and have come up with this:

Background

• I'm a computer science student almost done with my master's degree with an all-in-all IT-experience in several fields of about 10 years.
• I didn't work with FreeNAS yet but have some years of experience with Linux systems setting them up as web-/mail-/voicechat-/etc.-servers, also in form of VMs via Proxmox VE.
• I didn't work with ZFS yet at all. All information I do have about it is what I've read about it on these forums (especially the great FAQs and presentations).
• I do understand that FreeNAS alone, without an additional backup strategy, is not an appropriate solution to keep data safe.
• I do understand that FreeNAS needs server-grade hardware and at least 8GB ECC-RAM to work reliably.
• I do understand that FreeNAS should be operated with a UPS to exclude data corruption due to power outages.

Intended Use

• I'm an IT-consultant for a lawyer office and they desperately need a good NAS-solution.
  • Right now they're using an ultra-low-end Buffalo NAS which was on RAID-0 when I first looked at it. It managed to transfer a brutal 4MByte/s and could go haywire any second.
  • No backup strategy was used at all, so I at least turned that NAS to RAID-1 and recommended an entirely different solution.
• The lawyer office will use the NAS-function for storing delicate data of their clients. I tend to encrypt it via geli. Windows and OS X machines will use the server as network storage.
  • Right now there should be 10 simultaneous users for the local network storage tops.
• FreeNAS needs decent hardware to run reliably so I thought we could as well make it run jails that act as a webserver (either LAMP-stack with Webmin or lightweight with nginx), mailserver and groupware (Horde).
• External backup-target via NFS for some of my own stuff, single user.

Proposed Build

• CPU: Intel Xeon E3-1245v5 Boxed
• Mainboard: SuperMicro X11SAE-M
• RAM: 4 x Crucial 16GB DDR4-2133 ECC (CT16G4WFD8213)
• HDD: 5 x WD Red 3TB (WD30EFRX) + 1 8TB Backup-HDD
• Chassis: Corsair Obsidian 650D (used, in good condition)
• SSD: Samsung 850 Evo 250GB M.2
• Hot-Swap: 5.25" to 3.5" SATA hot-swap slot
• UPS: APC Back-UPS Pro BR900G-GR
• PSU: Platimax 500W Enermax (80plus Platinum)

Reasoning behind Proposed Build

• CPU: Maximum HT Xeon E3 v5 with iGPU (if the need for GPU-acceleration may arise)
• Mainboard: (Hopefully enough) server-grade mainboard with
  • dual Gigabit-Ethernet
  • 8xSATA
  • M.2-slot (due to PCIe should not occupy one of the 8x SATA-ports, right?)
  • digital video output
• RAM: DDR4-ECC unbuffered RAM is very pricy. This one seems to have enough bang for the buck.
• HDD: On these forums WD Red HDDs were recommended a lot. 4 of the 5 HDDs will be used in RAID-Z2 (2 + 2) and the 5th will be used as spare for instant resilvering if the need may arise. The 8TB Backup-HDD will be used in the hot-swap slot for an externally stored backup of the zpool. This way 6 SATA-ports will be occupied at all times. (4 for the vdev, 1 for the spare and 1 for the hot-swap-slot) The 3TB WD Red HDDs have the better GByte per Euro-ratio than the 4TB WD Red HDDs right now for me.
• Chassis: Nothing special. I just happen to have a spare Corsair Obsidian 650D left and it should work well as a chassis.
• SSD: There were a lot of discussions whether to use USB or SATA DOM or SSD. Since the mainboard supports M.2 I thought of using an M.2 SSD, so as to not occupy a SATA port. For now I didn't want to use it as L2ARC (and for sure not as ZIL, since I don't have a mirror) but only as a boot drive. I may do that in the future, though, so I chose 250GB as the size.
• Hot-Swap: I read that backing up to external USB is discouraged because of bad behaviour concerning SMART and so on. You guys encouraged using eSATA but I don't have eSATA on this board. My solution is thus using a hot-swap slot that is connected to a SATA-port.
• UPS: I'm not sure about the chosen dimension of the UPS. All it should do is be big enough to let the server shut down properly as soon as possible after a power outage. From what I could calculate, this guy should hopefully be big enough.
• PSU: The server will run 24/7 so the PSU needs to have a good efficiency. Thus it will be platinum-grade.

Questions

1. Did I mess up somewhere with my build idea under the context of the intended use? How would you rate my build overall?
2. Is the backup strategy rigid enough, so the chance of losing all data is relatively low? (RAID-Z2 + ONE external regular backup) If not, what would you guys propose? What would you guys think is rigid enough for a lawyer office?
3. Is it wrong to plan using a FreeNAS server simultaneously as a server for other services? (Web/Mail/Groupware) I've read that there are other people using jails for this purpose successfully.
   1. In this context, is it clever to use one of the ethernet ports for the local network storage-purpose and the other ethernet port for the jails?
4. How much RAM should I feed FreeNAS with, so I can use the rest for the jails?
   1. I don't think I will need deduplication at first but if I do so later, can I just activate it?
5. I know expanding the vdev is possible by resilvering with bigger hard drives without changing their numbers, but what if I use the external backup? Remove the zpool, create a new vdev with more or less HDDs and put the data from the backup back in. Would this work?

 

[Sakuru]

1. Be very careful using FreeNAS's encryption feature, it's quite easy to lock yourself out. If you need encryption, look into something like TrueCrypt or one of its forks. Your hardware choices look good.
2. I'm personally not a fan of "rotating backups". I prefer to replicate to another system or use a cloud backup service. In your case I would highly recommend replicating to another FreeNAS box offsite.
3. Jails are fine
4. I'm not sure I understand this question. FreeNAS will use all of the RAM attached to the motherboard.
   1. Like encryption, be very careful with deduplication. Unless you have many copies of the same thing it's likely not worth it.
5. You can do it both ways, but it's safer to replace 1 drive at a time and let it resilver.

 

[m0nkey_]

The lawyer office will use the NAS-function for storing delicate data of their clients. I tend to encrypt it via geli

Only comment I have on this is make sure you backup your recovery and GELI keys and store them in a safe place. The last thing you want is to lose that delicate data. Using encryption does not prevent file level access to the server, it's only intended to prevent access in the event physical drives are stolen. Also note that drive replacement of an encrypted pool is more complicated. Providing you understand this and are able to follow the documentation provided, you should be okay. That said, regular backups and test your encrypted pool for recovery before going live.

 

[Chiaki]

@Sakuru

1. Is there a good alternative to the native GELI encryption that works with a minimal setup? TrueCrypt seems to be safe right now but then again it is not being maintained anymore so I've heard.
2. What do you think about using the Buffalo NAS as a backup-storage? It's slow as hell, though, and I don't see the advantage over a rotating off-site backup. External backup storages aren't an option, because the data needs to stay within the lawyers office. Maybe if the data would be encrypted this could be an option but I don't know of a minimal setup and cheap solution that could be used to restore the FreeNAS data. Is there a guide for restoring from external data source which is encrypted?
3. That's great to hear
4. Maybe I had the wrong idea here but for VMs you provide a slice of the physical RAM they can use. Do jails just use the RAM that they need? Do you not need to define the amount of RAM they may use?
   1. I will not use deduplication I guess. I see no real scenario where a lawyers office would have a lot of identical data lying around.
5. My problem is that I'd like to increase the number of drives in the vdev to 4 + 2 sometime to increase the speed and storage. You can't do this by resilvering. I guess I will have to destroy the zpool, create a new one and put the data from the backup in it.

@m0nkey_
In the unlikely event of a confiscation or theft of the whole server I want all of the data to be encrypted so as to not only restrict direct SATA access from a foreign machine but also access via the server itself. This should be achievable through GELI and password-restriction on the server, right?

EDIT: I just realized, that upgrading CPU and mainboard from E3 v5 to E5 v4 would cost only ~150€ more. This would mean changing from the proposed to

• CPU: Xeon E5-2620v4
• Mainboard: Supermicro X10SRi-F
• RAM: 4x Crucial 16GB DDR4-2133 CL15 ECC RDIMM CT16G4RFD4213
• SSD: Samsung 850 Evo 250GB (not as M.2 but as SATA, due to 10 SATA-ports instead of 8 and no M.2 port)
  • The mainboard can use SuperDOMs but I don't agree with paying the price of a premium 250GB SSD for a 32GB SuperDOM, that's ridiculous.

Do you guys think we should do this? It's in the budget.

 

[m0nkey_]

In the unlikely event of a confiscation or theft of the whole server I want all of the data to be encrypted so as to not only restrict direct SATA access from a foreign machine but also access via the server itself. This should be achievable through GELI and password-restriction on the server, right?

No. The built-in encryption is there to prevent access in case of theft, as per the documentation:

Note: the encryption facility used by FreeNAS® is designed to protect against physical theft of the disks. It is not designed to protect against unauthorized software access. Ensure that only authorized users have access to the administrative GUI and that proper permissions are set on shares if sensitive data is stored on the system.

http://doc.freenas.org/9.10/freenas_storage.html#encryption

If you want file-level encryption, then you will need to use something like VeraCrypt encrypted containers.

 

[Chiaki]

@m0nkey_
But the documentation you quoted says: "Ensure that only authorized users have access to the administrative GUI and that proper permissions are set on shares if sensitive data is stored on the system."
So what if I enable a password on power-on of the server that has to be put in before FreeNAS even properly loads? (BIOS password won't work due to BIOS battery removal, so it has to be after boot but before initialization of FreeNAS.)

This way a shutdown or removal of power supply should restrict any access to the data. Mounting the disks on a foreign machine won't work and you can't access them on the server itself because you're missing the password for the proper initialization of the OS.

 

[RichTJ99]

Wouldnt single disks be useless in the Raidz file format? It seems like if someone stole all disks you would have a problem? Or is it a matter of stealing the whole machine, installing a new freenas install on new USB & importing the disks to the pool?

 

[Chiaki]

@RichTJ99 Why do you think I would create a vdev with a single disk?

It's a problem if anyone who isn't allowed to (even if legally) can view the contents of the zpool. Delicate data of clients will be stored there. If someone wants that data I want them to have to go through court just like what happened with Apple.

 

[danb35]

Maybe if the data would be encrypted this could be an option but I don't know of a minimal setup and cheap solution that could be used to restore the FreeNAS data. Is there a guide for restoring from external data source which is encrypted?

Crashplan will do this, at least in the US (I assume they're available in Europe as well). It's not the default, but there's a setting you can use to generate and store encryption keys locally, so that nobody else will be able to use the data. Of course, that means that having a good backup of the key(s) is critical, because otherwise the data is unusable.

There's a FreeNAS plugin for Crashplan, but some folks (including myself) have had trouble getting it to work properly. I run it in an Ubuntu VM, and mount the appropriate directories from my FreeNAS server as NFS exports.

 

[Sakuru]

Some people have gotten it to work in a jail: https://forums.freenas.org/index.php?threads/tested-crashplan-4-5-setup.41418/

 

[danb35]

Some people have gotten it to work in a jail: https://forums.freenas.org/index.php?threads/tested-crashplan-4-5-setup.41418/

Indeed. The fundamental problem, IMO, is that Crashplan simply isn't designed to operate headlessly. Whether Code42 are intentionally hostile to that use case, or merely indifferent, I don't know. I do know, though, that (1) their instructions for headless operation are needlessly complicated, introducing an SSH tunnel that has no reason to be there; (2) the details of what's needed to operate headlessly (e.g., the existence, location, and/or contents of the ui.info file) change significantly from version to version; and (3) at least in the case of the FreeNAS plugin, the UUID changes every time it restarts, requiring that the "client" computer edit ui.info to reflect the new number. I used it, and had it working fine, until about version 4.3, IIRC. After that, I couldn't get the plugin to work any more, so I just spun up an Ubuntu VM and did it that way. No more headless hassles to deal with.

I'd love to find something Crashplan-like that would run on FreeNAS, wasn't coded in Java, and was comparably-priced, but I don't know of any such thing.

 

[Sakuru]

Well, there's Duplicati, but it's not perfect either.

 

[RichTJ99]

@RichTJ99 Why do you think I would create a vdev with a single disk?

It's a problem if anyone who isn't allowed to (even if legally) can view the contents of the zpool. Delicate data of clients will be stored there. If someone wants that data I want them to have to go through court just like what happened with Apple.

Hi - Sorry - I meant if you had a multi disk Vdev, someone grabbing 'a' drive would have nothing at all - just a portion of the entire drive but no readable data. It would only work if the thief had all the drives (right?)

 

[Chiaki]

Hi everyone, sorry for taking so long to answer.

@danb35 @Sakuru It's ok if the encryption is dangerous in the sense that if I lose the encryption keys all data might be gone. But I do not want additional costs which Crashplan would generate if I understood correctly. Is an offsite-backup really not good enough? Are we speaking about data rendundancy here or a countermeasure for the "shit hits the fan"-scenario? What about using the Buffalo NAS as one (daily) offsite redundancy node and the 8TB Hot Swap HDD as a second (weekly/monthly) offsite rendundancy node?

@RichTJ99 To put it in very simple and precise words: I do not want anyone, except for a few authorized people, to be able to view the data on the zpool in any possible way. I do not want them to be able to do this, even if they steal or confiscate the whole server.

Please do comment on the upgrade to LGA 2011-3 platform because I am on the verge of ordering the parts.

Thanks for the help up till now, everyone :). Nice community.

 

[SweetAndLow]

Please do comment on the upgrade to LGA 2011-3 platform because I am on the verge of ordering the parts.

Thanks for the help up till now, everyone :). Nice community.

I would skip the GPU nonsense. Other than that CPU and motherboard look good, the x10srl will be a little cheaper if you want to look into that. You keep worrying about not using a sata port for your ssd but you are only using a couple drives to begin with so stop worrying about your sata ports.

You should also get a better power supply from a more reliable company.

 

[Chiaki]

@SweetAndLow
Aside from the GPU nonsense or not discussion I want to comment on the original version of your post.

First off, it seems that you edited out the statement of yours saying that my later mentioned parts are cheaper, older than the original parts and not an upgrade. I'm glad you realized that you were wrong. Thanks for the mainboard recommendation I will look into it and thanks for "signing" my choice of hardware in the end :).

It looks like I will be using a very fast USB flash drive for FreeNAS after all, after I've read some more on the topic, so using up SATA ports is no problem anyway.

I don't see how Enermax is not a reliable PSU company. Enermax has a good reputation in my local IT scene. Can you please provide a source for Enermax not being reliable?

 

[Chiaki]

@SweetAndLow
I checked the X10SRL and decided to still choose the X10SRi-F over it, because the latter costs only 5€ more for me and provides i350 LAN chipset instead of i210.
This way I get more interfaces, Intel VT-c and On-chip QoS.