Sanity check on hardware. Advice on case, SSD

[TomatoSoup]

First post on the forums, but I've lurked a while and read up on the documentation.

My intended core hardware is

Intel i3 6100 https://www.amazon.com/dp/B015VPX2EO/?tag=ozlp-20
Supermicro MBD-X11SSL-F-O http://www.newegg.com/Product/Product.aspx?Item=N82E16813182980
2x (1 to start) Crucial CT7982341 http://www.crucial.com/usa/en/x11ssl-f/CT7982341

I have 4x 4tb drives I intend to run in RAIDZ1, and 2x 80gb Intel SSDs (of, admittedly, somewhat dubious origin. They were $20 each on ebay, I couldn't resist).

It'll mostly serve media, although I'd also like to be able to use it for some small VMs from my ESXi server. The only plugins/jails/etc I'd like to run would be a cloud backup service, a personal cloud server, and a MySQL server. Incidentally, what cloud backup is in vogue with FreeNAS these days? I heard rumors that Crashplan was getting to be too much of a hassle.

I intend to use compression and encryption. And perhaps I'll turn on dedup for a 1-2tb store for my VM backups. I'll check the benefits first, of course.


My three questions are:
Does that hardware make sense for that use-case? Does that hardware make sense, period? I can't imagine upgrading in a while, this will hold all of my existing data more than twice over if I backed up literally everything I have, so I'm okay with the limited SATA ports.
What is a good case for this? Considering I need to hold 4x 3.5in drives, 2x 2.5in, and a Micro ATX motherboard.
What's the best thing I can do with the SSDs? I believe I'll see a benefit with a ZIL (I believe ESXi over NFS and MySQL use sync writes). Likewise, I might see a benefit for an L2ARC. How should I best partition my SSDs?

 

[nightshade00013]

It's going to have poor performance for VM's unless it's only used for the backups. RaidZ1 with 4TB drives isn't a great idea either and adding anything like ZIL, SLOG or L2ARC will only make your headaches worse. The hardware you are selecting would be fine for some work as a file server/media server but it's likely going to be bad for throwing everything you described at it especially with deDup enabled.

You REALLY need to click on the links in my signature and read up before you jump into this. It wouldn't hurt to search the forums about VM storage as well.

 

[TomatoSoup]

Poor performance for VMs is totally acceptable. The two I have in mind are my torrent/vpn VM and a VM that contains a number of webscrapers and node.js-based socket relays. The other VMs will be backed up, though.

Why do you say that RaidZ1 with 4tb is a bad idea? I understand that another disk failure can kill the pool, but I've purchased my 4 disks from 4 different vendors, so any failures shouldn't be so correlated that I don't have a chance to replace and rebuild. Incidentally, speaking of your sig, you should remove the one about RAID5. It's totally non-applicable for ZFS. If ZFS encounters a URE it just marks that block as unrecoverable and carries on its merry way.

Why do you think that the various caches will give me headaches? I understand that they're complicated to set up and aren't applicable to many situations, but I believe I have a use-case where they will provide a measurable benefit. At least, for the ZIL. I agree I probably won't see any benefit from the L2ARC.

Why does dedupe make my problems even worse? I was under the impression it just required a fair chunk of RAM.

 

[SweetAndLow]

The article about raid5/raidz1 isn't irrelevant. It basically says if during a rebuild you have a read error you will lose that data and if that data is zfs meta data your will lose the pool. Most people lose their pools. Dedup is usually worthless and there isn't any value in using it.

For the slog you need to do testing with your use case to decide if it's going to help you.

 

[TomatoSoup]

I was under the impression ZFS keeps 2 copies of metadata, with pool-critical kept in triplicate. I know dedupe ranges from eh to total waste of time, which is why I said I'd measure its benefit first.

I agree I need to test if the slog is useful or not. I guess my problem here is that I already have the SSDs and I'd rather like to find them a home. Perhaps the better question is just, what *should* I do with them?

 

[nightshade00013]

Use one or both SSD's in a mirror as the boot devices. If you only use one then place the jails on the other one.

As far as Raid5 RaidZ1 being bad, it's Murphy's law vs What you read somewhere. If it can go wrong it will and my guess is that while the size of your pool is currently sufficient eventually you will need to grow and if you are like some other infamous people you will either throw 6TB drives in as a second vDev or you will just grow the pool by replacing the drives. Either case is a bad move since losing one vDev is a loss of the pool and the math shows that a 6TB drive is at around a 60% chance of having a URE during a rebuild. Even if the chance is as low as 25% do you really want to risk it. That also does not account for a second drive failing and buying drives from different sources will not guarantee reducing the risk. ZFS is a great storage system but it's not infallible.

You are free to choose whatever you want to do, you could throw all four drives in stripe and have great speed from the pool but you better have great backups to go with it or prepare to deal with the consequences of your decision. All we are trying to do is explain the direction your choice will likely take you.

 

[joeschmuck]

Here is my take on this thread...
@TomatoSoup I highly recommend you build your system and "test" with it. Do not commit any data to it so when it dies while you are testing, you haven't lost anything. What I recommend you do during that testing is to put a large chunk of data on your system, run the dd command and try to fill up one of the hard drives with zero's in order to simulate a failure (you can do something else but you want to corrupt the data on it). Fix it. Find several different ways to cause harm to your data and see if you can fix it.

The results will likely be that you find out encryption is not worth it and that RAIDZ1 really needs to be RAIDZ2. If you don't know this already, you cannot add a single drive at a later date to the pool to make a RAIDZ2, you would need to destroy the pool and recreate it as a RAIDZ2. And you will find out that if you have an appreciable amount of data on your drive, the resilvering process will take a considerable amount of time. Make sure to test using the ZIL/L2Arc as well but test using appropriate tools.

My overall advice would be: No encryption, No ZIL, No L2Arc, use one SSD as a boot device, know what your storage requirements are for the next 3 years and double that value then buy the proper hard drives to achieve that goal and make the pool a RAIDZ2.

Let me also point out that you only have 6 SATA ports on this motherboard and if you are not planning to use an additional HBA then you should know that limitation. I'd shoot for five hard drives in a RAIDZ2 and one SSD as the boot device. Wait, I've already said that.

If you find that your system performance is lacking when using it to host the VMs, increase your RAM because there is no substitute for it. Remember, the RAM is the primary cache and if you are running a VM which could fit into the cache, your performance will go up.

Sorry for rambling on, it's definitely a sign of age.

Case selection will be up to you but one of the main things to consider is air flow. Sure you need to consider if the components will fit but you need to keep the hard drives and motherboard cool. Selecting a case which has proper air flow across the drives is key for the life of the hard drives. I prefer a case with fans which pull air into the case in the front and force it across the drives and then out the back of the case. For a FreeNAS system, you generally do not create a lot of heat from the CPU but some of the motherboard components will get hot so again, make sure there is air flow. If you are thinking about a specific small case, post the link to the case so someone can give you feedback.

Done!

 

[TomatoSoup]

@nightshade00013
UREs aren't death for me, though. Most of the data will be media and I can easily recover that. I know multiple sources doesn't guarantee that another drive won't die, but it certainly helps my odds.

@joeschmuck
Why do you say encryption isn't worth it?

But alright. You guys have sold me on going for RaidZ2. I'd prefer not to go with the 5-drive version, though. The price:storage ratio is a little bit too unfavorable for me there. I'd prefer to use 6-8 drives, which also means I'll be using a different motherboard, as I'd like to use the SSDs for a mirrored boot and jail drive. I'm sold on not using L2ARC, but I'll definitely experiment with a few gigs of ZIL.

Besides cost and the increased odds of two more drives failing during a rebuild, is there a reason not to go with 8 drives?

 

[nightshade00013]

I believe the biggest nail in the coffin on encryption is the support for it in FreeNAS 10 was not confirmed the last I read about it.

You can use the board you looked at to begin with but you will have to pick up an HBA to attach more drives to. A lot of people use one based on the SAS 2008 like the IBM 1015.

You can go with as many drives as you like but generally for a raidZ2 we try to keep it around no more than ten. Eight should be just fine though. At one point in time there was a reason to stick with certain numbers of drives but that point is moot with compression enabled which in and of itself is good and will have minimal impact on performance with data that is not compressible but will see great benefits with data that is.

 

[joeschmuck]

Why do you say encryption isn't worth it?

Replacing a failed drive. I strongly recommend that if you want encryption that you create a scenario and replace a hard drive. There are very specific steps you must go through or you will not replace the drive properly, end up loosing all your data, and just not have a fun day. And once you have it working, I'd recommend you create a guide customized for your system so in 3+ years when drives start failing, you can fall back on something you wrote and replace the drive without much pain.

If you need some of the data encrypted then you are better off using some third party tool like TrueCrypt.

but I'll definitely experiment with a few gigs of ZIL.

If you are going to use a ZIL (which is a bad idea in my opinion) then ensure you take all the appropriate precautions of using the correct type of SSD.

Besides cost and the increased odds of two more drives failing during a rebuild, is there a reason not to go with 8 drives?

More heat so more cooling is required, more power needed, but functionally no issues at all and it can be faster for you if you do it right, but I think the question is: Do you need 8 drives? I run six drives, I started out with four drives and eventually added two more drives. I had to destroy my pool but that was okay. I now have been running my system for several years and all my drives have exceeded the 3 year warranty but they are still going strong. I have used only around 55% of my pool size and it's doubtful I will use much more over the next 3 years. Most of my data are backups of my computer systems in the house. I have already planned to reduce my drive count down to four drives when I have my first failure. This makes for a slower system but for my uses that is fine. So my point was you could start with a smaller set of drives and add later but you would have to destroy all your data in order to do that. Since you maintain backups, that should be no problem.

 

[TomatoSoup]

Using True/VeraCrypt over a share sounds like a much better idea, once you put it that way.

Considering I don't intend to run anything performance-critical that uses sync writes, I may forgo ZIL anyway for the sake of ease of administration.

You're right. I definitely don't need eight drives. The total storage on everything in my house right now is 5tb, so 24 is far beyond enough. But on the other hand, I've very often opted not to download something due to lack of space and I've often deleted stuff that I would later redownload. So I figure, why not. It's not that much more and it gets me both a redundancy and price:storage ratio I'm happy with.

If I'd done it all over I would have started with 8x2tb and then, as they failed, expanded to 4tb drives.

Thank you very much for your help.

 

[Spearfoot]

If you are going to use a ZIL (which is a bad idea in my opinion) then ensure you take all the appropriate precautions of using the correct type of SSD.

@TomatoSoup, I agree with @joeschmuck about not using a ZIL SLOG device.

These are appropriate in enterprise systems, particularly servers that are hosting umpteen VM users. But in your case, where you're only going to be running a few non-critical VMs, just turn off synchronous writes for the dataset where you store the VMs. You'll actually get much better performance than you will with a ZIL SLOG -- I know, 'cause I ran my system that way for nearly a year. Keep good backups, in case of a catastrophic power failure, and you're Good-To-Go.

If you do decide to use a ZIL SLOG, pick a low latency, fast-write-speed SSD with high write endurance, like the Intel DC S3700/S3710.

 

[joeschmuck]

The total storage on everything in my house right now is 5tb, so 24 is far beyond enough.

If I'd done it all over I would have started with 8x2tb and then, as they failed, expanded to 4tb drives.

If your total storage for a NAS is 5TB then might I recommend you look at six 3TB drives which would give you ~9.4TB of uncompressed usable storage. That is almost double your current capacity. Also, use one of the RAID calculators in my signature to calculate how much storage you get with the drives you select because eight 3TB drives is 24TB of raw capacity but after formatting, swap space, and subtracting 10%, you end up with a lot less than you would expect, but the positive thing to take away is a very robust file system where your data is safe from drive loss and one of the safest for data corruption.

 

[TomatoSoup]

I considered 3tb drives, but I was under the impression that they were a bit of an interim technology? The 3tb Seagate drives had awful reliability, and I'm not certain I trust the other manufacturers with that.

Also, I happen to have jumped the gun and bought 4x4tb in preparation, thinking the only thing I had left to decide was the processor and motherboard. So I jumped the gun again earlier today and bought another 4.

Sure, 5tb is all I have, but I delete things I'd rather hold on to in order to make room for new things. So I think my normal rate of acquisition justifies about 20tb of storage.

 

[joeschmuck]

Also, I happen to have jumped the gun and bought 4x4tb in preparation, thinking the only thing I had left to decide was the processor and motherboard. So I jumped the gun again earlier today and bought another 4.

Damn, you sound like me. So do you have a motherboard in mind? Hopefully you find something with 9 SATA ports and then you could use all eight hard drives and one SSD as the boot device. Or you could just use seven of the drives and keep the other one as a spare but that is such a waste in my opinion unless you have another use for it. You can of course use a USB device for the boot drive but if you can use a SSD, you will be better off in the long run. USB flash drives fail, and yes SSDs will fail but odds are 10+ years for that and the USB flash drive could be 6 months, 1 year, 2 years. If you go with USB, get a USB 2.0 device and buy a good one. I could recommend a few but USB 3.0 devices tend to die due to heat buildup. Of course if you must have USB 3.0, I have one I've been using for almost 5 years and it's constantly in use but it wasn't cheap, not even sure they make this type these days but I'll give you the info on it if you desire it.

 

[TomatoSoup]

I've decided to go with the motherboard I was first eyeballing and get the LSI HBA. I'll use the SSDs for jails (mirrored) and use USB for boot drives (again, mirrored).

Turns out there aren't that many LGA 1151 Micro ATX boards with IPMI and >6 SATA ports. And the ones that are out there cost a fair bit more than what the cheaper motherboard and HBA would run you.

For the case, I've decided on the Node 804. 8 3.5in drives, 2 2.5in drives, Micro ATX, hella fans. What more could I ask for.