New FreeNAS build, input welcome :)

[RobiWahn]

First of all, hello and thanks to you all for all the resources I already found!

As I have built some FreeNAS servers at work already (but always in a hurry, so no time to ask here first), I already could make use of many of your ideas and recommendations for those - and the servers at work run exellent, so again, thank you!

Now I'm in the process of wanting to switch the single drives with stuff (bluray backups, music library etc) in my desktop (Windows 10) at home over to a RAIDz2 on a file server to be built. As this is nothing where I have my boss sitting in my neck, I have time and thought I could Do It Right(tm) this time ;)

What I want (listed by priority):

• Have the data safe and secure (At least photos and videos of family, the backups i could make again any day, but I want to get the big drives out of my desktop anyway and FreeNAS is perfect for that)
• Have all the storage available via 10GbE for my desktop to quickly offload data and bluray backups to (DP 10GbE Intel X540 in desktop already, would use extra direct connection for that)
• Have all the storage available via 1GbE (or what the ports/wireless makes) via switch/wireless router for my other machines (Raspberry Pi with Kodi as MediaCenter, windows desktop of GF, windows/linux laptops, e-reader and so on)
• Be able to add my desktop's 4 Seagate ST8000AS0002 8TB drives as soon as burn-in testing and copy to RAIDz2 pool is complete (as another RAIDz2 vdev i guess, or is expansion of the first RAIDz2 possible/useful?)

This is what i came up with for now, if you have any suggestion, I'd love to hear it:

(I found the idea of a PCpartPicker List in another thread very useful, so here is mine)

CPU: Intel Xeon E3-1230v5 4x3.4GHz
MoBo: Supermicro X11SSH-TF (DP 10GbE Intel X550 on board)
Memory: 4x Samsung 16GB DDR4-2133 ECC (M393A2G40DB0-CPB)
Storage (Boot): 2x Supermicro SSD-DM032-PHI (mirrored)
Storage (Data): 8x WD Red WD80EFZX 8TB 5400rpm 128MB Cache
Case: LianLi PC-343B (already have it)
HDD Cages: 4x LianLi EX-H34B (already have them)
Power Supply: SeaSonic Platinum 760 (SS-760XP2)
HBA: LSI SAS9200-8e (have it already, in use in desktop with 2 SFF-8088 to SATA cables)
HBA: LSI SAS9260-8i (have it already, atm unused, also have 2 SFF-8087 to SATA cables)

My questions:

• Would this work well with FreeNAS 9 and likely with 10 as soon as it's ready?
• Would you recommend something else anywhere instead to ensure safety of the data (or to avoid any bottlenecks)?
• How would you go with setting up the RAIDz2 over those disks (single RAIDz2 vdev of the new 8TB WDs or anything else)?
• What about adding the now still filled 8TB Seagates, just copy over to the new WDs as soon as they are ready and insert them as new RAIDz2 vdev or is there a good way to use those to expand the original vdev (or do something totally different I didn't come up with yet)?
• Having future expansion in mind, would you first fill up the LSI's ports or use the internal SATA ports of the Motherboard?
• What firmware would I need to flash those LSIs to to make them the best possible match for FreeNAS?

Thanks for all your help!
RobiWahn

 

[Spearfoot]

Hello, @RobiWahn, and welcome to the forums!

Would this work well with FreeNAS 9 and likely with 10 as soon as it's ready?

Yes, I believe you've designed a good system that should provide years of reliable service.

Would you recommend something else anywhere instead to ensure safety of the data (or to avoid any bottlenecks)?

Regarding the dual SATA DOM boot devices... they're nice, of course, but you will be just as well served using a pair of small, inexpensive SSDs.

How would you go with setting up the RAIDz2 over those disks (single RAIDz2 vdev of the new 8TB WDs or anything else)?

I would be comfortable creating an 8-disk RAIDZ2 vdev. You might consider RAIDZ3 if you're paranoid... :)
Regarding the Seagate Archive drives you already own, there are long discussions about them in this thread and also in this thread. My impression is that they are optimized for reads vs. writes and so might serve well as, ahem, archival storage, but wouldn't work very well at all for write-intensive operations.

What about adding the now still filled 8TB Seagates, just copy over to the new WDs as soon as they are ready and insert them as new RAIDz2 vdev or is there a good way to use those to expand the original vdev (or do something totally different I didn't come up with yet)?

You can expand a pool by adding additional vdevs, but for the reason I mentioned above, I wouldn't recommend combining Seagate Archive HDDs with Western Digital Red drives in the same pool. They are too dissimilar.

Having future expansion in mind, would you first fill up the LSI's ports or use the internal SATA ports of the Motherboard?

Eh... it probably doesn't matter, but I personally would use the HBA ports first.

What firmware would I need to flash those LSIs to to make them the best possible match for FreeNAS?

I'm pretty sure FreeNAS will require version P20 of the firmware for your 9200 HBA, which is available at Broadcom's website. I recommend using version 20.00.07.00 - that's what most of us use for LSI 9211/IBM M1015/Dell H200 HBAs.

Good luck!

 

[Ericloewe]

or is expansion of the first RAIDz2 possible

Nope. The structure of RAIDZ vdevs is immutable. Useful? Hell yeah it would be useful, but it's also essentially impossible to implement in ZFS.

Memory: 4x Samsung 16GB DDR4-2133 ECC (M393A2G40DB0-CPB)

You can probably get away with 32GB. If not, the other 32GB are easily added later.

HDD Cages: 4x LianLi EX-H34B (already have them)

Have you tested them? Apparently, one poor fellow ended up with an incorrectly-wired one in his hands. Damned thing destroyed a hard drive and posed quite a fire hazard. And boy, did I hate assembling that thing in my old server, it was painful as hell to attach the SATA cables before placing it in the chassis...

HBA: LSI SAS9200-8e (have it already, in use in desktop with 2 SFF-8088 to SATA cables)

That's good.

HBA: LSI SAS9260-8i (have it already, atm unused, also have 2 SFF-8087 to SATA cables)

That's not, but I hear they sell well on ebay.

Would this work well with FreeNAS 9 and likely with 10 as soon as it's ready?

Yup.

Would you recommend something else anywhere instead to ensure safety of the data (or to avoid any bottlenecks)?

No, looks fine.

How would you go with setting up the RAIDz2 over those disks (single RAIDz2 vdev of the new 8TB WDs or anything else)?

Yeah. Do note that adding a RAIDZ2 vdev with four drives to that may lead to weird performance. It's probably nothing to worry about.

What about adding the now still filled 8TB Seagates, just copy over to the new WDs as soon as they are ready and insert them as new RAIDz2 vdev or is there a good way to use those to expand the original vdev (or do something totally different I didn't come up with yet)?

If you can juggle everything, two vdevs of six drives in RAIDZ2 would quell OCD concerns as well as doubts about mismatched vdevs.

Having future expansion in mind, would you first fill up the LSI's ports or use the internal SATA ports of the Motherboard?

Irrelevant.

What firmware would I need to flash those LSIs to to make them the best possible match for FreeNAS?

P20 IT mode.

 

[RobiWahn]

Yes, I believe you've designed a good system that should provide years of reliable service.

Regarding the dual SATA DOM boot devices... they're nice, of course, but you will be just as well served using a pair of small, inexpensive SSDs.

Thanks, that's good to hear - And those SATA DOMs are also something to try for my own, as I never had one before, so I think I'll just try them now (as they are just a fraction of the cost of the whole system)

I would be comfortable creating an 8-disk RAIDZ2 vdev. You might consider RAIDZ3 if you're paranoid... :)
Regarding the Seagate Archive drives you already own, there are long discussions about them in this thread and also in this thread. My impression is that they are optimized for reads vs. writes and so might serve well as, ahem, archival storage, but wouldn't work very well at all for write-intensive operations.
You can expand a pool by adding additional vdevs, but for the reason I mentioned above, I wouldn't recommend combining Seagate Archive HDDs with Western Digital Red drives in the same pool. They are too dissimilar.

This is interesting, I never thought much of the difference, as I had setup FreeNAS servers at work with just the one or the other type of HDD (SMR or not), but I didn't think of the consequences of mixing those... I guess I'll just make two different pools, with the SMR drives being solely for long term storage with not that much random writes.

I'm pretty sure FreeNAS will require version P20 of the firmware for your 9200 HBA, which is available at Broadcom's website. I recommend using version 20.00.07.00 - that's what most of us use for LSI 9211/IBM M1015/Dell H200 HBAs.

Good luck!

Thanks for the link, I'll make sure to use that version - and I'll see what's needed to be done on that front as soon as FreeNAS 10 is out!

Many thanks for your feedback, it was very valuable for me!

 

[RobiWahn]

Nope. The structure of RAIDZ vdevs is immutable. Useful? Hell yeah it would be useful, but it's also essentially impossible to implement in ZFS.

That's good to know, so I can completely stop thinking in that direction :)

You can probably get away with 32GB. If not, the other 32GB are easily added later.

I have the feeling the prices for DDR4 won't get down that much anymore, and before I forget it until it's needed (and prices might be far up again by then), I thought I better set up the system itself as if it already has all the disks I might want to add in the near/middle future - but good to know that it has a bit of headroom, that would be used as cache anyway, no? Perhaps that could even help writing to the second pool with the SMR drives?

Have you tested them? Apparently, one poor fellow ended up with an incorrectly-wired one in his hands. Damned thing destroyed a hard drive and posed quite a fire hazard. And boy, did I hate assembling that thing in my old server, it was painful as hell to attach the SATA cables before placing it in the chassis...

I have those running in my desktop (in the same case I'll be using for the server then), and yea, they are not that easy to setup the cabling, but all of them worked fine from the beginning, have good connections and their fans are spinning happily as well... got lucky there, I guess!

That's good.

That's not, but I hear they sell well on ebay.

Would you mind detailing a bit what the difference of those is that makes one of them worse than the other? The 9260-8i was a gift from friends and I was happy to use it in the upcoming server to be able to switch from the 9200-8e's "external SFF-8088 to internal SATA" cables (set up like the old USB3 back plates, where you span an external port back through a slot have it connect internally again) to easy internal "SFF-8087 to SATA" cabling of the 9260-8i with no part of that being outside of the case... Even if that optical nuisance would only be gone until I needed to add more disks than the first one would be able to handle on its own and I had to make use of the 9200-8e again... But I'd love to learn the differences, as for my train of thoughts it was just a "-e" to "-i" change up to now, I just thought they are the same ballpark, both being 92xx parts...

Yeah. Do note that adding a RAIDZ2 vdev with four drives to that may lead to weird performance. It's probably nothing to worry about.

If you can juggle everything, two vdevs of six drives in RAIDZ2 would quell OCD concerns as well as doubts about mismatched vdevs.

I guess the best thing would still be different pools for those totally different disk architectures? Or would you think it'd be the "lesser evil" to make two 6-device vdevs even with those SMR/non-SMR mixed? Would any of those options have any real impact on the transfer rate of 10GbE?

Many thanks for your insights, I love learning!

 

[Spearfoot]

Would you mind detailing a bit what the difference of those is that makes one of them worse than the other?

The LSI 9260 board isn't a good choice for FreeNAS because it can't be flashed to IT mode, where the drives connected to it are passed directly to FreeNAS. Instead, it is strictly a RAID controller. The LSI 9211-8i card and it's OEM clones (IBM M1015, Dell H200) are very popular for ZFS users, because they support the desired IT mode.

 

[RobiWahn]

The LSI 9260 board isn't a good choice for FreeNAS because it can't be flashed to IT mode, where the drives connected to it are passed directly to FreeNAS. Instead, it is strictly a RAID controller. The LSI 9211-8i card and it's OEM clones (IBM M1015, Dell H200) are very popular for ZFS users, because they support the desired IT mode.

Ah, that makes sense (except for LSI and its decision to not have IT mode firmware for it)... And there was I, hoping I could just flash your linked firmware to both of them :) Okay, that controller unfortunately won't help with my FreeNAS build then...
Thanks for your really quick response, i guess I'll use the internal SATA ports for the SMR drives then, at least there are enough of those to use as well ;)

 

[SweetAndLow]

After reading that thread about the smr drives I don't think I would ever put them in my zfs server. The performance is just way too slow. There is nothing dangerous about adding a second pool with them so you can try if you want. Just make darn sure you don't expand your original pool when you add them.

Sent from my Nexus 5X using Tapatalk

 

[RobiWahn]

After reading that thread about the smr drives I don't think I would ever put them in my zfs server. The performance is just way too slow. There is no nothing dangerous about adding a second pool with them so you can try if you want. Just make darn sure you don't expand your original pool when you add them.

I have built two FreeNAS servers at work with those 8TB SMR drives (no mix, SMR only), and even if those are just getting big video files or backups, respectively, streamed to them via rsync (only one via 10GbE though, the other has 1GbE), their performance is nothing i'd like to miss... and i didn't notice any other hiccup with any of those servers' disks either, they're performing like other FreeNASes I setup there - the only thing I can't say much about is the random write performance, as those two servers don't really experience that ;)
OTOH, as I wrote in the first post, I didn't have time to ask here for those builds, and want to do it right this time, so I won't add them to the pool without SMR drives - That's a thing I know now :)
Oh, and thanks for your input!

 

[MatthewSteinhoff]

After reading that thread about the smr drives I don't think I would ever put them in my zfs server.

SMRs are the midground between disk and tape. We looked at purchasing them for rarely-accessed long-term storage. In the end, the cost savings wasn't worth the uncertainty.

With the 8TB SMR at $224 and the 8TB WD Red at $303, we felt it was worth an extra $79 a drive to have a proven product with much better performance.

Cheers,
Matt

 

[RobiWahn]

SMRs are the midground between disk and tape. We looked at purchasing them for rarely-accessed long-term storage. In the end, the cost savings wasn't worth the uncertainty.

With the 8TB SMR at $224 and the 8TB WD Red at $303, we felt it was worth an extra $79 a drive to have a proven product with much better performance.

Cheers,
Matt

I think so too (for my own build, and that's why I chose the WD Red as well), but a) had to be cheap at work (not really a budget but storage was needed, and we have a few more than 8 there) and b) after having seen they work well there now have four of them at home, which I don't really want to leave in my desktop, as soon as I have that neat file server setup - but now I know to put those into a pool of their own, that already helps a lot :)