Configuration Questions: Server Specs, SATA Controller and # of Drives

[HeloJunkie]

Hello Everyone -

First let me say that this forum is a fantastic wealth of information! I have spent a month or so reading hundreds of posts.

I have been running freenas for a while now but on pretty small systems. Mostly Dell 2950s with four to six drives. Basically tinkering around with it, watching and reading the forums, trying new stuff (iscsi). I like it a lot and it has been very solid, so I decided to take the next step and actually build a production system.

That opened a whole can of worms that I am hoping I can get some help with before moving forward. Most of my (perceived) issues revolve around conflicting information I am reading about freenas and how to set it up and configure it. Because I eventually want to use freenas in a production environment, I would like to get some good, clear understanding of a couple of key point (for me).

I have several questions, but let me start by first sharing my current hardware configuration:

Supermicro Superserver 6028R-TRT
1 x X10DRi-T Supermicro Motherboard
1 x Intel Xeon E5-2650 V3 LGA2011-3 Haswell 10 Core 2.3GHz 25MB 5.0GT/s
4 x 16GB PC4-17000 DDR4 1233Mhz Registered ECC Dual-Ranked 1.2V Memory
9 x 4TB Western Digital WD40EFRX Red NAS SATA Hard Drives
Dual 740 Watt Platinum Power Supplies
Dual APC 1500 UPSs (one for each power supply)
8GB USB Thumb Drive for booting

The X10DRi-T motherboard has two Intel X540 based 10Gb Ethernet adaptors along with an Intel C612 Express chipset and 10 SATA3 ports.

The system will be used for several things:
1) Time Machine for five laptops and two desktops
2) Several iSCSI connections to Windows 2012 Servers for a software application that will not permit a share
3) CIFS and AFS Shares for ten or so people (NFS is doable as opposed to CIFS if the performance gain is worth using it)
4) Some transcoding via Plex for training videos for classrooms (four streams max)
5) Backup for another freenas system which is offsite (2.5TB) (preseeded)


Initially after doing a bunch of reading, I was going to install 8 x 4TB drives and do a RAID10 mirror, but after reading more and more about RaidZ3, I decided that would be the best way to go given space and performance. None of my required uses really seems to meet the needs of the RAID10 performance and the RaidZ3 seemed to give me the most space and performance for what I was trying to do. Of course, this was based on the reading that I did. I thought the optimal was an even number of drives plus parity. In my case, six drives and three parity drives.

So with the RaidZ3 decision firmly behind me, I ordered a 9th 4TB hard drive, installed it in the server and promptly was told that a 9 drive RaidZ3 configuration was sub-optimal. So I went back to my bookmarks and read and re-read the following:

Solaris Internals RAIDz Recommendations
RAIDZ Configuration Requirements and Recommendations
A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.

• Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
• Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
• Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
• (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
• The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups.

The emphasis is mine, but hopefully you get the point where I got the whole "9 drive" thing. So after I got the sub-optimal message I started reading more and more about the drive requirements. While I never got a complete picture of the requirements (there appears to be different schools of thought) it appears that the requirements has to do with how the data is physically written to the drives and in what size chunks they are written. That seems to make sense.

Then I was reading somewhere about compression making that a non issue since with compression freenas could alter the size of the data to meet the necessary drive requirements. What I was never able to locate was a definitive explanation as to what would happen if I choose to move forward with my sub-optimal configuration. It was at this point that I decided that I needed to ask questions and get answers before trying to put something into production that I did not fully understand, and why I am here now.

So that is really my first question or series of questions:

What happens if I go forward with a 9 drive RaidZ3 configuration? Do I risk losing data, losing space, losing performance, all three? Will this be noticeable to me and my staff with so little traffic going to this NAS or is it considered a hard line in the sand you do not ever want to cross.



What is the "actual" optimal configuration for number of hard drives and where is the definitive Freenas guide on this topic? Do I follow the Solaris post, or go with the latest Freenas manual which says:



When determining how many disks to use in a RAIDZ, the following configurations provide optimal
performance. Array sizes beyond 12 disks are not recommended.
• Start a RAIDZ1 at at 3, 5, or 9 disks.
• Start a RAIDZ2 at 4, 6, or 10 disks.
• Start a RAIDZ3 at 5, 7, or 11 disks.
The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple
groups.

OK, so here I am with two conflicting (at least to me) drive recommendations. So I am still left wondering if my 9 drive configuration will work with RaidZ3, why is it sub-optimal and what actual performance hits I am going to take by using this instead of maybe a RaidZ2 and make my new 9th drive a spare!

So my research into the drive question led me to the discussion about on-board SATA controllers vs. something like the IBM ServerRAID M1015 and an expander (like the Intel RES2SV240). My onboard controller is the Intel C612 which seems to be a good chipset from what I have read. I cannot find enough information to make a good judgement call about on-board vs. PCI SATA controllers. I have put enough energy and effort into this project that a few hundred more dollars for a dedicated SATA card is worth it if, in fact, they will perform better than my on-board controller.



To recap, the questions I have relate to the specific number of drives for RaidZ3, why do the Solaris folks and the freenas folks differ on this number and which one is correct?

What sort of penalties am I facing if I do not use the "optimal" drive configuration for RaidZ3? Is this a 5% performance hit and a 7.265% hard drive space hit or....??

My other question related to the use of on-board SATA controller (Intel C612 on the Supermicro X10DRi-T vs the IBM ServerRAID M1015 and an expander. Which one will provide the best overall performance?

Thank You

 

[zambanini]

raidz3... and big disks, you must have balls....

even raidz1 kills your io speed, this will give you maximal penaltiy on every write. also think about how much time a resilver process will need, really...think about this. you have also a good chance to kill the other disks within that time ( much more load on them)

do yourselfe a favor, fo for mirrored pools with this disks. you can also do a 3 way mirror ( one data disk is mirrored to two addiotnal disks )

or at least multiple raid z1 in a stripe.

the optimal disks analytics maybe answered frome others

 

[Ericloewe]

I've never seen any hard logic behind the 9-drive RAIDZ3 recommendation. The 5, 7, 11 drive RAIDZ3 recommendation is meant to align ZFS blocks (my terminology on this low-level ZFS stuff may be off) with HDD sectors, which are powers of 2. This recommendation is said to be of less relevance when using compression, as the ZFS blocks are now of varying sizes instead of a nice uniform power of 2.

There is also the theory that non-aligned blocks lead to loss of space, besides the obvious performance implications.

To be completely honest, there's a lack of empirical evidence (widely circulated around here, at least) supporting any postulate regarding vdev sizing (besides the "do not make them excessively wide" rule - Cyberjock proved that one with his insanely wide vdev). I personally stuck with a 6-drive RAIDZ2, because I had nothing to lose from following the recommendation and the resulting size was appropriate for my workload and my hardware.

As for controllers, the Intel PCH SATA controllers are fine. Probably the best SATA (not SAS) controllers you can get. Performance differences are most likely trivial to nonexistent, if Intel is being transparent. However, there's talk the last four ports on the PCH may be obtained via port multipliers internally or that they're otherwise shady. There is no real data on this yet.
LSI SAS 2008/2308 controllers and LSI SAS expanders (including the Intel RES2SV240) are certainly a safe, tested bet. Whether it's actually faster is open to debate.

 

[HeloJunkie]

Thank you both for your replies.

Zambanini - so raidz3 and large hard drives are not the way to go based on how long the resilver process is going to take after a drive failure? In looking, I have found that people seem to think just the opposite, that with large drives raidz3 is the way to go, just not too many of them (~19). In fact one article specifically pointed folks with large hard drives to RaidZ3:

4TB Hard Drives & RAID6

Is this the only reason that you say to stay away from large hard drives? What is the cutoff limit for RaidZ3 on hard drive size? Are any of the ZFS RaidZ levels able to be used with large hard drives or is the ZFS RaidZ system going to die out as a result of larger and larger hard drives? 6 and 8TB hard drives are on the market and their prices are already starting to come down, in fact I really toyed with the possibility of using 6TB drives as opposed to 4TB drives, but 4TB was the real sweet spot at $190 per WD Red drive. I also looked around trying to figure out exactly how long a resilver would take on a 4TB drive under RaidZ3 (trying to think about it as you suggested) and I cannot seem to find a good answer. I appreciate the input and would love to know what I am up against before putting this system into production.

I guess one option would be to throw the data on the box, pull a drive as if it totally failed and then replace it with a "new" drive and test the resilver time.

I am concerned that Freenas may not be up to my task if my only option with large hard drives is RAID10 and a 50% loss of usable space. Why even have RaidZ if it is not safe to use it with large drives?


Ericloewe - Thank you as well. I see that freenas has a built in performance test. Is this the best way to determine how my system will perform and IF Intel is doing something shady with their 10 SATA ports on the motherboard?

Thanks to both of you!

 

[Mlovelace]

A single raidz3 vdev is fine depending on the work load; you'll want to test the pool to ensure it meets your needs. If you need higher IO then you might consider multiple striped vdevs. There are very experienced people here that have raidz3 vdevs with 6tb drives without issue. Drive size might be a concern for single parity vdevs during resilvering (raidz1) but raidz3 resolves that issue.

If you are planning on running 10gbe with the on-board Intel X540s there is an issue currently in freenas. I will refer you to the bug report https://bugs.freenas.org/issues/4560#change-23492 It's scheduled to be fixed in the 9.3 release, but you might run into problems untill then.

 

[cyberjock]

I am concerned that Freenas may not be up to my task if my only option with large hard drives is RAID10 and a 50% loss of usable space. Why even have RaidZ if it is not safe to use it with large drives?

Two things:

1. FreeBSD's basic philosophy is to be as powerful as possible and let the admin decide what is and isn't appropriate. It's not considered anyone's place to decide what technology should or shouldn't be allowed. Generally speaking, there are *always* those weird exceptions that some "bad" technology is useful.
2. Plenty of people use RAIDZ1 for quick simple storage for things like temp files and stuff. If a disk fails they replace it. If the pool fails then they just build a new pool. Since it stores temp files it shouldn't matter aside from the downtime while recreating the pool. (See.. I just made an exception like I said in #1)

The basic fact is that FreeBSD(and FreeNAS) gives you a whole host of tools to get work done. Whether you chose to use them properly, or use them improperly, is up to you. The onus is on *you*, the administrator of your server, to not do things that might result in what we might call "an unpleasant situation". It's like the age old motto "with great power comes great responsibility". This isn't Windows and the OS won't protect you from doing stupid things. If someone is inadequately prepared to use the OS and they lose their data we're going to point them to the applicable section of the FreeNAS manual, the stickies, etc and say 'see.. the info was here.. don't tell us we didn't warn you'. In fact, I make this particular philosophy perfectly clear in my noobie guide.. slide 7 I believe.

Again, your data, your responsibility to get it right. Nobody will do hand-holding. If this kind of responsibility scares (and it should at least make you think twice before doing this with your company's data) you should give iXsystems a call. They will gladly sell you a system they've designed so you won't make those fool-hearty mistakes. As a general rule, if you fubar something badly you won't know what you fubared until you can't get access to your data. And at that point it's almost always too late for anyone to do anything.

 

[HeloJunkie]

Cyberjock -

Thanks. I have read so many of your posts it is scary!

That is why I am here asking questions. I do have the manual and I won't put the system into production until I understand the strengths and weaknesses and make a determination from there. I read your powerpoint before putting my first test system into place. In fact, your slide deck saved me money as I was going to get mirrored SSDs for ZIL but we do ZERO NFS right now, so after reading your slide on ZIL, I opted not to install the SSD drives.

The questions I am asking here, I just could not find (what I considered) definitive answers. A lot of opinions, but not a lot of answers. And as your post above points out, maybe I never will, being relegated to simple trial-and-error to get the answers to my questions.

I am not new to Linux, but I am new to FreeBSD although they seem very, very familiar so I have a learning curve there as well. I am not opposed to testing and trying (what I am doing now) with various configurations and running some performance tests (albeit the one built into freenas) to see the actual performance differences for myself. (RaidZ3 with 7 drives vs RaidZ3 with 8 or 9 drives, RaidZ3 vs RAID10, etc). I also very clearly understand things like item #3 on your slide #28 and item #2 on slide #29. Please do not mistake my questions for lack of desire to do the research on my own.


I think I understand the sub-optimal configuration being related to how the data gets written to the drives, but I also thought I understood that compression pretty much negated that issue. I read your stuff and a bunch of Solaris stuff and other posts and just when I think I have it figured out I see:

raidz3... and big disks, you must have balls....

followed immediately by this:

There are very experienced people here that have raidz3 vdevs with 6tb drives without issue.

Maybe you can understand my desire for more information and more answers!


As far as iXsystems, I have been in constant contact with the good folks over there for a quote for another project and have even asked them directly some of the same questions, and I got quoted a "best practice" but no meat and potatoes as to the WHY you don't want to do things that way.

Optimal VDev Configuration: With FreeNAS and ZFS, the concept of a logical disk, traditionally referred to as a “LUN” in hardware RAID terminology, is instead known as a Virtual Device (VDev). The best practice for configuring your volume is to use VDevs configured with the metric of powers of two, plus the parity level. For the most optimal configuration, you will want such VDevs to be installed in pairs. Paired VDevs allow information to stripe evenly across each set, improving overall performance. For example, if you have decided that using five drives per VDev is appropriate for your RAID-Z volume, you would want ten drives in two such VDevs to get the most from this configuration.

Optimal VDev size and VDev pairs are not a requirement, but is a general best practice that will help keep your FreeNAS optimized for best performance at the volume level. Knowing the size of your VDevs also makes planning for future expansion simple. Once you have established an appropriate number of disks for your VDevs, you can easily expand space by adding additional paired VDevs to extend your volume. ZFS performance begins to significantly decline as a pool approaches 90% of it’s capacity. As you are approaching this percentage of utilization, you will approach your sales agent to explore your future expansion options.

As an interesting note, in my discussions with them, they address everything BUT RAID-Z3, their discussion ended at RAID-Z2!

So armed with all of this ambiguity, I did what any sane sysadmin would do and decided to ask others that may have already done it. I believe I would like to run RAID-Z3, I think it will do what I need and what I want and will provide my required level of data security, but I want to make certain I understand what it entails (good and bad) and getting answers to any lingering questions before becoming one of those guys that posts a message: Lost Data, please help me recover it!

This forum has been extremely helpful, your posts as well as others have been insightful into Freenas and I am convinced that it is the proper tool to help me replace a bunch of aging ReadyNas 1100s.

Thank You

 

[Ericloewe]

One thing is certain: You won't endanger data by going with non-optimal vdev sizing. All penalties can be overcome by spending more money in the future (doesn't sound too good, but I think you understand what I mean).

Honestly, I'd recommend you experiment a bit with the various options during testing. That way you'll answer your questions and give us an additional data point :D.

 

[cyberjock]

The reality is that there's so many factors with different vdev sizes that in the big scheme it's a "who cares".

You can do so many things right and still be unhappy at the end, or you can do everything wrong and still be happy. The threshold for "happy" and "unhappy" is completely arbitrary based on the user, and how much a given configuration will suffer compared to an optimal configuration depends on your usage, the usage history of your pool, and other factors that are going to be unique to your one server and only yours. So data points from my server mean absolutely *nothing* when trying to plan ahead.