Do you really need to buy a hardware raid controllers when using zfs?

[Mike77]

Been reading on the forums trying to decide if I should buy a raid controller for my freenas machine.

Now i've found a couple of posts where it is mentioned that it might not be very smart to setup a hardware raid in freenas, because freenas want's control of the disks and the raid. So now i'm wondering if it has any use to buy the raid controller, and if so, which one and what should i look at?

At the moment i'm using a couple (2x 2TB) of sata drives (zfs) that are connected to the mainboard. But with this setup the read/write speeds are very slow (40Mb/s). So i'm trying to speed up things.

 

[Glorious1]

I think most people get such a card only to provide more device ports, and turn off the RAID part of it. At least that's my understanding. I just ordered this one which seems to be highly recommended (at least I hope that's the right one!?) and will flash it to IT mode, which I understand just presents the disks to FreeNAS. Lots of instructions on how to do that and they're all different, so I haven't crossed that bridge yet:
https://forums.freenas.org/index.php?threads/my-socket-2011v3-build.24842/#post-155960
http://vwilmo.wordpress.com/2014/06...l-checking-m1015-queue-depthflashing-it-mode/
https://forums.servethehome.com/index.php?threads/ibm-m1015-firmware-what-to-flash-with.97/
http://www.servethehome.com/ibm-serveraid-m1015-part-4/
http://www.abisen.com/lsi2308-with-solaris11.1.html

Then you get some cables like this to fan out to the drives:
http://www.monoprice.com/Category?c_id=102&cp_id=10254&cs_id=1025406

 

[jgreco]

No, you do NOT want a RAID controller, you want an HBA - a Host Bus Adapter. Often these will also include simple "RAID" functionality and units like the LSI 9240 are classified as an "entry level" RAID controller, but can be flashed to a pure HBA mode. There's a lot of information in the LSI controller thread. LSI makes about 90% of all the HBA's out there and about 100% of the ones you want to use with FreeNAS.

If you do not need the extra ports that an HBA would give you, the motherboard SATA ports on any server-grade motherboard are almost certainly fine. There are a few possible exceptions such as the Marvell chipset ports on the ASRock Avoton boards. You can mix and match HBA and mainboard SATA ports as well.

Your performance problem is likely to be something else, but you haven't followed the forum rules and posted details about your setup. We can't help you until you do.

 

[Mike77]

No, you do NOT want a RAID controller, you want an HBA - a Host Bus Adapter. Often these will also include simple "RAID" functionality and units like the LSI 9240 are classified as an "entry level" RAID controller, but can be flashed to a pure HBA mode. There's a lot of information in the LSI controller thread. LSI makes about 90% of all the HBA's out there and about 100% of the ones you want to use with FreeNAS.

If you do not need the extra ports that an HBA would give you, the motherboard SATA ports on any server-grade motherboard are almost certainly fine. There are a few possible exceptions such as the Marvell chipset ports on the ASRock Avoton boards. You can mix and match HBA and mainboard SATA ports as well.

Your performance problem is likely to be something else, but you haven't followed the forum rules and posted details about your setup. We can't help you until you do.

Sorry man.

It's FreeNAS-9.2.1.8-RELEASE-x64 (e625626) (on a thumbdrive)
Mainboard: ASUS P8Z77-V LX
Processor: Intel Core i3-2100 CPU @ 3.10GHz
Ram: 4x 2GB DDR3 Dual
Disks:; 2x 2TB Seagate Baracuda (ZFS Striped)

At the moment this is used as a testing machine, to see how Raid controllers and different Raid setups work.

 

[Ericloewe]

I could swear a thread with the exact same title showed up about two weeks ago...

 

[Mike77]

I could swear a thread with the exact same title showed up about two weeks ago...

Where? Do you have a link?

 

[Mike77]

No, you do NOT want a RAID controller, you want an HBA - a Host Bus Adapter. Often these will also include simple "RAID" functionality and units like the LSI 9240 are classified as an "entry level" RAID controller, but can be flashed to a pure HBA mode. There's a lot of information in the LSI controller thread. LSI makes about 90% of all the HBA's out there and about 100% of the ones you want to use with FreeNAS.

If you do not need the extra ports that an HBA would give you, the motherboard SATA ports on any server-grade motherboard are almost certainly fine. There are a few possible exceptions such as the Marvell chipset ports on the ASRock Avoton boards. You can mix and match HBA and mainboard SATA ports as well

Thanks I just found your post (Confused about that LSI card? Join the crowd). It's clear that with FreeNas you don't need the Raid controller.. But seeing that the system is slow, and I will probably need more sata ports, the questions is, does a HBA do anything for the speed and is there an HBA that just works without having to flash the firmware.....

And another question is, does a ZFS machine outperform a Raid controller (and a different OS) ?

 

[jgreco]

Your system is probably slow because you have an ASUS consumer grade board with a Realtek ethernet. It is likely that one of the hamsters that moves bits in your Realtek chip has died... they're pretty lazy even when healthy.

There are good threads on picking appropriate hardware for FreeNAS in the hardware forum. Putting an HBA on a poor motherboard won't get you more speed if it is the network that's busted. You should also be using ECC memory, because ZFS depends on ECC for part of its pool integrity strategy. You may be able to find extra SATA ports that work but around here we like HBA's because they're basically enterprise-grade disk controllers and it is extremely unusual for people to have problems with them.

A ZFS machine might or might not outperform a RAID controller. This is a trick question because in the ZFS model the host system itself is acting as the RAID controller. If you put a hardware RAID controller in your existing machine, that is essentially no load on the machine itself and the RAID controller is doing the work. With ZFS, your CPU and host resources are doing the work. So a hardware RAID controller will often beat ZFS if you just compare them directly. The thing is, though, you can throw massive resources at a FreeNAS server - lots of RAM, L2ARC, more disks than a hardware RAID controller can handle ... and it can easily be competitive with a six-figure SAN RAID, for a tenth of the cost. So you have to construct a more meaningful question.

You can certainly put a nice LSI 9361-8i ($600) plus a CacheVault ($200) and CacheCade Pro 2.0 license ($250) along with an SSD ($200) so for about $1250 you get a killer single host RAID subsystem that'll be hard to beat. The thing is that we kind of like to invest those dollars in extra resources for the ZFS system, and you get a network-capable device out of it all as well.

 

[Mike77]

Your system is probably slow because you have an ASUS consumer grade board with a Realtek ethernet. It is likely that one of the hamsters that moves bits in your Realtek chip has died... they're pretty lazy even when healthy.

There are good threads on picking appropriate hardware for FreeNAS in the hardware forum. Putting an HBA on a poor motherboard won't get you more speed if it is the network that's busted. You should also be using ECC memory, because ZFS depends on ECC for part of its pool integrity strategy. You may be able to find extra SATA ports that work but around here we like HBA's because they're basically enterprise-grade disk controllers and it is extremely unusual for people to have problems with them.

A ZFS machine might or might not outperform a RAID controller. This is a trick question because in the ZFS model the host system itself is acting as the RAID controller. If you put a hardware RAID controller in your existing machine, that is essentially no load on the machine itself and the RAID controller is doing the work. With ZFS, your CPU and host resources are doing the work. So a hardware RAID controller will often beat ZFS if you just compare them directly. The thing is, though, you can throw massive resources at a FreeNAS server - lots of RAM, L2ARC, more disks than a hardware RAID controller can handle ... and it can easily be competitive with a six-figure SAN RAID, for a tenth of the cost. So you have to construct a more meaningful question.

You can certainly put a nice LSI 9361-8i ($600) plus a CacheVault ($200) and CacheCade Pro 2.0 license ($250) along with an SSD ($200) so for about $1250 you get a killer single host RAID subsystem that'll be hard to beat. The thing is that we kind of like to invest those dollars in extra resources for the ZFS system, and you get a network-capable device out of it all as well.

Hahaha. Yes I do have a consumer grade board with a Realtek port. Even worse it also has an extra PCI realtek port... But haven't found the hamsters yet. Maybe they're on a vacation. ;-)

That ECC is needed is clear to me, but I believe this board doesnt support it.. Futhermore I wasn't aware that with FreeNas is doens't really matter which ports u use (Which is a great feature). This is because I was actually going for a good Raid controller and was trying to find the parts to build te machine around it. So is guess the ting to do is to find a mainboard with enough ports (i'll need 8) and maybe with a dual nic and a 1155 socket. This way I'll probably won't need the HBA, and can keep using the I3. If you have any sugestions for a mainboard that woudl be great. Finding the right board is a real problem nowadays. There are way to many around.

I just found a Supermicro X10SAE, that seems to tick most boxes. But I'm not sure that it backwards compatible with the current I3 and dimms (I know i'll have to replace them). And then add two Kingston Reg ECC DDR3L-1600 8GB SR x4. Is this a good way to go?

 

[danb35]

The SuperMicro X10 series are Socket 1150 boards, and thus won't work with a Socket 1155 chip. Of the X10 boards, the X10SL7-F seems to be the favorite, with 6 SATA + 8 SAS ports. For a Socket 1155 chip, you'd be looking at an X9 series board. I don't think any of them come with more than 6 SATA ports onboard. The X9 server boards require ECC and won't POST without it.

 

[Mike77]

The SuperMicro X10 series are Socket 1150 boards, and thus won't work with a Socket 1155 chip. Of the X10 boards, the X10SL7-F seems to be the favorite, with 6 SATA + 8 SAS ports. For a Socket 1155 chip, you'd be looking at an X9 series board. I don't think any of them come with more than 6 SATA ports onboard. The X9 server boards require ECC and won't POST without it.

Thanks.

I'v looked at the SL7-F. I believe the main difference is the extra sas ports. But I think that the 8 sata ports will be enough.
Are the two Kingston Reg ECC DDR3L-1600 8GB SR x4 the right memory for the x10 motherboard?
To bad that it´s an 1150. This way I won´t be able to use my old I3. Any tips for a good processor? a 4 gen xeon maybe? It doens't have to be the fastest and most expensive.

@jgreco : no hamsters or chipmunks in this system?

 

[jgreco]

Kingston Reg ECC DDR3L-1600 8GB SR x4

No, you don't want registered memory. The Kingston part is KVR16LE11/8HB but be warned that there's been some trouble with Kingston memory in recent (X10) times, search the forum for Elpida or Kingston.

To bad that it´s an 1150. This way I won´t be able to use my old I3.

You can certainly obtain something like an X9SCM which lacks the massive disk attachment capability but is - I think - compatible with your existing CPU, but you don't need the built-in Intel graphics on the CPU and I'm not sure that the i3-2100 supports ECC, so you might be best off repurposing the existing CPU/MB combo.

@jgreco : no hamsters or chipmunks in this system?

I can't think of the last time someone complained about the throughput of an Intel network card. By the way, putting an Intel network card into your existing ASUS is also an option. The Intel PCIe desktop cards (~$35) are fine.

 

[Ericloewe]

Thanks.

I'v looked at the SL7-F. I believe the main difference is the extra sas ports. But I think that the 8 sata ports will be enough.
Are the two Kingston Reg ECC DDR3L-1600 8GB SR x4 the right memory for the x10 motherboard?
To bad that it´s an 1150. This way I won´t be able to use my old I3. Any tips for a good processor? a 4 gen xeon maybe? It doens't have to be the fastest and most expensive.

@jgreco : no hamsters or chipmunks in this system?

There's a Supermicro X10 memory guide. It's linked in my sig.

Regarding processor, the four common options are: i3 4130, i3 4330, the i5-equivalent Xeon (E3-1220 v3?) and the i7-equivalent Xeon (E3-1230 v3?)

 

[Mike77]

No, you don't want registered memory. The Kingston part is KVR16LE11/8HB but be warned that there's been some trouble with Kingston memory in recent (X10) times, search the forum for Elpida or Kingston.

Thanks for informing me! I just read the post by EricLoewe. Better not take that risk.

You can certainly obtain something like an X9SCM which lacks the massive disk attachment capability but is - I think - compatible with your existing CPU, but you don't need the built-in Intel graphics on the CPU and I'm not sure that the i3-2100 supports ECC, so you might be best off repurposing the existing CPU/MB combo.

To bad I wont be able to re-use the parts that I already have. This is going to cost a lot more then I had budgeted. I'll probably need something like a X10 board, because of the ports, and I might want to ad a SSD ZIL and L2ARC in the future. I just read a couple of posts regardring the X10 where there is mentioned that it might be necessary to flas the board with LSI "fimware"? Will this be needed?

I can't think of the last time someone complained about the throughput of an Intel network card. By the way, putting an Intel network card into your existing ASUS is also an option. The Intel PCIe desktop cards (~$35) are fine.

I''m not sure if the problem is in the network and/or the memory. Is there a way I can test this, and is there a way I can calculate the speeds that should be reached? I just did a couple of parkdale (found it on the internet. Don't know if this is a good/trust worthy progamme) benchmarks of the read/write speeds of the NAS drives (the striped zfs drive (2x 2TB baracuda)).
Settings1:
Files size: 1000 MByte
Block size: 1 MByte

Results:
Seq. Write Speed: 89.2 MByte/sec
Random QD32: 8.7 Mbyte/sec, 2231.6 IOPS, 0.45 ms
Seq. Read Speed: 49.6 MByte/sec
Random QD32: 9.3 Mbyte/sec, 2285.8 IOPS, 0.42 ms

Settings2:
Files size: 1000 MByte
Block size: 64 kByte

Results:
Seq. Write Speed: 52.4 MByte/sec
Random QD32: 8.7 Mbyte/sec, 2231.7 IOPS, 0.45 ms
Seq. Read Speed: 38.1 MByte/sec
Random QD32: 9.3 Mbyte/sec, 2406.9 IOPS, 0.42 ms

Are these normal speeds acceptable, or should I be worried that there is something really wrong here?

 

[jgreco]

See the LSI sticky in the hardware forum. The LSI controllers were designed to be both HBA ("just SAS ports") and entry level RAID controllers but for FreeNAS you are much better off flashing the version of the firmware that has them working only as HBA. It's a bit faster and more foolproof.

Those read speeds seem way low. I wouldn't expect a memory problem per se, because the memory is immensely faster than the network. The usual problem when a Realtek is present is the Realtek. You can easily see whether or not this theory is correct by using iperf to test your network and some dd tests on your pool. Note that both tests will show substantially better speeds for each individual subsystem - with a NAS, you're layering complexity and any performance problems get amplified as complexity increases.

 

[Mike77]

@jgreco : Q: Because I´ve been wrong so far, I really need to aks. Is the Intell E1g42ET dual NIC a good one, or am I wrong again? :-S

 

[jgreco]

It's good if you can find it inexpensively. It's based on the 82576 so it uses the igb driver, which is fine. However, it isn't going to be substantially better than something like the Intel CT desktop card, so don't go paying a ton for it.

 

[Mike77]

It's € 170. Kinda expensive for the moment. But I just found a Intel X520 DA2 10-gigabit PCIe x8 - 10 GigE - dual port card for € 240. Thats cheap for a dual port 10Gb card right? But just bought 2 intel CT nics. Guess I'll need one in two computers to test it. Thanks again for all your help!!

 

[Mike77]

@jgreco I've installed the two intel nics and the the test speed has increased to about 100 Mb/s read and write. IOPS still at aout 9. So it seems to have helped. Sadly the real world speed is still about 20 Mb/s. So I''ll still will have to do something about that and also try and find something fun to do with the two not used NIC ports on the Free NAS.

Bus first it's time for the hollidays! So thank for all your help an have a mery christmas and a happy new year!!!

 

[jgreco]

I hope you mean "100 MByte/sec"? "Mb/s" is a horribly vague badness, does it mean "megabit/second" (what most professionals would read it to mean) or is it a bastard mistype of "MB/s" (megabyte/sec). Byte is capital-B, bit is lowercase-b. Best to spell it out.

So if the speeds are now hitting 100 MByte/sec for test traffic that's sequential, that's not awesome but at least reasonable. IOPS testing is basically meaningless since every tool measures IOPS in a different manner.

Real world speeds are going to be dominated by various factors. Write speeds will be impacted largely by things like the number of disks and the fullness of the pool. Read speeds will be impacted by the number of disks, the fragmentation on the pool and the nature of the data being read (small random files are harder than long sequential unfragmented data).

You can still take some time to validate the individual subsystems with iperf and dd testing to help eliminate unexpected bottlenecks.