Get an ServeRaid M1015 or not?

[crisman]

Hi,

I currently have an Asus E45M1-M Pro board with 8Gb Ram, 5 HD's WD 1 Tb Green in Raid-Z, I currently use for backups and later on maybe video streaming, since I plan to use CIFS protocol does it worth to get an ServeRaid M1015 controller?
Will I get better performance? any other recommendations please?

Thanks.

 

[Joshua Parker Ruehlig]

Hi,

I currently have an Asus E45M1-M Pro board with 8Gb Ram, 5 HD's WD 1 Tb Green in Raid-Z, I currently use for backups and later on maybe video streaming, since I plan to use CIFS protocol does it worth to get an ServeRaid M1015 controller?
Will I get better performance? any other recommendations please?

Thanks.

The below link has sopme great information about choosing a raid card for use in non-raid mode.
http://blog.zorinaq.com/?e=10

Looks like a good card, based on the LSI SAS2008 which would be perfect for the Asus board you (and I have). It supports PCIe 2.0 and our motherboard has PCIe 2.0 x4 so you'd be getting enough bandwidth if you hooked 8 harddrives to it.
I ended up going with a RocketRAID 2720sgl which is based on the Marvell 88SE9485, essentially same specs as yours but I found it for $80 on ebay used.

You may need to be on FreeNAS8.3 to have this card recognized out of the box, and remember to flash non-raid bios to it if it needs one. Also you can pick up the miniSAS to 4xsata adapers for $1 on ebay.
http://www.ebay.com/itm/MiniSAS-Min...t=LH_DefaultDomain_2&var=&hash=item1e6ed5e821

 

[crisman]

The below link has sopme great information about choosing a raid card for use in non-raid mode.
http://blog.zorinaq.com/?e=10

Looks like a good card, based on the LSI SAS2008 which would be perfect for the Asus board you (and I have). It supports PCIe 2.0 and our motherboard has PCIe 2.0 x4 so you'd be getting enough bandwidth if you hooked 8 harddrives to it.
I ended up going with a RocketRAID 2720sgl which is based on the Marvell 88SE9485, essentially same specs as yours but I found it for $80 on ebay used.

You may need to be on FreeNAS8.3 to have this card recognized out of the box, and remember to flash non-raid bios to it if it needs one. Also you can pick up the miniSAS to 4xsata adapers for $1 on ebay.
http://www.ebay.com/itm/MiniSAS-Min...t=LH_DefaultDomain_2&var=&hash=item1e6ed5e821

Hi Joshua,

What you think about "our" board? Is it possible to get good read/write performance on CIFS? does the Gigabit Ethernet adapter will saturate? Does the ServeRaid M1015 will give better R/W performance?

Thanks.

 

[Joshua Parker Ruehlig]

I was actually doing some benchmarking yesterday. I used NFS (which may perform better or our dual core board then the single threaded CIFS)
I ran iperf to our realtek NIC and was getting weird spikes, from my ubuntu desktop to my FreeNAS server through an unmanaged gigabit switch. iperf gave 560Mb/s but I noticed it was spiking to 800+ then dropping several times.
I then did it with an Intel PCIe NIC (you can pick one up cheap) and was getting 900+Mb/s on iperf just fine.

I then wrote a 14GB file of zeros on my ubuntu desktop, got 230MB/s on my new Maxtor SSD, read it at 300MB/s. So my desktop shouldn't be the limiting factor (except possibly for it's realtek NIC).

I then moved the file to the Nas using NFS on the realtek IP, got 50-60MB/s but it drops down slowly.
Read it back to my desktop at similar speeds.
Then rewrote it but this time on the intel IP, got 80MB/s at first but it crept down to 50-60MB/s.
I then read the file and was getting full 123MB/s speeds at first, but it then quickly dropped down to 80MB/s.
I also ran top on FreeNAS and it showed low idle percent, like 9% to 2% so I'm guessing the NFS transfer was CPU bound.

TAKEAWAY (might be wrong)
NFS transfers CPU bound on our motherboard, if your ok with 50MB/s then this board is a perfect low power solution. I have 16GB of ram and I believe the ARC cache came into play when I read the file the second time. The e350 also had to work less because it was sending the data over an intel NIC, not the realtek nic. In actual performance the NIC made some difference but is not crucial.

With CIFS, which I believe is single threaded, you may get even worse performance on this low CPU power board. But I don't think CIFS relies as much on the ZIL so you may actually get better performance.

I believe I could increase NFS write speed by adding a dedicated ZIL drive. Thinking of the Intel 313 but can't get myself to spend the money.
I am thinking of swapping the motherboard for a Gigabyte and g530 but don't need the performance increase so I can't get myself to spend the money either. I could also add 2*8GB of ram and possibly get more caching in. The only problem s I believe the g530 would use an additional 10W when idling which is almost $1/month of cost for me.

Not really worth is for the little performance I get, and the only time I 'need' Gigabit speed is when I move a movie over to my desktop for running mkvmerge, and send it back. But I probably will eventually upgrade just cause...

 

[crisman]

TAKEAWAY (might be wrong)
NFS transfers CPU bound on our motherboard, if your ok with 50MB/s then this board is a perfect low power solution. I have 16GB of ram and I believe the ARC cache came into play when I read the file the second time. The e350 also had to work less because it was sending the data over an intel NIC, not the realtek nic. In actual performance the NIC made some difference but is not crucial.

Well I'm not sure if 50MB/s will be enough from streaming some HD movies on my actual Desktop Asus board with Atom D525 board using XBMC, You refer "e350 also had to work hard" but on our board "Asus E45M1-M Pro" the processor is an DC E-450 probably you're confused, wright?

I'll do some testing late this weekend and I'm also waiting to get the ServeRaid M1015 Card to improve the FreeNas performance, but I still have another Asus board "P8H77-I" with an Intel core i3 2100T that I can use in last case and then I expect to get better results on R/W performance for streaming videos, this setup will have more power consumption but I think that using most of the time at weekend the difference won't be so much.

Thanks for your analysis.

 

[Joshua Parker Ruehlig]

Trust me 50MB/s will be fine. That's 400Mb/s, that like multple 4K films uncompressed!

Ohh, my board is e35m1-m Pro, looking at cpubenchmark.net the e-450 has basically the same cpu power as the e-350. I guess all they did was upgrade the GPU, which doesn't help at all with FreeNAS..

 

[crisman]

Trust me 50MB/s will be fine. That's 400Mb/s, that like multple 4K films uncompressed!

Ohh, my board is e35m1-m Pro, looking at cpubenchmark.net the e-450 has basically the same cpu power as the e-350. I guess all they did was upgrade the GPU, which doesn't help at all with FreeNAS..

What kind of setup you have?

I haven't created mine yet, I have 5 Tb WD Green HD, I was supposing to put them in Raid-Z, Do you recommend having L2ARC? Is it necessary having another HD to L2ARC?

Thanks.

 

[Joshua Parker Ruehlig]

What kind of setup you have?

I haven't created mine yet, I have 5 Tb WD Green HD, I was supposing to put them in Raid-Z, Do you recommend having L2ARC? Is it necessary having another HD to L2ARC?

Thanks.

I have 10 Samsung Green 2TB in RAIDZ2. I didn't make the zpool in FreeNAS webui because a raid using whole disks gives the best performance. I also did the 4k trick when creating the zpool. It also doesn't give you swap but I think my 16GB ram should avoid me from going into swap.
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

I used L2ARC before but now use that SSD in another computer, I doubt it would do too much as L2ARC. For very small files accessed often they'd be in ARC (ram) which would be faster then L2ARC. If you have lots of large file (10GB movies) you access not too often the L2ARC would probably get blown out too often. I want to test out ZIL, because it will make some difference I believe.

Usually for L2ARC you add an SSD that is fast at reading and can hold whatever your 'working dataset' is. My working dataset is a few GB of thumbs from xbmc shared across the network, so I think it can all be in my 16GB ram.

 

[paleoN]

I didn't make the zpool in FreeNAS webui because a raid using whole disks gives the best performance.

We need to keep in mind this is true for Solaris. Here is some chat about that [post=29023]very thing and partitions[/post]. I have since learned what Solaris actually does when you give it a whole disk.

1. It partitions the drive for you!
2. It turns on the write cache for the disk.

This is the reason whole disks are the best practice on Solaris. By default if you give it a slice it won't automatically enable write cache as other filesystems shouldn't necessarily have it on. FreeBSD enables the write cache regardless.

If you have lots of large file (10GB movies) you access not too often the L2ARC would probably get blown out too often. I want to test out ZIL, because it will make some difference I believe.

The L2ARC, without tuning, won't do much for a 10GB movie file, i.e. streaming workload.

 

[Joshua Parker Ruehlig]

@paleoN
Thanks for the info, so I guess I could just set swap to 0, and let FreeNAS create the zpool with slices. Good to know, but guess it works either way =]

 

[crisman]

I have 10 Samsung Green 2TB in RAIDZ2. I didn't make the zpool in FreeNAS webui because a raid using whole disks gives the best performance. I also did the 4k trick when creating the zpool. It also doesn't give you swap but I think my 16GB ram should avoid me from going into swap.
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

I used L2ARC before but now use that SSD in another computer, I doubt it would do too much as L2ARC. For very small files accessed often they'd be in ARC (ram) which would be faster then L2ARC. If you have lots of large file (10GB movies) you access not too often the L2ARC would probably get blown out too often. I want to test out ZIL, because it will make some difference I believe.

Usually for L2ARC you add an SSD that is fast at reading and can hold whatever your 'working dataset' is. My working dataset is a few GB of thumbs from xbmc shared across the network, so I think it can all be in my 16GB ram.

It seems you're an expert on Freebsd/linux OS, so for you should be easy to try all different solutions without any kind of difficulty. I'm a noob on this OS's, my experience is mostly on Windows, I would like to make the best setup in my system to avoid loosing data or getting bad performance in the system caused by bad configuration. Maybe during this proccess I will contact you if you're available of course to give me some help/hints on the final setup.

Thanks a lot for all this questions.
Crisman.