Expanding existing FreeNAS Server

[Fireball81]

Hello,

i have build a FreeNAS 19" server some time ago for backup purposes, based on a SuperMicro X11SSL-CF with 32GB RAM in a 4U XCase. (the non-expander, low-noise version)
Thanks to the embedded LSI 3008, (IT) i connected two SAS connectors to the backplane of the XCase which gave me 8 drive bays.
I bought 7x8TB Seagate Ironwolfs HDDs and build myself a RaidZ2 vdev out of these 7 drives. (the last drive bay could be used for a hot-spare for example)
Everything is working as it should (well most of the time the machine is offline anyways) but i need more storage capacity now.

I would like to go with another RaidZ2 vdev of 7x8TB, to backup a new production storage volume we created recently.
To do this, i need to buy a new HBA and i wanted to have your opinion which one works best with the latest version of freenas and the 8TB Ironwolf HDDs.
I would probably go with something rather inexpensive here, cause performance is not a big concern, so one of those community choices like the M1015 would do the trick but given
that the HBA is a little bit older, i am not sure if it will work fine with the above mentioned 8TB drives.
An expander card would also work for me if that is a viable alternative in your opinion.

As always thanks in advance for your feedback and opinions.

Dennis

 

[Johnnie Black]

cause performance is not a big concern, so one of those community choices like the M1015 would do the trick but given
that the HBA is a little bit older, i am not sure if it will work fine with the above mentioned 8TB drives.

Any SAS2008 based controller like the M1015 works fine with 8TB (and larger) disks, and performance wise with HDDs will be the same as the newer models.

An expander card would also work for me if that is a viable alternative in your opinion.

Also an option, e.g., Intel RES2SV240, though it's usually more expensive on ebay than a second HBA.

 

[nightshade00013]

Instead of buying another SAS controller just get a SAS expander.

You will end up with more ports for further expansion at the same or less cost as a SAS2 controller.

https://www.amazon.com/HP-468406-B21-Sas-Expander-Card/dp/B0025ZQ16K

 

[Fireball81]

First of all thank you for your quick replies.
I found the HP expander on ebay pretty cheap
http://www.ebay.de/itm/HP-24-Bay-PC...886211?hash=item25b2670643:g:JgMAAOSwZd1VdbXS
Is that the one you linked on amazon? It seems to be.
Its so cheap that it raises the question if there is some difference to consider between the HP and the Intel.
I would need new cables to route from the expander to the backplane, two regular SFF-8087 cables should do, right?

 

[Johnnie Black]

HP is SATA2 only, SATA1 max if you get one with an older firmware, and you'll need a HP controller to update it, Intel is SATA3, fewer ports though.

 

[nightshade00013]

Yes that is pretty much the same one. Most of the people selling large amounts of them have already updated the firmware since they want to sell it as a SAS 2

Yes it's just extra cables, technically only have to route one to the expander and then fan off from it to the drives. If you are using hard drives SAS2 is more than fine for transfer. Even SAS1 is faster than most drives can handle but it had a nasty 2TB limit that is an issue for most people.

There are other SAS expanders out there but a good chunk are built into the backplanes.

As far as the post talking about SATA1, SATA2, and SATA3 I think they mean SAS.

 

[Johnnie Black]

As far as the post talking about SATA1, SATA2, and SATA3 I think they mean SAS.

No, I mean SATA since I assume he'll be using SATA disks.

SATA1=150MB/s
SAS1/SATA2=300MB/s
SAS2/SATA3=600MB/s

This is important for an expander link since each x4 mini SAS link will have 4 times the bandwidth of the disks used, so if all disks link at SATA2 speed there will be a 1200MB/s link for all disks, 2400MB/s if dual link is used, if disks link at SATA3 speed it will double the available bandwidth, and half that if they all link at SATA1 speed.

 

[Stux]

I skipped the HP Expander (too many problems) and went with an Intel.

I'd also suggest going with an Intel SAS Exoander. You don't have a lot of PCIe slots. And an Intel exoander doesnt *need* to use a slot.

 

[Fireball81]

What were your problems with the HP expander, if i may ask?
I was already a liuttle bit set for this solution, it seems the right approach for a Backup machine where performance is not a big concern.
On the other hand the Intel SAS expander is double the price of a regular M1015 which will suffice for what i have in mind.

 

[nightshade00013]

No, I mean SATA since I assume he'll be using SATA disks.

SATA1=150MB/s
SAS1/SATA2=300MB/s
SAS2/SATA3=600MB/s

This is important for an expander link since each x4 mini SAS link will have 4 times the bandwidth of the disks used, so if all disks link at SATA2 speed there will be a 1200MB/s link for all disks, 2400MB/s if dual link is used, if disks link at SATA3 speed it will double the available bandwidth, and half that if they all link at SATA1 speed.

SATA standards do not apply to a SAS controller in the way you are applying.

SAS-1: 3 Gbit/s
SAS-2: 6 Gbit/s
SAS-3: 12 Gbit/s

https://en.wikipedia.org/wiki/Serial_Attached_SCSI

If you are referring to a SAS controller you should be supplying SAS specs. Just because they are semi compatible it does not mean they are interchangeable.

 

[Stux]

What were your problems with the HP expander, if i may ask?

I read some of this thread: https://hardforum.com/threads/hp-sas-expander-owners-thread.1484614/

And decided to skip.

I went with a 36 port Intel SAS Expander (RES2CV360) instead. It hasn't arrived yet. This will allow me to run 28 drives and dual-link to a single HBA, sharing the full bandwidth of the HBA (8x PCIe2.0)

I have two M1015/M1115 already, and because I'm upgrading my 24 bay system to an ESXi system, I need to pass-through the HBAs, which means I need another HBA card, instead of using the onboard SATA. And then with a PCIe SSD and a 10gbe I'm out of PCIe slots, and bottlenecking some... so by using an expander, I'll gain a spare HBA for other projects, and save two PCIe slots.

That's my reasoning. So, although the Intel 24 port expander is perhaps double the price... if you end up buying it anyway one day... then that'd work out even more expensive :)

I figure if I can't find a use for the 2nd HBA (unlikely), Its pretty easy to move on a genuine, already flashed and working with freenas HBA :)

 

[Johnnie Black]

SATA standards do not apply to a SAS controller in the way you are applying.

All SAS HBAs and expanders that can work with both SAS and SATA devices have a max link speed with each type of device, usually they link at the same speed, i.e., SAS1/SATA2, SAS2/SATA3, the HP is an exception, and if the user is using it with SATA disks the SATA speed of that expander is what matters, e.g.,:

Intel RES2SV240 will link wth SATA disks @ 600MB/s providing 2400MB/s total bandwidth per link.

HP SAS expander will link with SATA disks @ 300MB/s or 150MB/s if it's one with an older firmware (IIRC <1.52) providing 1200 or 600MB/s per link, 600MB/s per link can be a bottleneck with FreeNAS, possibly even 1200MB/s, depending on the hardware used.

I own both expanders so my observations are based on experience.

 

[Stux]

So, end result, the HP SAS Expander has a limit of 600MB/s per uplink with old firmware, and 1200MB/s with new firmware, when using SATA drives, and you need an HP SAS Controller to update the firmware, and if you don't have an HP Server to use use your HP Controller to update your HP Expander, you also need to hack the firmware updater.

And that's why they're cheap.

Alternatively, any LSI based SAS2 expander (ie Intel's) will uplink at 2400MB/s (per uplink), and possibly 4800MB/s for a SAS3 version.

Correct?

 

[Johnnie Black]

So, end result, the HP SAS Expander has a limit of 600MB/s per uplink with old firmware, and 1200MB/s with new firmware, when using SATA drives, and you need an HP SAS Controller to update the firmware, and if you don't have an HP Server to use use your HP Controller to update your HP Expander, you also need to hack the firmware updater.

And that's why they're cheap.

Correct

Alternatively, any LSI based SAS2 expander (ie Intel's) will uplink at 2400MB/s (per uplink), and possibly 4800MB/s for a SAS3 version.

4800MB/s only if SAS3 devices are used, since there's no SATA at equivalent speeds.

Note also that the linking speed is the theoretical max, there's always some overhead, though little compared to other protocols, this is the real world max speeds I measured using SSDs:

2400MB/s link - 2200MB/s max usable
1200MB/s link - 1100MB/s max usable

Dual linking doubles these speeds.

Also, avoid mixing disks with slower linking speeds, i.e., SATA2 and SATA3 disks on the same expander, it will bring the whole link speed down if there is simultaneous access to them.

 

[Stux]

Thanks.

And the common SAS2 LSI HBAs have only 8x PCIe2 bandwidth to the host system... which is 4GB/s excluding overhead, so that would reduce the maximum SAS2 dual-link speed from an estimated 4400MB/s even further.

 

[Johnnie Black]

And the common SAS2 LSI HBAs have only 8x PCIe2 bandwidth to the host system... which is 4GB/s excluding overhead, so that would reduce the maximum SAS2 dual-link speed from an estimated 4400MB/s even further.

Dual SAS2 link would be best with a PCIe 3.0 HBA, since PCIe has a lot of overhead, in my tests the max usable bandwidth of a x8 PCIe 2.0 slot varies from 2500 to close to 3000MB/s max.

 

[Stux]

Dual SAS2 link would be best with a PCIe 3.0 HBA, since PCIe has a lot of overhead, in my tests the max usable bandwidth of a x8 PCIe 2.0 slot varies from 2500 to close to 3000MB/s max.

Did you try dual-linking between two different 8x PCIe 2.0 cards on two different slots?

 

[Johnnie Black]

Did you try dual-linking between two different 8x PCIe 2.0 cards on two different slots?

I'm almost certain that's not supported, but never did try.

 

[Fireball81]

After reading all of this and thinking about it a little my preference would be also the RES2CV360SAS Expander.
The only thing that bothers me is the fact that i have to fiddle around a little bit to fixate it in my XCase, maybe i could do it with some kind of foam tape underneath the pcb of the expander to bring it close to the backplane.
There is a good offer for those 36 port expanders on ebay right now, so that might be worth it.
Anything i should be aware of before going down that road with the RES2CV360?

Thank you

 

[Stux]

There is a good offer for those 36 port expanders on ebay right now, so that might be worth it.

I suspect this is the offer I took up :)