SAS Expanders

[rwhitlock]

I’ve done many days of research including reading through the entire Confused about that LSI card? Join the crowd ... thread and I have one question for cyberjock. Does the LSI firmware version have to match the FreeNAS driver? Lol!

In all seriousness, even after reading many threads including Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA I still am left with questions concerning SAS expanders. There are many conflicting rules on the internet:

• Don’t use expanders with SATA
• Don’t cascade expanders
• Don’t mix SATA drive and SAS drives on the same expander
• Don’t use expanders with ZFS
• Don’t use expanders with SATA on an external disk shelf
• Don’t use two expanders on the same HBA

Beyond these commonly asked questions that get a variety of answers that I would still like to discuss here, I have some new questions.

• When Dual Linking, does both links need to come from the same HBA?
• How much will the expander affect latency?
• Reliable vendor for SAS cables

There appears to be 4 main vendors. Is there a HBA / expander compatibility Matrix Wiki somewhere?

• Intel
• HP which uses (PMC)
• LSI
• Chenbro

__________________________________________________________________________


My Plan…

2x SAS 9207-4i4e, in IT mode --> HP expander with Dual Linking from 2 HBA’s --> 4x Mini-SAS SFF8087 on the HP backplane --> 16 300GB HP SAS drives.

It is my understanding the backplane on the HP DL380 G6 SFF HDD cage is pass-through and not an expander.

The external connectors on the SAS 9207-4i4e would go to a second expander (maybe a Chenbro) residing in a 16 bay Norco. In the Norco case would contain 16 3Tb SATA drives. Also with future expandability in mind, I would like to be able to add a second external disk shelf.

Both the external disk shelf and SAS drives would be in their own zpool. Both with the same configuration 8 vdev’s with mirrored drives.

__________________________________________________________________________


Stuff I already own pertinent to this endeavor:

• Hp DL380 G6 with 72GB of ECC RAM and 8 300GB SAS drives. Currently connected to a HP P410.
• PCIe slot space is limited. I would prefer to only use 2 slots for storage purposes, but no more than 3.
• 16 bay Norco case with 4 3TB SATA drives. Currently not in use.

 

[sfcredfox]

I'll give you a few things from running similar configurations, and let people with references answer the questions above.

I ran a 380G5 for a while. In my experience, the internal backplanes are pass thrus, at least it was on the G5. It used two 8484 connectors which sucks because you had to eat up two 8087 ports for the internal drives. Since doing this will ensure there will never be bandwidth issues for those internal drives, that's a great place to put SLOGs and ARCL2s if your system ever needs that.

I used HP's MSA70 for external storage. Its 3G 2.5" drives up to 25 on a single 8088. The MSA60 is the 3.5" drive version. I like them better than what I saw of the Norco for things like redundant power supplies and simple other things, but I never owed a Norco, so take that for what it's worth. The 6G version is the D2600/2700. With SATA platter disks, you can't get much more than 150MBps anyway, so I don't see a reason to worry about going 6G, if you want to start using SSDs, that would be a reason, but then you'll need to read some of the SAS posts about bandwidth.

Some of the experts should weigh in on this more, but I think you can definitely cascade enclosures, I think some of the rules apply in difference scenarios and don't in others. For 6G SAS, one 8088 port carries 24MBps in four channel (6Gx4). If you cascade enclosures, you're just splitting that even more. Again, if you're using spinning SATA, they don't get much better than ~150-160MBps unless they're awesome. You just don't want to put so many disks on there that they contend with each other for bandwidth. For 3G SAS, its 12. If you had enclosures with SSDs however that get up to 500 or so MBps, you can see that cascading those would be a problem. Evan backplanes with built in expanders can be a problem if you put too many disks in them.

If you want better feedback on storage config, you'll need to specify more details of the system, and what it's for. Doing 16 SAS drives hints that it's not a home use media storage system, but you don't have anymore details.

 

[Ericloewe]

Don’t use expanders with SATA

SATA devices are nominally fully supported by the SAS spec, even behind expanders. It is expected to work.

Don’t cascade expanders

Nothing inherently wrong with cascaded expanders. Disclaimers about available bandwidth apply, obviously, so don't overdo it. But it is a valid config.

Don’t mix SATA drive and SAS drives on the same expander

Mixing shouldn't be a problem, either. SATA traffic is encapsulated in SAS connections up the tree.

Don’t use expanders with ZFS

Absolutely ridiculous. Expanders are a valid solution.

Don’t use expanders with SATA on an external disk shelf

Again, a very valid solution, with some aspects that need to be considered. Mostly a datacenter-grade "DO NOT TOUCH" solution.

Don’t use two expanders on the same HBA

If it makes you feel better. ZFS, FreeNAS and the HBA couldn't care less.

When Dual Linking, does both links need to come from the same HBA?

SAS supports a number of configurations. Specifics may be iffy and anything beyond single paths needs proper testing and validation and manufacturer support. Dual-pathing is possible, as is allocating 8 lanes instead of 4. Topologies with interconnected expanders also exist and things can get real crazy.

How much will the expander affect latency?

Latency? Assuming sufficient bandwidth, it'll be dwarfed by seek times on the drives. SLOG should be as close to the processor as possible, physically and logically. L2ARC is more flexible. Not much info regarding all-SSD pools out there.

There appears to be 4 main vendors. Is there a HBA / expander compatibility Matrix Wiki somewhere?

All SAS devices are nominally compatible. In practice, people stick to LSI throughout around here. Intel uses LSI controllers, as well, so they're a rebrand mostly.

is pass-through and not an expander.

Keep in mind maximum cable lengths. I think it's around 3-5m for SAS.

 

[rwhitlock]

Sfcredfox, you have sparked my interest in the HP D2600! It looks much better than the Norco, but obtaining sleds from HP to use generic HDD's looks difficult. Besides the sleds, are there any other obstacles ie. SMART pass through?

Honestly, I included a short description of my hardware only to provide context and relevance to the discussion. Cyberjock and the Resident Grinch have amazing mind dump's and primer articles. I was hoping to entice them and revival the need for an SAS expander primer.

___________________________________________________________________________________________________________________________________________

After my initial post I have researched more, and would now completely have to agree with Ericloewe.

There does appear to be a class of crappy hardware that does not adhere to the SAS standard. I think Highpoint is a good example of this. They use Marvell controllers to reduce costs, but at the cost of quality.

Please correct me if I’m wrong, but it seems LSI and PMC are the market leaders for SAS and SAS expanders. Intel and Chenbro use LSI controllers. It looks like most enterprise grade SAS hardware is interchangeable. I started to create a compatibility matrix… so attached is 8 hours worth of research. It is far from complete, but it quite possibly is a waste of time.


_________________________________________________________________________________________________________________________________________


Here is an excerpt from “MindShare Technology Series SAS Storage Architecture” by Mike Jackson. After reading through these restrictions a believe two initiators (HBAs) can connect to one expander.

"
Topology Restrictions
When discussing SAS typologies, it is important to note several rules regarding legal construction of domains that serve to reduce the cost of the network by limiting its complexity.


Limited Addressing
First, as mentioned earlier, the number of SAS addresses that may be used by an edge expander set is limited to 128, and that includes addresses from attached devices as well as those addresses used internally for virtual Phys within an expander.


Dual Port Configurations
Second, as shown in Figure 4-13, it is permissible for a device to have a wide port in order to provide a redundant path or better bandwidth. However, it is not acceptable for an expander to have more than one path available to an endpoint device, as Figure 4-14 illustrates. Software will disable one path to fix this multipath problem if it is detected.
To clarify this rule: while it is acceptable for an end device to have more than one path available to reach another end device, it is not acceptable for expanders to have a choice about which path to take to the same device.

No Loop Topologies
Third, along the same lines, Figure 4-15 illustrates a loop error in the topology. Loops are not allowed in the paths between expanders, both to maintain simplicity in the topologies and because of the very simple method that expanders use to inform devices in the domain about detected changes to the fabric. For example, when an expander detects that an address has been added or removed, it sends a broadcast message on all the rest of its SAS ports to inform connected devices that a change has occurred. Each initiator must then query the expanders to learn what has changed. If two expanders have more than one path to each other, this can result in an infinite loop of change broadcasts as one
broadcasts the change, then the other sees the broadcast and interprets it as another change and repeats the process by broadcasting back to the first one. The fix for this case is similar to the previous one: disable one of the paths to prevent the loop condition.
"
__________________________________________________________________________________________________________________________________________

Considerations when mixing SAS and SATA drives

Awesome White Paper SAS Primer

Brief SNIA SAS presentation

If you feel like being overwhelmed! Here is 600 page book on SAS

 

[sfcredfox]

obtaining sleds from HP to use generic HDD's looks difficult. Besides the sleds, are there any other obstacles ie. SMART pass through?

You can get the drive trays from ebay just like the D2600.
There are zero issues with smartctl getting drive information as long as you use a real HBA and not any form of RAID controller. I use the M1015, flawless victory. I also have an LSI 92078e that is a 6G contoller. It saves on having to buy 8087 to 8088 adapters and using another PCI slot.

The older MSA60 and MSA70 had a drive size restriction, I think 2TB or smaller. I don't think the 2600/2700 have that restriction, but I haven't researched that thoroughly yet.

I am actually doing both (using the M1015 with an adapter and the LSI 9207-8e). I'll likely end up with three enclosures in a few years when I'm done.

 

[cyberjock]

@rwhitlock

Sorry I didn't answer you before. I've been busy and away from the forums for a few weeks. Did you get all of your questions answered? If not, can you restate the ones you'd like me to touch on?

 

[rwhitlock]

Thanks Cyberjock.

After reevaluating the PCIe cards I intend to place in the server I require a HBA with 2 internal and 2 external ports. The only one I can find is the Areca 1320ix-16 which does not appear to be recommended anywhere. I would also need an expander that does not take up a PCIe slot.

 

[Ericloewe]

Thanks Cyberjock.

After reevaluating the PCIe cards I intend to place in the server I require a HBA with 2 internal and 2 external ports. The only one I can find is the Areca 1320ix-16 which does not appear to be recommended anywhere. I would also need an expander that does not take up a PCIe slot.

Expanders don't need PCI-e connectivity. Some expanders can use a PCI-e slot for power, but they can be powered directly, instead (typically molex).

 

[cyberjock]

Yeah.. I like Areca for hardware RAID (I've owned several over the years) but I don't recommend them at all for FreeNAS.

I almost lost my pool because I was using Areca. Don't do it.. it's not worth it. :P