IBM ServeRAID M1015 & JBOD & with a side of Performance

[sfcredfox]

Question: Has anyone ever used their M1015 to connect to external storage (JBOD) enclosure?
If so, can you use SFF-8088?

Scenario:
I am interested if you can connect an HP MSA70 (it's a 25 disk enclosure, uses SFF-8088) to an IBM M1015.
This would require the M1015 being able to be cabled to an enclosure using SFF-8088, which the M1015 uses 8087 internal ports.

References:
https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/
Post about the M1015. Describes flashing and drive configuration, but not connections to external. I have heard you can get different SAS cables from JGreco, but not sure if it supports the above scenario.

http://sale.sfpcables.com/cab-8088-8087-p1m?gclid=CNXB5qj17cECFeU-Mgod7zMAvA
Here's the cable in question, it exists, but would doing this work? Hmm.

http://h20565.www2.hp.com/portal/si...ac.admitted=1415548382258.876444892.492883150
MSA70. It's currently connected to an LSI SAS HBA (using SFF-8088) and working perfectly, just curious about the M1015 to do the same thing.

=============================
The side of performance question:
Does anyone have data or found posts about SAS expanders? I'm trying to understand the channels and bandwidth aspect of SAS, but it's still not clear.

I guess the performance question is: Does using an architecture that utilizes a SAS expander cut your available bandwidth to disk? (I think yes, because we are splitting/switching in some manner)

If so, is it significant enough to even matter or can you loose performance for capacity? Are we hurting read/write performance significantly?

Tons of storage scenarios use expanders, does FreeNAS care? Do you?

References:
https://forums.freenas.org/index.php?threads/sas-expanders.22626/
Suggests that they are supported and are being used

http://www.sasexpanders.com/
Provides pretty good example of the scenario in question, also some example LSI cards
http://www.sasexpanders.com/vs/controllers/ -see nonHBA cards, just looking at examples, not a purchase list

 

[jgreco]

Yes, you can. The disk system has no clue about external or internal. It just sees storage attached in the SAS topology. SFF8088 Is the correct way to cable SFF8087 externally. There is some concern as to the number of devices a M1015 can address (may? be limited to 32)

 

[Ericloewe]

You can use simple passive adapter to convert between internal and external cabling. You can also get HBAs with external ports (LSI model numbers ending with -4e4i or -8e).

Available bandwidth is distributed by the expander. There's plenty to spare if you're using mechanical HDDs, so that won't be a problem.

 

[jgreco]

"simple passive adapter"? In the biz we call it "cabling" ;-)

 

[Savage]

Kinda just randomly info dumping and rambling as a reply here. I am no expert on SAS, just done a lot of random tinkering.

Although I own a M1015 for one of my FreeNAS boxes, I have not run a loop out to external SAS cabling with it specifically. However, I have done it with a Highpoint 4320 on a different non-FreeNAS ZFS system, either with cabling adapters or a Chenbro ck12803 SAS expander in the mix.

In some info sources I have read offered up some concern expressed about running SATA drives w/ SAS host controllers as regards to cable length issues. It might be possible to get yourself into a scenario where you have signaling problems from too much cable / too many connectors when using a SATA on SAS host setup. One of my rigs temporary but functional rigs got all sorts of media sense related errors that I was only able to attribute to this.

My highpoint 4320 worked fine w/ a SFF-8087 (male) to SFF-8087) male cable going to a PCI panel adapter that had SFF-8087 internal and SFF-8088 external. From there I went 8088 to the Chenbro on an external case loaded out with drives. In the highpoint BIOS the added ports were enumerated with some extra digits appended, but everything worked just fine. The internal cables + External cables between the host + expander was roughly 2-3 feet in this instance.

Insofar as I know w/ a SAS expander your throughput will be limited to the bandwidth of the 4 lanes running upstream to the host controller. A SAS lane x4 or whatnot. Not sure how adding SATA tunneling (or whatever it is called) affects this stuff. I have read that some expanders can actually be connected w/ 2 SAS connectors (8 lanes) to their host, regardless of how the board's ports are labeled. I have not looked into this in detail so I can't really say. Also personally curious about how well connecting a SAS1/SATA2 expander to a SAS2/SATA6 host works.

When doing some light bench testing of my SAS expander after purchase the batch of 8 spindle drives I used for various hardware raid modes and ZFS testing achieved the same throughput both through the expander and directly connected. (800MB to 1GB / sec) Expander bandwidth probably isn't going to be an issue for most folks outside of some rather high spec, heavy duty or edge case builds w/ spindle disks at least.

 

[Ericloewe]

"simple passive adapter"? In the biz we call it "cabling" ;-)

Apple has twisted the notion of cabling so much that cable can mean anything from "bunch of wires with appropriate connectors and shielding" to "unholy combination of electrical conductors and proprietary logic in proprietary hardware transferring data with proprietary protocols".

Kinda just randomly info dumping and rambling as a reply here. I am no expert on SAS, just done a lot of random tinkering.

Although I own a M1015 for one of my FreeNAS boxes, I have not run a loop out to external SAS cabling with it specifically. However, I have done it with a Highpoint 4320 on a different non-FreeNAS ZFS system, either with cabling adapters or a Chenbro ck12803 SAS expander in the mix.

In some info sources I have read offered up some concern expressed about running SATA drives w/ SAS host controllers as regards to cable length issues. It might be possible to get yourself into a scenario where you have signaling problems from too much cable / too many connectors when using a SATA on SAS host setup. One of my rigs temporary but functional rigs got all sorts of media sense related errors that I was only able to attribute to this.

My highpoint 4320 worked fine w/ a SFF-8087 (male) to SFF-8087) male cable going to a PCI panel adapter that had SFF-8087 internal and SFF-8088 external. From there I went 8088 to the Chenbro on an external case loaded out with drives. In the highpoint BIOS the added ports were enumerated with some extra digits appended, but everything worked just fine. The internal cables + External cables between the host + expander was roughly 2-3 feet in this instance.

Insofar as I know w/ a SAS expander your throughput will be limited to the bandwidth of the 4 lanes running upstream to the host controller. A SAS lane x4 or whatnot. Not sure how adding SATA tunneling (or whatever it is called) affects this stuff. I have read that some expanders can actually be connected w/ 2 SAS connectors (8 lanes) to their host, regardless of how the board's ports are labeled. I have not looked into this in detail so I can't really say. Also personally curious about how well connecting a SAS1/SATA2 expander to a SAS2/SATA6 host works.

When doing some light bench testing of my SAS expander after purchase the batch of 8 spindle drives I used for various hardware raid modes and ZFS testing achieved the same throughput both through the expander and directly connected. (800MB to 1GB / sec) Expander bandwidth probably isn't going to be an issue for most folks outside of some rather high spec, heavy duty or edge case builds w/ spindle disks at least.

Connections between SAS devices and SATA devices use SATA signalling and are as such limited to 1m cables.
Communications between SAS devices always use SAS signalling. If one of the devices is an expander that needs to speak SATA, ATA commands are encapsulated in SCSI packets that are then transmitted normally via the SAS layer. This means that only the last drop uses SATA signalling.

Edit: Stupid half-written comments that sneak their way into final comments!

 

[cyberjock]

Apple has twisted the notion of cabling so much that cable can mean anything from "bunch of wires with appropriate connectors and shielding"

I don't consider Apple to be anything near the required "market leader" status to have their definition of cabling (whatever that might be.. I don't use or care about Apple) be of value to me. ;)

 

[sfcredfox]

Liking the replies on this topic.

Yes, you can. There is some concern as to the number of devices a M1015 can address (may? be limited to 32)

http://www.redbooks.ibm.com/technotes/tips0740.pdf
I should have included this in the references. It lists the M1015 as topping out at 16 devices, but I wasn't sure if that was only limited to the IBM ServRAID firmware for RAID versus the IT mode firmware for straight HBA function? Was why i was asking if anyone was using it as described. Is anyone using more than 16 devices?

Although I own a M1015 for one of my FreeNAS boxes, I have not run a loop out to external SAS cabling with it specifically. However, I have done it with a Highpoint 4320 on a different non-FreeNAS ZFS system, either with cabling adapters or a Chenbro ck12803 SAS expander in the mix....

...My highpoint 4320 worked fine w/ a SFF-8087 (male) to SFF-8087) male cable going to a PCI panel adapter that had SFF-8087 internal and SFF-8088 external. From there I went 8088 to the Chenbro on an external case loaded out with drives. In the highpoint BIOS the added ports were enumerated with some extra digits appended, but everything worked just fine. The internal cables + External cables between the host + expander was roughly 2-3 feet in this instance.

I guess your scenario proves that you can use a card that was designed for internal use to connect external disk as long as you use the correct connectors and cabling.

 

[Ericloewe]

Liking the replies on this topic.


http://www.redbooks.ibm.com/technotes/tips0740.pdf
I should have included this in the references. It lists the M1015 as topping out at 16 devices, but I wasn't sure if that was only limited to the IBM ServRAID firmware for RAID versus the IT mode firmware for straight HBA function? Was why i was asking if anyone was using it as described. Is anyone using more than 16 devices?


I guess your scenario proves that you can use a card that was designed for internal use to connect external disk as long as you use the correct connectors and cabling.

I believe Cyberjock has run some 24 drives with an M1015. There is quite a bit of confusion, though, as several documents list different limits for the LSI 2008 controller. I've seen, 16, 32 (M1015), 128 (LSI SAS 9211-8i packaging) and 256 (LSI SAS 2008 controller spec sheet).

 

[cyberjock]

I've had 24 devices on a single M1015 by using a SAS expander. But nobody here has tried to use more than 32 devices on an M1015, so that's a big question mark.

 

[sfcredfox]

Overall answer:
Looks like the answer to the original question is: Yes, an IBM M1015 can be used to connect external enclosures (up to 32 drives known, and possibly more, untested).

Performance:
No one has indicated that using SAS expanders causes any kind of significant issues of any kind, and is fully supported by FreeNAS (it doesn't know or care)

Hardware specific for the scenario (HP equipment):
A typical HP MSA70 configuration allows two MSA's to be connected in series when connected to HP RAID controllers, allowing one controller port to run up to 50 devices. Since we don't use RAID and we are talking about the IBM M1015 supporting 32 devices (known, maybe more untested), this does not allow multiple enclosures to be connected in series to a single port of the M1015. I do however suppose you can connect up to 2 HP MSA70's per M1015, one per port? (I'll test this when I get a second enclosure available to FreeNAS).

Swap a controller to test the M1015 (Question):
Let's say you want to test more than 32 device limit of the M1015.
You already have two pools using 20 devices (15 in one, 5 in another). You want to swap your existing SAS controller for the M1015.

Are there any implications there?
Can the controllers simply be swapped, allowing the system to detect the new controller and all the drives behind it, without risking your pool?
Will it require you to import your pools again? Or would they remain because they should all be found behind your new supported controller?
Has anyone ever done exactly this (swapping your controller with existing data already on disk/pools?

My uneducated assumption is the pools would just have to be imported again? I guess I'd want a backup before hand regardless...

 

[Ericloewe]

If the old controller wrote metadata to the drives instead of just presenting them, you'd probably need to recreate the pool from scratch.

That the IBM M1015 supports 32 drives is pretty much given. The doubt lies in whether or not it will take up to 128 or even 256 drives, or just the 32.

 

[sfcredfox]

That the IBM M1015 supports 32 drives is pretty much given. The doubt lies in whether or not it will take up to 128 or even 256 drives, or just the 32.

Ahh, thanks. Above conclusion edited to reflect the correct known devices supported to 32.

Looks like worst case is restore entire pool(s) from backup if metadata written, thanks. Hopefully will not be the case with original controller being HBA, but who knows.

Thanks again.

Does anyone think the results of doing a controller swap on a already active system is of use to the community, or is that something people don't do, so there would be no use? I was thinking for people who might have had a hardware failure and traded controller for not exact same model?

 

[Ericloewe]

Ahh, thanks. Above conclusion edited to reflect the correct known devices supported to 32.

Looks like worst case is restore entire pool(s) from backup if metadata written, thanks. Hopefully will not be the case with original controller being HBA, but who knows.

Thanks again.

Does anyone think the results of doing a controller swap on a already active system is of use to the community, or is that something people don't do, so there would be no use? I was thinking for people who might have had a hardware failure and traded controller for not exact same model?

If it's an HBA, everything should just be plug and play.

ZFS doesn't care (by design) what controller the drives are connected to. See the awesomely campy video from Sun Germany promoting how ZFS will happily work after doing all sorts of crap to a bunch of drives (crummy USB drives at that).