LSI 9200-8e, DS4246, SSD Speeds and ses: transfer speeds

[xnaron]

Are the interposers in the mix when you are having the issue?

 

[Ender117]

Are the interposers in the mix when you are having the issue?

No.

 

[xnaron]

It would be interesting to try the interposers with all of you shucked drives and see if the issue is still there.

 

[mjt5282]

I have been getting 2-8 SAS iuCRC errors detected daily (using a 3M cable to convert from the Netapp 4246 IOM6 to LSI 2008). I have purchased a separate 1M SAS cable which will take a couple of weeks to arrive. I also ordered a SAS9207-8E LSI SAS HBA which hopefully will avoid the SFF8088 to SFF8087 conversion (and reduce the SAS CRC errors). All of my current SATA drives are negotiating at 6.0 Gb/s (shucked 8TiB and 8TiB reds purchased direct from WD, no interposers). No drives have "dropped out" for several weeks and I have run several long-running zfs scrubs.

 

[xnaron]

I am running the 1m cables with no error except the negotiation issue. I read that cables >1m can be a problem with sas enclusres using sata drives. I'm not sure if that is true in this case but it certainly won't hurt reducing the length and getting rid of the 8087 to 8088 conversion.

How many drives are you running in total? It will be really interesting to see how you make out with the 9207-8e. That one gives the most negotiation 3/6gbps problems for me.

Please keep us posted.

 

[Chris Moore]

The problems you are having are all down to the NetApp shelves. Here is some good info for you:
http://www.plocmstart.com/index.php...drive-shelf-into-a-vendor-generic-jbod-array/

 

[xnaron]

Thanks. I am very familiar with that thread. It is where I got the idea to try the controllers from the HB-1235. The thread does not have anything to do with the issue I am having. It talks about IOM3 modules that only support 3Gbps. I am using IOM6 modules that support 6Gbps.

 

[Ender117]

It would be interesting to try the interposers with all of you shucked drives and see if the issue is still there.

Then they would effectively be SAS disk to systems, they can even do SAS multi-path. OTOH I don't know how to check SAS interface speed...

I am running the 1m cables with no error except the negotiation issue. I read that cables >1m can be a problem with sas enclusres using sata drives. I'm not sure if that is true in this case but it certainly won't hurt reducing the length and getting rid of the 8087 to 8088 conversion.

How many drives are you running in total? It will be really interesting to see how you make out with the 9207-8e. That one gives the most negotiation 3/6gbps problems for me.

Please keep us posted.

I believe the 1m restriction only applies to SATA signal. For us its just the disk to the SAS expander chip in the controller, from there on it's SAS.

 

[Johnnie Black]

I believe the 1m restriction only applies to SATA signal. For us its just the disk to the SAS expander chip in the controller, from there on it's SAS.

Correct.

 

[mjt5282]

11/30/2018 update: my 9207e arrived, i powered down the filer and installed the new HBA, disconnected the internal SAS cable and connected the external SAS cable directly to the LSI 9207e. Rebooted with 16 SATA drives (no interposers), all negotiated normal 6.0 Gb/s. I ran scrubs on both raidz2 pools,found 24K and 8K in errors that were repaired. So far, no SAS checksum errors with about 45 Tb scrubbed. I did notice however that one of the SATA disks is now running at 1.5 Gb/s. 1 Out of 16. I will report back after a few days after my shorter external SAS cable (1M) arrives. Still running with a 3M external QSFP+ to SFF cable. Also ordered a second chassis. I might take out the unused IOM6 module as I am not running HA multipaths.

 

[xnaron]

11/30/2018 update: my 9207e arrived, i powered down the filer and installed the new HBA, disconnected the internal SAS cable and connected the external SAS cable directly to the LSI 9207e. Rebooted with 16 SATA drives (no interposers), all negotiated normal 6.0 Gb/s. I ran scrubs on both raidz2 pools,found 24K and 8K in errors that were repaired. So far, no SAS checksum errors with about 45 Tb scrubbed. I did notice however that one of the SATA disks is now running at 1.5 Gb/s. 1 Out of 16. I will report back after a few days after my shorter external SAS cable (1M) arrives. Still running with a 3M external QSFP+ to SFF cable. Also ordered a second chassis. I might take out the unused IOM6 module as I am not running HA multipaths.

When I had a low number of disks like you I didn't have any issues with all drives negotiating to 6 Gb/s. It was when I went >20 drives that I had some negotiate to 3.0Gb/s. I haven't had any errors after scrubs. Are you using new drives?

 

[Ender117]

I did some testing this weekend. With 9 SATA HDDs and IOM3, only 3 of them were 3Gb/s and 6 are 1.5Gb/s. When I switch to HB-1235 all are 6Gb/s. This all refers to boot with disks plugged in, haven't tried with hot plug yet. But reboot the DS4243 itself seems to not change the interface speed in anyway.

 

[sunshine931]

For what it's worth, I've run DS4246 via a "Dell 6gbps SAS HBA" for years without issue. All HGST SAS 6gbps disks.