Netapp 4486 & LSI 9200-8e Issues > 2tb

[brad87]

My Hardware
Netapp 4486 disk shelf with 2x IOM6 (One cabled to Freenas host via QSFP - SFF8088 cable)
LSI 9200-8e (flashed with IT firmware on Freenas host)
Freenas Host: Supermicro X9 with 5640 xeons, 72gb ram

My Issue
When attempting to load in a caddy with 4/6 tb drives, I am getting the following in the dmesg log of the freenas host:

Code:

da1 at mps0 bus 0 scbus0 target 8 lun 2
da1: <HITACHI HUS724040ALE64DB NA01> Fixed Direct Access SPC-3 SCSI device
da0 at mps0 bus 0 scbus0 target 8 lun 1
da1: Serial Number PAGU83ES           
da0: <HITACHI HUS724040ALE64DB NA01> Fixed Direct Access SPC-3 SCSI device
da0: Serial Number PAGU82VS           
da0: 600.000MB/s transfersda1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: Attempt to query device size failed: ABORTED COMMAND, Internal target failure
da0: Command Queueing enabled
da0: Attempt to query device size failed: ABORTED COMMAND, Internal target failure

As a test I have populated a caddy with a single 1tb drive, and did not have this problem, which makes me really nervous.:

Code:

da25 at mps0 bus 0 scbus0 target 19 lun 1
da25: <HITACHI HUS724040ALE64DB NA01> Fixed Direct Access SPC-3 SCSI device
da25: Serial Number PAGS6N6S           
da25: 600.000MB/s transfers
da25: Command Queueing enabled
da25: 953869MB (1953525168 512 byte sectors)

Code:

root@constellation:~ # camcontrol devlist
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 8 lun 0 (pass1)
<HITACHI HUS724040ALE64DB NA01>    at scbus0 target 8 lun 1 (da0,pass2)
<HITACHI HUS724040ALE64DB NA01>    at scbus0 target 8 lun 2 (da1,pass20)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 9 lun 0 (pass3)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 10 lun 0 (pass4)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 11 lun 0 (pass5)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 12 lun 0 (pass6)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 13 lun 0 (pass7)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 14 lun 0 (pass8)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 15 lun 0 (pass9)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 16 lun 0 (pass10)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 17 lun 0 (pass11)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 18 lun 0 (pass12)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 19 lun 0 (pass13)
<HITACHI HUS724040ALE64DB NA01>    at scbus0 target 19 lun 1 (da25,pass51)
<NETAPP DS448IOM6 0173>            at scbus0 target 20 lun 0 (pass14,ses0)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 21 lun 0 (pass15)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 22 lun 0 (pass16)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 23 lun 0 (pass17)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 24 lun 0 (pass18)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 25 lun 0 (pass19)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 27 lun 0 (pass21)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 28 lun 0 (pass22)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 29 lun 0 (pass23)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 30 lun 0 (pass24)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 31 lun 0 (pass25)
<MARVELL LUIGI_V2_STSB_DB 2015>    at scbus0 target 32 lun 0 (pass26)
<DOGFISH 30G Q0707A>               at scbus2 target 0 lun 0 (pass27,ada0)
<HPT DISK 0_0 4.00>                at scbus6 target 0 lun 0 (pass28,da2)
<HPT DISK 0_1 4.00>                at scbus6 target 1 lun 0 (pass29,da3)
<HPT DISK 0_2 4.00>                at scbus6 target 2 lun 0 (pass30,da4)
<HPT DISK 0_3 4.00>                at scbus6 target 3 lun 0 (pass31,da5)
<HPT DISK 0_4 4.00>                at scbus6 target 4 lun 0 (pass32,da6)
<HPT DISK 0_5 4.00>                at scbus6 target 5 lun 0 (pass33,da7)
<HPT DISK 0_6 4.00>                at scbus6 target 6 lun 0 (pass34,da8)
<HPT DISK 0_7 4.00>                at scbus6 target 7 lun 0 (pass35,da9)
<HPT DISK 0_8 4.00>                at scbus6 target 8 lun 0 (pass36,da10)
<HPT DISK 0_9 4.00>                at scbus6 target 9 lun 0 (pass37,da11)
<HPT DISK 0_10 4.00>               at scbus6 target 10 lun 0 (pass38,da12)
<HPT DISK 0_11 4.00>               at scbus6 target 11 lun 0 (pass39,da13)
<HPT DISK 0_12 4.00>               at scbus6 target 12 lun 0 (pass40,da14)
<HPT DISK 0_13 4.00>               at scbus6 target 13 lun 0 (pass41,da15)
<HPT DISK 0_14 4.00>               at scbus6 target 14 lun 0 (pass42,da16)
<HPT DISK 0_15 4.00>               at scbus6 target 15 lun 0 (pass43,da17)
<HPT DISK 0_16 4.00>               at scbus6 target 16 lun 0 (pass44,da18)
<HPT DISK 0_17 4.00>               at scbus6 target 17 lun 0 (pass45,da19)
<HPT DISK 0_18 4.00>               at scbus6 target 18 lun 0 (pass46,da20)
<HPT DISK 0_19 4.00>               at scbus6 target 19 lun 0 (pass47,da21)
<HPT DISK 0_20 4.00>               at scbus6 target 20 lun 0 (pass48,da22)
<HPT DISK 0_21 4.00>               at scbus6 target 21 lun 0 (pass49,da23)
<HPT DISK 0_22 4.00>               at scbus6 target 22 lun 0 (pass50,da24)

 

[Elliot Dierksen]

If you have repurposed NetApp drives, they may be formatted with 520 byte sectors instead of 512. They won't work at all that way. I purchased some former NetApp drives from eBay, and I was able to reformat them using the commands from this page http://www.davidrusseltrask.com/blo...size-to-512-sector-size-using-freenas-freebsd. Keep in mind that the reformats this way take a REALLY LONG TIME.

 

[brad87]

These drives were not from a netapp, I sourced them from work where they were previously in some sort of ceph cluster. The last thing that I had done to them was run badblocks destructive test.

here is what im getting from diskinfo, does this output indicate that these need to be formatted?

root@constellation:~ # diskinfo -v da1
diskinfo: da1: ioctl(DIOCGMEDIASIZE) failed, probably not a disk.

 

[brad87]

Furthermore, this is the output of geom disk list, looks lik eit already has 512 sectors

Geom name: da0
Providers:
1. Name: da0
Mediasize: 0 (0B)
Sectorsize: 512
Mode: r0w0e0
descr: HITACHI HUS724040ALE64DB
lunid: 5000cca22bcb7ba7
ident: PAGU82VS
rotationrate: unknown
fwsectors: 0
fwheads: 0

Geom name: da1
Providers:
1. Name: da1
Mediasize: 0 (0B)
Sectorsize: 512
Mode: r0w0e0
descr: HITACHI HUS724040ALE64DB
lunid: 5000cca22bcb7bb9
ident: PAGU83ES
rotationrate: unknown
fwsectors: 0
fwheads: 0

 

[Elliot Dierksen]

I am not sure. Mediasize being 0 is a bad thing. Is it possible those drives have a jumper settings that they don't spin up until explicitly told to do so?

 

[brad87]

There doesnt appear to be any jumpers at all on the drives, also I am using a combination of hitachi / and seagate drives for this testing. The WD 1tb spun up and works, its just the larger drives that are giving me trouble.

 

[Elliot Dierksen]

That is an older controller based on the LSI 2008 chipset. That might be part of the problem. An LSI 9207-8e is based on the 3008 chipset, and those are ~= $50 on eBay. Do you have something else with a newer HBA where you could try the drives to confirm that they are working? It does seem unlikely that they would all be bad though.

 

[brad87]

ill order a newer HBA and see if that resolves anything. I know for sure that the drives are fine as they work when I hook them up to my Highpoint card

 

[Elliot Dierksen]

I know for sure that the drives are fine as they work when I hook them up to my Highpoint card

That is pretty close to a definitive answer there.

 

[brad87]

That is pretty close to a definitive answer there.

Im just hoping that the issue doesnt lie in the disk shelf, or the caddies, or the IOM6 ... The netapp site and firmware upgrade process is a little locked down to homelabbers.. I dont know where to start investigating firmware updates for the shelf

 

[Elliot Dierksen]

I am assuming those are 3.5" drives. I have seen people reuse NetApp shelves, but I haven't done it myself. I have some external drives in HP 2700 (2.5" x 25) and HP D2600 (3.5" x 12) shelves, and those work great. They are also very inexpensive on eBay. I am sure there are plenty of other options as well. The HP ones are the only ones with which I have firsthand knowledge.

 

[brad87]

I was mistaken when I said earlier that the drives function fine with other backplanes. WHen I moved the adapter into my linux host, it did not work there either. I hava also tried to use them with my RocketRaid card and an adaptec HBA, and am facing similar messages in dmesg, about 0 512 byte sectors

 

[Elliot Dierksen]

Perhaps it is an issue with the firmware on the drives, but I don't have any suggestions for addressing that.

 

[brad87]

the last thing I did with these drives was run badblocks -wsv -b 4096 (on a machine at work). Could this have anything to do with my issue

 

[Elliot Dierksen]

I don't think so, but I am not really sure. I would try putting them back into a machine where they did work and see if they still do.

 

[brad87]

The more I delve into this , the more I am thinking this may have something to do with SAS Pin 3, as the drives dont seem to spin up in the netapp enclosure at all. Seems to be a common trend among these drives. I have ordered some Kapton tape to test this theory