Multipath alerts with EMC KTN-STL3: "Multipath multipath/discX connections are not optimal. Please check disk cables.

[duckasylum]

Hi! So I am trying to build a FreeNAS storage system from old decommissioned hardware for my "lab". I have the hardware assembled and FreeNAS installed. After adding my second disk enclosure I got the error in the title. If I understand multipath correct the problem seems to be with 2 disks (error per path per disk).

My FreeNAS box (IBM x3650 M2) has it's HBAs (LSI SAS31601E) connected to the disk enclosures (EMC KTN-STL3) the following way:

• HBA 1 port 1 to disk enclosure 1 controller A double circle connector
• HBA 1 port 2 to disk enclosure 2 controller B double circle connector
• HBA 2 port 1 to disk enclosure 1 controller B double circle connector
• HBA 2 port 2 to disk enclosure 2 controller A double circle connector

My thoughts and reasoning are:

1. Have I connected the FreeNAS box and disc enclosures incorrectly? I couldn't figure a better way to connect the pieces and retain multipath. Daisy-chaining the DAEs, as the EMC solution is originally intended, seemed not so wise but maybe there is a point I am missing.
2. Is the second DAE faulty or the "disk cables" are indeed faulty? This problem appeared as soon as I attached the second DAE.
3. Are the disks actually faulty? I had one dead drive in DAE 1 and after replacing it another error disappeared. Also I have multiple spares to play around.

I did a quick check on the forums and all similar issues were not related to true multipath but to setups where the system mistakenly though there was multipath (i.e disks with same serials or such).

Has anyone countered a similar problem or has tried to build a solution like I am with multiple EMC KTN-STL3 DAEs? I thought I'd ask before starting to dismantle the system and pulling drives.

I am attaching a picture of the alerts and a picture of the DAE controllers for those not familiar with this particular unit. Also the multipath page from FreeNAS.

 

[blueether]

Not addressing your issue (but as you have very similar hardware I'll ask), I have a M3 or two and a single KTN-SL3 and 2 LSI 4i4e cards and I multipath doesn't seem to automagically do multipath. Freenas sees the 15 drives as 30 (TrueCommand shows onlt 15?).

Does haveing the pool already set up matter?
I'm using proxmox to pass both LSIs to freenas, will this matter?

 

[duckasylum]

My setup is full bare metal so I can't comment on the proxmox. As I understand 15 phsical disks are supposed to show up as 30 disks under /dev/ as da0-da29 and if you have multipath you should see 15 devices under /dev/multipath as /dev/multipath/disk1-15. With my setup under FreeNAS GUI storage > disks I see da0-da63 disks (60 for the storage, 2 for application pool and 2 for boot pool). Under storage > multipath I would ideally see 30 multipath/disk1- disk30 "multipath disks" with 2 da0 -da59 devices under each one (my numbering is a bit different because my 4 non data pool devices are enumerated between the DAE disks).

I have now swapped out three of the disks in the second DAE and I seem to have resolved the degraded/non-optimal multipath issue. Switching the disks also revealed a new problem. When I booted up the last time, one of the disks were totally dead. So I only got 29 multipath disks. Now when I started to swap the other three disks out somehow for reasons unknown the multipath disk count stayed the same and I always got two straggling disks. I finally decided to combine them manually into a multipath disk with camcontrol to verify the serials and gmultipath to create the multipath device. Now I can start thinking of how to actually combine the disks into vdevs and pool them together in the most optimal way.