Drives detected in some bays but not others

[mikesoultanian]

I have the strangest thing that I can't figure out. I inherited a FreeNAS system with three 24 drive JBODs that have been in service for the past couple years, so I know all of the drives and drive bays work. I deleted all the existing volumes and pulled out all of the drives as I wanted to scan the serial numbers on the drives. Most of the drives that I put back in would be detected and the system log would confirm that, but some drives wouldn't be detected. So, I'd try the same drive in a different drive bay of the same JBOD enclosure and it would work there. What's up with that? I even tried hooking the drive up to my workstation and Windows is able to detect it just fine, partition it, format it, etc. I tried using diskpart and the clear command, but that didn't help - also tried gparted to wipe it and still didn't help.

Here's another weird one along those lines. I have about 10 drives in a RAID10 configuration and I thought I'd test a drive failure by pulling it out of its drive bay (drive was plugged into bay #1) - as expected I received a degraded message - I plugged the drive back in, there were a bunch of flashing lights, and the volume was healthy again. This time, I wanted to test moving the drive to a different bay, so I removed it, volume reported degraded, and I moved it to bay #15 and it switched back to healthy. Here's where it gets weird - when I pull the drive and put back in #1, FreeNAS wouldn't detect it?! I removed it and reinserted it an nada, nothing. I put it back in #15 and now it's working.

Are drive bays somehow being marked unusable by FreeNAS? I now have about 16 bays that won't work with any drive I put in them, but I know they were all working drive bays. Unfortunately I can't reboot this system as I have some VMs running on another volume, otherwise I'd give that a try, but it just seems strange that I'd have to reboot anyways. I inserted all the other 40 drives without rebooting and FreeNAS detected them just fine so I don't understand what's up with the remaining disks.

Btw, I'm running 9.10 - please let me know if there's any more system information that I can provide.

thanks!
Mike

 

[Stux]

I inherited a FreeNAS system with three 24 drive JBODs that have been in service for the past couple years, so I know all of the drives and drive bays work

FWIW, I think that's a faulty assumption.

Can you provide output of zpool status , camcontrol devlist and glabel status in [ code ] tags?

 

[mikesoultanian]

FWIW, I think that's a faulty assumption.

It's not an assumption - the system was a fully operational production system which I took offline to upgrade old drives, move stuff around, and document serial numbers because the previous admins didn't do this.

Here is the information you requested:

Code:

[root@FMTSAN1] ~# zpool status
  pool: HYPERV-HDD-VOLUME1
state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		HYPERV-HDD-VOLUME1							  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/a11e360d-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/a28b6dd1-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/a43bad4b-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/a59bcaaa-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/a708f471-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/a81376aa-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/a9710789-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/aa4f7f5b-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/ab72b204-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/ac5086f4-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/ad6cc662-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/ae4e2e72-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/af6c476b-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/b0430f54-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/b1cdf512-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/b33834f7-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/b4de3e6d-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/b63ac36d-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		logs
		  mirror-9									  ONLINE	   0	 0	 0
			gptid/b766dda8-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/b82086fe-8328-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		cache
		  gptid/b9131e10-8328-11e7-8a76-001b21d46820	ONLINE	   0	 0	 0
		  gptid/b9b851d4-8328-11e7-8a76-001b21d46820	ONLINE	   0	 0	 0

errors: No known data errors

  pool: HYPERV-SSD-VOLUME1
state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		HYPERV-SSD-VOLUME1							  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/ee61aa6d-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/ef1c19f0-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/f01b7ff0-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f0d3330c-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/f1d335b8-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f28cdf8d-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/f38fe494-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f44d4dc8-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/f5517a2d-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f60a5300-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/f70d14dc-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f7c8d96a-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/f8cc8471-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/f988e3d9-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/fa88c20a-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/fb44a1cf-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/fc48934f-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/fd021d5e-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
		  mirror-9									  ONLINE	   0	 0	 0
			gptid/fe063f0b-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0
			gptid/fec13ddd-832a-11e7-8a76-001b21d46820  ONLINE	   0	 0	 0

errors: No known data errors

  pool: SSDV1
state: ONLINE
  scan: scrub repaired 0 in 0h55m with 0 errors on Sun Jul  9 00:55:19 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		SSDV1										   ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/e9f5b86a-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
			gptid/ea2a9eb2-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/20279188-8437-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
			gptid/ea9b1d24-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/4d95f4dd-583f-11e6-bd52-001b21d46820  ONLINE	   0	 0	 0
			gptid/45bd1375-55b6-11e6-bd52-001b21d46820  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/eb48048a-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
			gptid/aa07aa2b-3804-11e6-8133-001b21d46820  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/4875beb5-aaca-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
			gptid/ebec7c32-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/ec292391-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
			gptid/ec5d46d7-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/bc628eef-35f3-11e6-8133-001b21d46820  ONLINE	   0	 0	 0
			gptid/06c0b66d-ad08-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/ed100aeb-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
			gptid/ed4526ba-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/ed82b818-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
			gptid/616f81b7-aacd-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
		  mirror-9									  ONLINE	   0	 0	 0
			gptid/5c642ecf-600a-11e6-a116-001b21d46820  ONLINE	   0	 0	 0
			gptid/ee29c100-243b-11e6-b474-0cc47a55aaf0  ONLINE	   0	 0	 0
		  mirror-10									 ONLINE	   0	 0	 0
			gptid/76f25f22-aad0-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
			gptid/8ac780e7-89ae-11e6-a075-001b21d46820  ONLINE	   0	 0	 0
		  mirror-11									 ONLINE	   0	 0	 0
			gptid/cdfe712f-5dac-11e6-bd52-001b21d46820  ONLINE	   0	 0	 0
			gptid/e6388c64-aad2-11e6-a075-001b21d46820  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Fri Jul 21 03:45:35 2017
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			ada0p2  ONLINE	   0	 0	 0
			ada1p2  ONLINE	   0	 0	 0

errors: No known data errors

Code:

[root@FMTSAN1] ~# camcontrol devlist
<WD WD1001FYYG-01SL3 VR07>		 at scbus0 target 0 lun 0 (pass0,da0)
<WD WD1001FYYG-01SL3 VR07>		 at scbus0 target 1 lun 0 (pass1,da1)
<WD WD1001FYYG-01SL3 VR07>		 at scbus0 target 2 lun 0 (pass2,da2)
<WD WD1001FYYG-01SL3 VR07>		 at scbus0 target 3 lun 0 (pass3,da3)
<LSILOGIC SASX36 A1 7015>		  at scbus0 target 30 lun 0 (pass4,ses0)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 43 lun 0 (da4,pass5)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 44 lun 0 (pass6,da5)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 45 lun 0 (pass7,da6)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 46 lun 0 (pass8,da7)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 47 lun 0 (pass9,da8)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 48 lun 0 (pass10,da9)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 49 lun 0 (pass11,da10)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 50 lun 0 (pass12,da11)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 51 lun 0 (pass13,da12)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 52 lun 0 (pass14,da13)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 53 lun 0 (pass15,da14)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 54 lun 0 (pass16,da15)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 55 lun 0 (pass17,da16)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 56 lun 0 (pass18,da17)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 57 lun 0 (pass19,da18)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 58 lun 0 (pass20,da19)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 59 lun 0 (pass21,da20)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 60 lun 0 (pass22,da21)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 61 lun 0 (pass23,da22)
<ATA Samsung SSD 850 1B6Q>		 at scbus0 target 62 lun 0 (pass24,da23)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 8 lun 0 (pass25,da24)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 9 lun 0 (pass26,da25)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 10 lun 0 (pass27,da26)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 11 lun 0 (pass28,da27)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 12 lun 0 (pass29,da28)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 13 lun 0 (pass30,da29)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 14 lun 0 (pass31,da30)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 15 lun 0 (pass32,da31)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 16 lun 0 (pass33,da32)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 17 lun 0 (pass34,da33)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 18 lun 0 (pass35,da34)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 19 lun 0 (pass36,da35)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 20 lun 0 (pass37,da36)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 21 lun 0 (pass38,da37)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 22 lun 0 (pass39,da38)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 23 lun 0 (pass40,da39)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 24 lun 0 (pass41,da40)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 25 lun 0 (pass42,da41)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 26 lun 0 (pass43,da42)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 27 lun 0 (pass44,da43)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 28 lun 0 (pass45,da44)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 29 lun 0 (pass46,da45)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 30 lun 0 (pass47,da46)
<ATA Samsung SSD 850 2B6Q>		 at scbus1 target 31 lun 0 (pass48,da47)
<LSI SAS3x40 0601>				 at scbus1 target 32 lun 0 (pass49,ses1)
<WDC WD3000HLFS-01G6U1 04.04V02>   at scbus10 target 0 lun 0 (ada0,pass50)
<WDC WD3000GLFS-01F8U0 03.03V01>   at scbus11 target 0 lun 0 (ada1,pass51)
<WD WD1001FYYG-01SL3 VR07>		 at scbus12 target 0 lun 0 (pass52,da48)
<WD WD1001FYYG-01SL3 VR07>		 at scbus12 target 1 lun 0 (pass53,da49)
<WD WD1001FYYG-01SL3 VR07>		 at scbus12 target 2 lun 0 (pass54,da50)
<WD WD1001FYYG-01SL3 VR07>		 at scbus12 target 3 lun 0 (pass55,da51)
<LSILOGIC SASX36 A1 7015>		  at scbus12 target 32 lun 0 (pass56,ses2)
<ATA ST91000640NS BK03>			at scbus12 target 37 lun 0 (pass57,da52)
<ATA ST91000640NS BK03>			at scbus12 target 38 lun 0 (pass58,da53)
<ATA ST91000640NS SN03>			at scbus12 target 39 lun 0 (pass59,da54)
<ATA WDC WD1000CHTZ-0 6A01>		at scbus12 target 40 lun 0 (pass60,da55)
<ATA WDC WD1000CHTZ-0 6A01>		at scbus12 target 41 lun 0 (da56,pass61)
<ATA WDC WD1000CHTZ-0 6A00>		at scbus12 target 42 lun 0 (pass62,da57)
<ATA WDC WD1000CHTZ-0 6A00>		at scbus12 target 43 lun 0 (pass63,da58)
<ATA WDC WD1000CHTZ-0 6A01>		at scbus12 target 44 lun 0 (pass64,da59)
<ATA WDC WD1000CHTZ-0 6A01>		at scbus12 target 45 lun 0 (pass65,da60)
<ATA WDC WD1000CHTZ-0 6A00>		at scbus12 target 47 lun 0 (pass66,da61)
<ATA INTEL SSDSC2BB24 0355>		at scbus12 target 48 lun 0 (pass67,da62)
<ATA INTEL SSDSC2BB24 0355>		at scbus12 target 49 lun 0 (pass68,da63)
<ATA INTEL SSDSC2BB24 0355>		at scbus12 target 50 lun 0 (pass69,da64)
<ATA INTEL SSDSC2BB24 0355>		at scbus12 target 51 lun 0 (pass70,da65)

Code:

[root@FMTSAN1] ~# glabel status
									  Name  Status  Components
gptid/e9f5b86a-243b-11e6-b474-0cc47a55aaf0	 N/A  da24p2
gptid/ea2a9eb2-243b-11e6-b474-0cc47a55aaf0	 N/A  da25p2
gptid/20279188-8437-11e6-a075-001b21d46820	 N/A  da26p2
gptid/ea9b1d24-243b-11e6-b474-0cc47a55aaf0	 N/A  da27p2
gptid/4d95f4dd-583f-11e6-bd52-001b21d46820	 N/A  da28p2
gptid/45bd1375-55b6-11e6-bd52-001b21d46820	 N/A  da29p2
gptid/eb48048a-243b-11e6-b474-0cc47a55aaf0	 N/A  da30p2
gptid/aa07aa2b-3804-11e6-8133-001b21d46820	 N/A  da31p2
gptid/4875beb5-aaca-11e6-a075-001b21d46820	 N/A  da32p2
gptid/ebec7c32-243b-11e6-b474-0cc47a55aaf0	 N/A  da33p2
gptid/ec292391-243b-11e6-b474-0cc47a55aaf0	 N/A  da34p2
gptid/ec5d46d7-243b-11e6-b474-0cc47a55aaf0	 N/A  da35p2
gptid/bc628eef-35f3-11e6-8133-001b21d46820	 N/A  da36p2
gptid/06c0b66d-ad08-11e6-a075-001b21d46820	 N/A  da37p2
gptid/ed100aeb-243b-11e6-b474-0cc47a55aaf0	 N/A  da38p2
gptid/ed4526ba-243b-11e6-b474-0cc47a55aaf0	 N/A  da39p2
gptid/ed82b818-243b-11e6-b474-0cc47a55aaf0	 N/A  da40p2
gptid/616f81b7-aacd-11e6-a075-001b21d46820	 N/A  da41p2
gptid/5c642ecf-600a-11e6-a116-001b21d46820	 N/A  da42p2
gptid/ee29c100-243b-11e6-b474-0cc47a55aaf0	 N/A  da43p2
gptid/76f25f22-aad0-11e6-a075-001b21d46820	 N/A  da44p2
gptid/8ac780e7-89ae-11e6-a075-001b21d46820	 N/A  da45p2
gptid/cdfe712f-5dac-11e6-bd52-001b21d46820	 N/A  da46p2
gptid/e6388c64-aad2-11e6-a075-001b21d46820	 N/A  da47p2
gptid/84c3f5ec-c48c-11e6-b34c-001b21d46820	 N/A  ada0p1
gptid/850631f9-c48c-11e6-b34c-001b21d46820	 N/A  ada1p1
gptid/a11e360d-8328-11e7-8a76-001b21d46820	 N/A  da48p2
gptid/a28b6dd1-8328-11e7-8a76-001b21d46820	 N/A  da49p2
gptid/a43bad4b-8328-11e7-8a76-001b21d46820	 N/A  da50p2
gptid/a59bcaaa-8328-11e7-8a76-001b21d46820	 N/A  da51p2
gptid/a708f471-8328-11e7-8a76-001b21d46820	 N/A  da52p2
gptid/a81376aa-8328-11e7-8a76-001b21d46820	 N/A  da53p2
gptid/a9710789-8328-11e7-8a76-001b21d46820	 N/A  da54p2
gptid/aa4f7f5b-8328-11e7-8a76-001b21d46820	 N/A  da55p2
gptid/ac5086f4-8328-11e7-8a76-001b21d46820	 N/A  da57p2
gptid/ad6cc662-8328-11e7-8a76-001b21d46820	 N/A  da58p2
gptid/ae4e2e72-8328-11e7-8a76-001b21d46820	 N/A  da59p2
gptid/af6c476b-8328-11e7-8a76-001b21d46820	 N/A  da60p2
gptid/b0430f54-8328-11e7-8a76-001b21d46820	 N/A  da61p2
gptid/b1cdf512-8328-11e7-8a76-001b21d46820	 N/A  da0p2
gptid/b33834f7-8328-11e7-8a76-001b21d46820	 N/A  da1p2
gptid/b4de3e6d-8328-11e7-8a76-001b21d46820	 N/A  da2p2
gptid/b63ac36d-8328-11e7-8a76-001b21d46820	 N/A  da3p2
gptid/b766dda8-8328-11e7-8a76-001b21d46820	 N/A  da62p2
gptid/b82086fe-8328-11e7-8a76-001b21d46820	 N/A  da63p2
gptid/b9131e10-8328-11e7-8a76-001b21d46820	 N/A  da64p1
gptid/b9b851d4-8328-11e7-8a76-001b21d46820	 N/A  da65p1
gptid/ef1c19f0-832a-11e7-8a76-001b21d46820	 N/A  da5p2
gptid/f01b7ff0-832a-11e7-8a76-001b21d46820	 N/A  da6p2
gptid/f0d3330c-832a-11e7-8a76-001b21d46820	 N/A  da7p2
gptid/f1d335b8-832a-11e7-8a76-001b21d46820	 N/A  da8p2
gptid/f28cdf8d-832a-11e7-8a76-001b21d46820	 N/A  da9p2
gptid/f38fe494-832a-11e7-8a76-001b21d46820	 N/A  da10p2
gptid/f44d4dc8-832a-11e7-8a76-001b21d46820	 N/A  da11p2
gptid/f5517a2d-832a-11e7-8a76-001b21d46820	 N/A  da12p2
gptid/f60a5300-832a-11e7-8a76-001b21d46820	 N/A  da13p2
gptid/f70d14dc-832a-11e7-8a76-001b21d46820	 N/A  da14p2
gptid/f7c8d96a-832a-11e7-8a76-001b21d46820	 N/A  da15p2
gptid/f8cc8471-832a-11e7-8a76-001b21d46820	 N/A  da16p2
gptid/f988e3d9-832a-11e7-8a76-001b21d46820	 N/A  da17p2
gptid/fa88c20a-832a-11e7-8a76-001b21d46820	 N/A  da18p2
gptid/fb44a1cf-832a-11e7-8a76-001b21d46820	 N/A  da19p2
gptid/fc48934f-832a-11e7-8a76-001b21d46820	 N/A  da20p2
gptid/fd021d5e-832a-11e7-8a76-001b21d46820	 N/A  da21p2
gptid/fe063f0b-832a-11e7-8a76-001b21d46820	 N/A  da22p2
gptid/fec13ddd-832a-11e7-8a76-001b21d46820	 N/A  da23p2
gptid/ee4c98e2-832a-11e7-8a76-001b21d46820	 N/A  da4p1
gptid/ee61aa6d-832a-11e7-8a76-001b21d46820	 N/A  da4p2
gptid/ab5b5174-8328-11e7-8a76-001b21d46820	 N/A  da56p1
gptid/ab72b204-8328-11e7-8a76-001b21d46820	 N/A  da56p2
[root@FMTSAN1] ~#

 

[nightshade00013]

So from what I gather this is all being done on a hot system. Not being powered down. Sooooo, I am going to just let someone else's post from awhile back speak for this one.

https://forums.freenas.org/index.php?threads/hotswap-issue.19019/#post-105845

 

[mikesoultanian]

So from what I gather this is all being done on a hot system. Not being powered down.

Yes, this is all via hot-swapping. In the past we've been hot-swapping, and when I recently reconfigured the storage, I was hot swapping, but now I'm getting some inconsistent behavior as described in my first post. So is hot-swap incompatibility an intermittent type of thing, or is it something that just should or shouldn't work? In that post you linked to, the original poster was having some different errors than I was getting - I'm not getting any notification that the drive is being inserted, but when I move a few bays over in the same enclosure, it works just fine. Is there some kind queue or something that keeps track of drives that are inserted, or does FreeNAS disable bays for one reason or another? If so is there some way to see that list?

Is there a definitive way to figure out if an HBA supports hot-swapping with FreeBSD?

thanks!

 

[Ericloewe]

HBA and backplane firmware might need an update.

Speaking of which, what HBA is in use here?

 

[Ericloewe]

This caught my eye: <LSI SAS3x40 0601> - that's different from the other two expanders listed, which both call themselves <LSILOGIC SASX36 A1 7015>

Are they supposed to be different?

 

[mikesoultanian]

HBA and backplane firmware might need an update.

Speaking of which, what HBA is in use here?

So you might be onto something here - I was finally able to reboot and I pulled up the config utility during boot and listed all the drives - it wouldn't work with the new Samsung 850 but it would with the other older 850 drives that I have and also with older WD HDD - I had my other admin in there with me and he mentioned that maybe the HBA firmware doesn't support the newer drives - it's dated 2013 so that might be a clue ;) I'm gonna update the firmware and see if that fixes it.

The two bottom JBODs that are not working as expected are connected to two LSI SAS9207-4i4e HBAs. The top JBOD (which is working fine) is connected to a newer Avago SAS9300-8e.

stay tuned...

 

[mikesoultanian]

Continuing on this, the multiple chassis that hold all the drives are different - two out of the three are the same, the other one is different (newer) - the newer one is already filled with SSDs. After a lot of testing while viewing the HBA drive display, it looks like some of the drive bays aren't working right, so that would be in line with what @Stux said in that there might be faulty bays. It looks like it will accept an HDD, but it won't accept the SSD, hence the reason it was working before, but now that I'm trying to populate it with SSDs, it's not working right.

I found another newer chassis and that one works, but it has the wrong connectors to connect to the existing HBA so I had to daisy chain off of the other chassis until I purchase another 9300-8e... so maybe it's a firmware issue on the older chassis or maybe it's just bad? I might also try sourcing another backplane on ebay as I'm not sure if these have a firmware that can be updated, but I'll double check with supermicro.

thanks!

 

[Stux]

Can sometimes use cabling to convert from one type of sas connector to another. What was the connector?

 

[Ericloewe]

I found another newer chassis and that one works, but it has the wrong connectors to connect to the existing HBA

That's not a problem, you can just get an SFF-8088 to SFF-8644 cable. You'll obviously be limited by the slower SAS2 device, but it'll work.

so maybe it's a firmware issue on the older chassis or maybe it's just bad? I might also try sourcing another backplane on ebay as I'm not sure if these have a firmware that can be updated, but I'll double check with supermicro.

Definitely try to update them to the latest version.

 

[mikesoultanian]

Can sometimes use cabling to convert from one type of sas connector to another. What was the connector?

I could convert, but I also want to be careful about saturating the link with 24 SSDs so I don't mind upgrading to the newer card.

I'll still have the older JBOD with the older card, but it'll be hosting an HDD pool so I'm not quite as worried about speed on that one as it'll be hosting VMs that don't need to be hosted on fast storage.


I'm going to contact SuperMicro shortly to see if they offer an upgrade. But even then, I probably still won't use it with the SSDs anyways. Maybe if I find another 24 HDDs lying around I'll bring it back, or maybe I'll add it to our backup storage.

Thanks!