Drive replacement, double checking

[captaintofuburger]

I haven't had to do this for a long time, and my memory is fuzzy on what I need to do. I have a dying disk da3 on a zfs array. I have a few questions so I don't mess this up replacing it.

First off, the disk is still online. But I don't know how to go about finding the physical disk. I have a c2100 packed full of drives with no room to spare. I also may or may not have my backplane plugged in backward. How do I see what actual slot the drive is in? Outside of shutting it down and pulling each drive to look at the serial.

Second, I see in the GUI there's an option to replace the drive under the volume status. The problem is I don't have a free drive to replace it with. I have to pull the dead one, put in the replacement, and rebuild it. So how do I go about doing this?

I could shut the system down for some time, but it wouldn't be ideal, but also wouldn't be the end of the world.

I'm just hesitant to do what I think I should do, just because I have shot myself in the foot a few times in the past assuming I knew what I was doing.

I'm assuming I should just set the drive to offline in freenas, shut the box down, pull the drive, replace it, go back in to the GUI, click replace disk, pick da3 again, and rebuild? Just want to be sure I'm doing it correctly.

 

[Chris Moore]

I have a c2100

What is that? Please give a full breakdown of your hardware per the forum rules:

Updated Forum Rules 4/11/17
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

I also may or may not have my backplane plugged in backward.

How ?? I don't understand what you are trying to say with that.

I have to pull the dead one, put in the replacement, and rebuild it. So how do I go about doing this?

Exactly what you said, after you identify the correct drive to pull, "I have to pull the dead one, put in the replacement, and rebuild it." That is exactly what you have got to do. Identify the correct drive. Remove it from the chassis. Put the replacement in place of it. Tell FreeNAS to use that drive to rebuild the missing drive. The hard part (in your instance) appears to be identifying the correct drive to pull.

shut the box down,

If you have a chassis that supports hot-swap, there is no need to shutdown. I have replaced dozens of drives without ever downing the server.

I'm assuming I should just set the drive to offline in freenas

That is the safe way to ensure that the process goes smoothly.

You might want to review the resource:

Replacing a failed/failing disk
https://forums.freenas.org/index.php?resources/replacing-a-failed-failing-disk.75/

 

[Chris Moore]

PS.

But I don't know how to go about finding the physical disk. I have a c2100 packed full of drives with no room to spare. I also may or may not have my backplane plugged in backward. How do I see what actual slot the drive is in?

If your backplane is in backward, this may not help, but if it is not in backward, you should be able to type the command diskinfo -v /dev/da3 and part of the output will be the slot in the backplane. That works for me, but I can't guarantee it will work for you.

Edit: 5 July 2018 - to add more clarification:
Just for future reference, if someone finds this later, on my 48 bay server, with 2 x SAS expander backplanes, this is what I get using the command above:

Code:

root@Emily-NAS:~ # diskinfo -v /dev/da3
/dev/da3
		512			 # sectorsize
		100030242816	# mediasize in bytes (93G)
		195371568	   # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		12161		   # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		HITACHI HUSSL4010BSS600 # Disk descr.
		XSVMYWMA		# Disk ident.
		id1,enc@n5001c4500095debd/type@0/slot@3/elmdesc@Slot_002		# Physical path
		Not_Zoned	   # Zone Mode

Code:

root@Emily-NAS:~ # diskinfo -v /dev/da31
/dev/da31
		512			 # sectorsize
		2000398934016   # mediasize in bytes (1.8T)
		3907029168	  # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		243201		  # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA ST2000DM001-1ER1	# Disk descr.
		W4Z29GQY		# Disk ident.
		id1,enc@n5001c4500095debd/type@0/slot@b/elmdesc@Slot_010		# Physical path
		Not_Zoned	   # Zone Mode

Code:

root@Emily-NAS:~ # diskinfo -v /dev/da45
/dev/da45
		512			 # sectorsize
		4000787030016   # mediasize in bytes (3.6T)
		7814037168	  # mediasize in sectors
		4096			# stripesize
		0			   # stripeoffset
		486401		  # Cylinders according to firmware.
		255			 # Heads according to firmware.
		63			  # Sectors according to firmware.
		ATA ST4000DM000-1F21	# Disk descr.
		Z300WTNP		# Disk ident.
		id1,enc@n5001c4500095debd/type@0/slot@5/elmdesc@Slot_004		# Physical path
		Not_Zoned	   # Zone Mode

So, if you look at "Physical path", it gives some information that might be able to be interpreted to give guidance but it is pretty cryptic. Sorry.

If you have a SAS expander backplane, you can use the command sesutil status to get the following status from the backplane (or backplanes in my case):

Code:

root@Emily-NAS:~ # sesutil status
ses0: OK
ses1: OK

You can use the command sesutil map to get the following list, except mine is long, so I am just showing part of it as an example. Also, you will want to SSH into the system with something like PuTTY so you can scroll back and read the listing.

Code:

ses0:
		Enclosure Name: 1401-00 80H10024001A0 1401
		Enclosure ID: 5001c4500095d8bf
		Element 0, Type: Array Device Slot
				Status: Unsupported (0x00 0x00 0x00 0x00)
				Description: ArrayDevicesInSubEnclsr0
		Element 1, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 000
				Device Names: da36,pass38
		Element 2, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 001
				Device Names: da37,pass39
		Element 3, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 002
				Device Names: da2,pass2
		Element 4, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 003
				Device Names: da38,pass40
		Element 5, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 004
				Device Names: da39,pass41
		Element 6, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 005
				Device Names: da0,pass0
		Element 7, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 006
				Device Names: da40,pass42

My second backplane gives similar results, but notice how the slot number start over but the da numbers are different:

Code:

ses1:
		Enclosure Name: 1401-00 80H10024001A0 1401
		Enclosure ID: 5001c4500095debf
		Element 0, Type: Array Device Slot
				Status: Unsupported (0x00 0x00 0x00 0x00)
				Description: ArrayDevicesInSubEnclsr0
		Element 1, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 000
				Device Names: da42,pass44
		Element 2, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 001
				Device Names: da43,pass45
		Element 3, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 002
				Device Names: da3,pass3
		Element 4, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 003
				Device Names: da44,pass46
		Element 5, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 004
				Device Names: da45,pass47
		Element 6, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 005
				Device Names: da1,pass1
		Element 7, Type: Array Device Slot
				Status: OK (0x01 0x00 0x00 0x00)
				Description: Slot 006
				Device Names: da46,pass48

 

[Redcoat]

That works for me, but I can't guarantee it will work for you.

I just tried that out on my Dell FS12-TY/C2100 (backplane not in backwards) and it didn't give me any slot info.

 

[Chris Moore]

I just tried that out on my Dell FS12-TY/C2100 (backplane not in backwards) and it didn't give me any slot info.

The other command is dependent on the SAS controller.

If it is a SAS2 controller, this should work: sas2ircu 0 display

If it is a SAS3 controller, this should work: sas3ircu 0 display

The 0 (zero) tells it which controller to use, so if you have two controllers, you might need controller 1 instead.

That lists everything...

 

[captaintofuburger]

How ?? I don't understand what you are trying to say with that.

Exactly what you said, after you identify the correct drive to pull, "I have to pull the dead one, put in the replacement, and rebuild it." That is exactly what you have got to do. Identify the correct drive. Remove it from the chassis. Put the replacement in place of it. Tell FreeNAS to use that drive to rebuild the missing drive. The hard part (in your instance) appears to be identifying the correct drive to pull.

If you have a chassis that supports hot-swap, there is no need to shutdown. I have replaced dozens of drives without ever downing the server.

I think based on what you said I might use this opportunity to shut down label my drives. I do have the ability to hotswap, but with the problem of finding the drive it might be easier to shutdown. All I was getting at with the backplane is if I were to search for the slot in software it might not correlate correctly with the physical placement of the drive.

 

[captaintofuburger]

The other command is dependent on the SAS controller.

If it is a SAS2 controller, this should work: sas2ircu 0 display

This does work, but I'm pretty sure I swapped my backplane around, so I'm not 100% certain if those slot numbers will line up or be flipped around. I think I'm just going to be stuck for a shutdown to double check. I would rather do that instead of guessing wrong.

 

[Chris Moore]

This does work, but I'm pretty sure I swapped my backplane around, so I'm not 100% certain if those slot numbers will line up or be flipped around. I think I'm just going to be stuck for a shutdown to double check. I would rather do that instead of guessing wrong.

Probably safest.
I have two backplanes in my chassis at home so slot number are duplicated and I have to be cautious about which "enclosure" is listed for the slot number or I will be looking at the wrong slot. I printed serial number labels for all my drive trays. No question that way.

 

[captaintofuburger]

Probably safest.
I have two backplanes in my chassis at home so slot number are duplicated and I have to be cautious about which "enclosure" is listed for the slot number or I will be looking at the wrong slot. I printed serial number labels for all my drive trays. No question that way.

Yeah I have a feeling this is probably the safest way to go and will also reduce headaches down the road.