Drive replacement issue?

Status
Not open for further replies.

Pointeo13

Explorer
Joined
Apr 18, 2014
Messages
86
Build: FreeNas-11.1-U6

Checking to see if anyone else has this issue, I’m in the process of changing out 18 3TB drives to 8TB drives. I currently have 13 vdev’s with six drives in each vdev setup as Raidz2 and as of right now I’m working with three of the vdev’s to replace all 18 drives.

Here is my issue, if I put the drives da2, da8 and da14 in offline mode, pull the 3TB drives, put in the new 8TB drives in and use the replace option, I will only see two of the three 8TB drives. I have noticed that the drive in vdev Raidz2-0 (da2) will not show up, but the drives in Raidz2-1 (da8) and Raidz2-2 (da14) will show up as an option to use as a replacement disk using the same disk name da8 and da14. So far I’v done this process four times and the only time I saw all three disks show up in the gui was the very first set of disks (da1, da7 and da13.)

Now what I have done, if I run the command sas2ircu 0 display, I see all three 8TB drives just fine. If I run the command smartctl -a /dev/da2 on the drive I currently can’t see in the gui, it says it can’t detect the drive type or something, just gives me an error. If I reboot the freenas server, I now can now see all three new 8TB drives giving them new drive names /dev/da*.

Then I thought, I wonder if it’s having an issue showing all three new drives at one time only during the offline/replacement procedures, so I removed da14 and sure enough I now can see da2 along with da8, so I told freenas to replace the disk with da2, then da8, then put da14 back into the server and now I could replace da14. So quick summary

-First time replacing the disks, it saw all three new 8TB drive da1, da7 and da13 (no issues here)

-Second time replacing the set of disk, it only would see da8 and da14, I had to remove da14 to get it to see da2

-Third time replacing the set of disk, it would only see da9 and da15, if I reboot the server I can now see all three disk da3, da9 and da15 but typically with new disks names /dev/da*.

-Fourth time replacing the set of disks, I put in all three drives, didn’t see da4 so I pulled out da16 and did a replacement on da4 and da10 first and then put da16 back in and did a replacement option.


Raidz2-0
da1
da2
da3
da4
da5
da6

Raidz2-1
da7
da8
da9
da10
da11
da12

Ridz2-2
da13
da14
da15
da16
da17
da18

Edit: Not sure if it matters, but the the drives I'm using are the Western Digital My Book 8TB, I have shucked the drives and they all contain the WD80EZAZ drive aka Ultrastar He10-8 SATA which have the 3.3v reset pins. (Picture is missing three of the drives, waiting on the last three drives today.)

drive_01.png
drive_02.png
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I have done a lot of drive replacements myself and it is always a bit finicky. It just takes a little work, as you have already seen.
 

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
Do the drive names come up with the leading zeros, or is that something that has been manually assigned? In other words, I'd expect to see da1, not da01.

If those have been manually assigned, that could easily cause confusion.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Can we see the output of zpool status?

It should look something like this:
Code:
# zpool status
  pool: mypool
 state: ONLINE
  scan: none requested
config:

	   NAME		STATE	 READ WRITE CKSUM
	   mypool	  ONLINE	   0	 0	 0
		 raidz2-0  ONLINE	   0	 0	 0
		   ada0p3  ONLINE	   0	 0	 0
		   ada1p3  ONLINE	   0	 0	 0
		   ada2p3  ONLINE	   0	 0	 0
		   ada3p3  ONLINE	   0	 0	 0
		   ada4p3  ONLINE	   0	 0	 0
		   ada5p3  ONLINE	   0	 0	 0

errors: No known data errors
 

Pointeo13

Explorer
Joined
Apr 18, 2014
Messages
86
Do the drive names come up with the leading zeros, or is that something that has been manually assigned? In other words, I'd expect to see da1, not da01.

If those have been manually assigned, that could easily cause confusion.

Sorry about that, they have no zero's, i'll fix that in my post, I was referencing my spreadsheets and with my OCD I felt like i needed to see everything daXX, that and the way i sorted things it helped having the 0 in front of the single number.

Can we see the output of zpool status?

Code:
root@:~ # zpool status Raidz_2_SATA
  pool: Raidz_2_SATA
 state: ONLINE
  scan: resilvered 5.58T in 0 days 07:07:54 with 0 errors on Fri Sep 14 06:12:41 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		Raidz_2_SATA									ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/c0d5e371-15a7-11e8-ab59-a0369f484a68  ONLINE	   0	 0	 0
			gptid/731d04df-9fd7-11e7-8ebe-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/25c4fcf6-9283-11e7-8ebe-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e078d3f2-1bee-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/707e7c41-891c-11e7-8d2a-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/dd4e684f-9f14-11e7-8ebe-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/cf88ceb0-165c-11e8-ab59-a0369f484a68  ONLINE	   0	 0	 0
			gptid/08c08b57-1724-11e8-ab59-a0369f484a68  ONLINE	   0	 0	 0
			gptid/f57817a3-17f3-11e8-ab59-a0369f484a68  ONLINE	   0	 0	 0
			gptid/c266fc1a-18c9-11e8-ab59-a0369f484a68  ONLINE	   0	 0	 0
			gptid/a1d42969-193e-11e8-b2bc-a0369f484a68  ONLINE	   0	 0	 0
			gptid/ce9ac33c-19a5-11e8-b2bc-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-2									  ONLINE	   0	 0	 0
			gptid/8a305bc3-b5fd-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/7e1eadaa-b64c-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/fafc4062-b6d7-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/0f2c8e34-b724-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/bdd63c8a-b77e-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/a4449699-b7d1-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-3									  ONLINE	   0	 0	 0
			gptid/58a70f71-b763-11e6-9118-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/8769c8e0-c325-11e6-9118-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e190fb70-c29e-11e6-9118-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/b4313200-c0a8-11e6-9118-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/bfe5c8f8-a796-11e6-9118-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/eb24c94c-1d82-11e8-b2bc-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-4									  ONLINE	   0	 0	 0
			gptid/0ebeae35-d9f5-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/5c9b81c3-e50b-11e7-a57a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/e1738310-daf0-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/e2c55c3f-d8cb-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/62de80ff-d79d-11e7-a3c8-a0369f484a68  ONLINE	   0	 0	 0
			gptid/ca314311-d62e-11e7-a3c8-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-5									  ONLINE	   0	 0	 0
			gptid/c55c76b7-d9f4-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/beacf4e9-dbf8-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/6297aa25-daf0-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/9f689504-d8cb-11e7-b15a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/722cf68d-d71e-11e7-a3c8-a0369f484a68  ONLINE	   0	 0	 0
			gptid/85d6f320-d62e-11e7-a3c8-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-6									  ONLINE	   0	 0	 0
			gptid/2a5228d8-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/2b8c0b23-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/2cc53e86-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/2e0403b9-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/2f3e74bf-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3076d876-8a12-11e5-9905-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-7									  ONLINE	   0	 0	 0
			gptid/70c3d74e-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/72c3e734-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/74c1d3e1-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/76a320ea-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/787028e4-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/7a4744e8-b33a-11e5-83e2-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-8									  ONLINE	   0	 0	 0
			gptid/df118278-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e00ae5d7-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e10dc1c8-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e2121190-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e306c84f-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/e3f9b4dc-ed2e-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-9									  ONLINE	   0	 0	 0
			gptid/456df18a-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/465657e6-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/47418cad-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/4826457b-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/490ee34f-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/4a01eb1f-ed2f-11e5-aaaf-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-10									 ONLINE	   0	 0	 0
			gptid/39be88a8-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3b34d341-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3c5d28df-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3d7920e3-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3edca0cf-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
			gptid/3ff30819-1beb-11e7-a096-3440b5b0233c  ONLINE	   0	 0	 0
		  raidz2-11									 ONLINE	   0	 0	 0
			gptid/fa7ec530-b64c-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/6e60fc6e-b6d8-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/e2265afd-b77f-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b8ed3d10-b724-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/162c05d7-b7d3-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/f4ed5ea8-b5fc-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-12									 ONLINE	   0	 0	 0
			gptid/71c4646d-b77f-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/444ad7d3-b725-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/f952906a-b6d8-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
			gptid/85674958-b64d-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/8788fd86-b5fc-11e8-bc7a-a0369f484a68  ONLINE	   0	 0	 0
			gptid/1d9eb7c6-b7d2-11e8-b121-a0369f484a68  ONLINE	   0	 0	 0
		  raidz2-13									 ONLINE	   0	 0	 0
			gptid/af9f90f6-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b12bb3dd-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b2c15ad1-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b44ffec6-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b5ebeadd-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0
			gptid/b7aa1120-9069-11e8-b9d5-a0369f484a68  ONLINE	   0	 0	 0

errors: No known data errors
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
with my OCD I felt like i needed to see everything daXX
Understandable. I have done that. Just wanted to ensure what it was the computer is seeing. That looks like 84 drives. What kind of mix of drives do you have? All 8TB models like your first post? What kind of hardware are you using to mount / control all that?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I should say, I've done hard drive swaps like this in version 9.X and never saw an issue, but at the same time, the last time i did drive swaps three at a time was in version 9.X. This would be the first for me to do three drive swaps at a time in version 11.X
The 'finicky' nature that I have seen is at least partly related to hardware. With my old SAS HBA, when I removed a drive and put a different drive in the same slot, the replacement drive would get the same number as the slot. I went to a new SAS controller in a new chassis that has two expander backplanes and now I am getting the next number in line, from the end of the list, even when I put the drive back in the same slot. I recently replaced da17 and the new drive got the number da35. Makes my list look crazy and drives me nuts. It is persistent across reboots too.
 

Pointeo13

Explorer
Joined
Apr 18, 2014
Messages
86
Understandable. I have done that. Just wanted to ensure what it was the computer is seeing. That looks like 84 drives. What kind of mix of drives do you have? All 8TB models like your first post? What kind of hardware are you using to mount / control all that?

Ten of the vdev's are all Toshiba X300 5TB, then the other three vdev's are the new Ultrastar He10-8.

Hardware:
IBM x3650 M4
CPU: 2X - Intel Xeon E5-2680 v2 @ 2.70Ghz
Memory: 384GB
Network: 6 X 10Gb ports
Three LSI (I.T Mode) cards that have external sas ports that handle the three 36bay Supermicro CSE-847 using the backplane BPN-SAS2-846EL

Then another set of storage that has 24 SSD for the virtual machines with a dedicated LSI card and it's own set of backplane.

The 'finicky' nature that I have seen is at least partly related to hardware. With my old SAS HBA, when I removed a drive and put a different drive in the same slot, the replacement drive would get the same number as the slot. I went to a new SAS controller in a new chassis that has two expander backplanes and now I am getting the next number in line, from the end of the list, even when I put the drive back in the same slot. I recently replaced da17 and the new drive got the number da35. Makes my list look crazy and drives me nuts. It is persistent across reboots too.

I have to think about it now, trying to look back, but you know what, maybe I didn't do three drive replacement before, trying to verify now, mind is foggy, I'v done a ton of 1-2 drive replacements so for now I take my statement back and removed it.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

Pointeo13

Explorer
Joined
Apr 18, 2014
Messages
86
I bet that you could upgrade those system boards with these V2 processors and save a little on electricity and you might even get a little more performance.
https://www.ebay.com/itm/Intel-Xeon-E5-2680V2-SR1A6-2-80GHZ/142930742553
How heavily is the system used?

Made the correction, they are the v2 processor, freenas/OS just shows a 0 instead of v2

cpu.PNG


The CPU's really don't do anything, it's pretty much a waste, FreeNas is here to only to serve the storage through iscsi and SMB. Everything else is done on my other four IBM X3650 M4's through vmware.

cpu_info.PNG
 
Last edited:
Status
Not open for further replies.
Top