Member disk missing when upgrading drives

[Yennz05]

I am a long-time user of FreeNas/TrueNas but my knowledge of details and troubleshooting is very limited.

I am trying to upgrade my main pool to larger drives. I currently have 4x 4TB WD Blue (WD40EZRZ) drives (RAIDZ1) and I want to replace them with 4x 12TB WD Red Plus (WD120EFBX) drives. I checked and these are CMR drives. I followed the instructions to offline one of the drives, shut down the computer, install the new drive, reboot, and select "Replace." The Member Disk dropdown is empty, so what am I missing here? Hardware details are in my signature.

As a side note, I have unreadable sectors on my boot drive and I just bought a new drive to replace it. I have not installed the new drive yet though. Could this have an effect on the main pool drive upgrades?

Thanks in advance!

 

[sretalla]

You may need to wipe the replacement disk first before it will appear.

As a side note, I have unreadable sectors on my boot drive and I just bought a new drive to replace it. I have not installed the new drive yet though. Could this have an effect on the main pool drive upgrades?

It shouldn't.

 

[Yennz05]

I went to Storage > disks and the new replacement disk (12TB) doesn't show up there for me to wipe it. If I wiped it externally with another computer would that accomplish the same thing?

 

[sretalla]

I went to Storage > disks and the new replacement disk (12TB) doesn't show up there for me to wipe it. If I wiped it externally with another computer would that accomplish the same thing?

I suspect not.

Do you see it in camcontrol devlist or can you see anything about it in dmesg output?

 

[Yennz05]

Don't see it in camcontrol devlist.

dmesg doesn't show it either, only the same drives as camcontrol.
Also, in dmesg I have a bunch of CAM status: ATA Status errors for ada0 which is my boot drive. Also, ada3 has CAM status: Uncorrectable parity/CRC error. This is one of my main pool drives.

 

[Yennz05]

I suspect not.

Do you see it in camcontrol devlist or can you see anything about it in dmesg output?

I had initially started with ada4 and that didn't work so I tried ada3 and the drop-down now shows ada3 as a replace option. However, once I select ada3 I can't select the "Replace Disk" button. It turns darker grey but nothing happens when I click it. Any ideas why it won't proceed?

Also, now the new drive does show up in camcontrol devlist, so that is good.

 

[sretalla]

I think we need to see more of what's going on... I can't make any sense out of what you're saying is happening.

Can you show some screen shots?

 

[Yennz05]

Sorry for the confusion, I appreciate the help.

In the Pool Status menu, you can see that ada3 is offline and I put in the new 12TB drive. I did wipe the drive as well before trying replace.

I select Replace on the new drive and the "Replace Disk" button is greyed out.

After selecting ada3 as the member disk you can see that the "Replace Disk" button brightens but I can't click it.

 

[sretalla]

OK, that seems clear enough now... I think you updated your signature too, which is the key missing information... 13.0...

You need to look in the release notes for 13.0, which mention that drive replacement doesn't work and offer a CLI workaround if you can't wait for 13.0-U1 to arrive.

/core/gettingstarted/corereleasenotes/

 

[Yennz05]

OK, that seems clear enough now... I think you updated your signature too, which is the key missing information... 13.0...

You need to look in the release notes for 13.0, which mention that drive replacement doesn't work and offer a CLI workaround if you can't wait for 13.0-U1 to arrive.

/core/gettingstarted/corereleasenotes/

What if I just reinstalled the boot drive with 12.0? Wouldn't that work too?

 

[sretalla]

What if I just reinstalled the boot drive with 12.0? Wouldn't that work too?

You could set your boot environment back to 12.0 (assuming you didn't upgrade your pool already after installing 13.0)... the workaround is quicker than doing that and doesn't require a reboot... the choice is yours.

 

[Yennz05]

You could set your boot environment back to 12.0 (assuming you didn't upgrade your pool already after installing 13.0)... the workaround is quicker than doing that and doesn't require a reboot... the choice is yours.

I did not upgrade the pool. I will try installing 12.0 on the boot disk.

 

[sretalla]

Well, if your pool has the draid feature, it's 13 only for you... if not, you can still run 12

zpool get feature@draid

 

[Yennz05]

Well, if your pool has the draid feature, it's 13 only for you... if not, you can still run 12

zpool get feature@draid

The draid feature was not enabled so I was able to install 12.0 on the boot drive and upload the config file.

I added the new drive again and now replace IS WORKING! It is resilvering as we speak.

Thanks for the help! Hopefully, I won't run into any other problems as I replace the other drives.

 

[Yennz05]

Well, if your pool has the draid feature, it's 13 only for you... if not, you can still run 12

zpool get feature@draid

My first drive ada3 was completed successfully. Then I couldn't get the second one to work. The drive wouldn't show up and member disk was blank again. I tried a different new hard drive in the same location and now it is working. Does this mean that hard drive is faulty and I should return it for a new one?

 

[sretalla]

The drive wouldn't show up and member disk was blank again. I tried a different new hard drive in the same location and now it is working. Does this mean that hard drive is faulty and I should return it for a new one?

No... sometimes the UI is a bit picky about seeing drives that have just been messed with. Sometimes a forced refresh of the browser can fix it up, other times a reboot.

 

[Yennz05]

No... sometimes the UI is a bit picky about seeing drives that have just been messed with. Sometimes a forced refresh of the browser can fix it up, other times a reboot.

I have two drives complete. I tried a third and the boot stops with this message, "S.M.A.R.T. status bad, backup and replace" on the new drive. I did see that this drive does have physical damage of the metal case so I think it may have been dropped hard. After looking up the error am I safe to assume this drive is bad and I should return it? Or are there other tests I can run?

I also got a CAM status error on the drive in boot. The drive does show in camcontrol devlist.
I get the following in dmesg:
- ada2, CAM status: ATA Status Error
- ada2, FAILED SMART self-check.
- ada2, 8 currently unreadable (pending) sectors
- ada2, Failed SMART usage attribute: 22 Unkown_Attribute.

 

[sretalla]

OK, so you can't even boot with that drive in the system?

That looks pretty much like a confirmation... maybe if you have a USB dock, you can try that drive on a PC and see if it can be subjected to any kind of smart test that would help you to prove to the retailer or manufacturer that it's no good... but the physical damage sounds obvious enough that it would be assumed your claim is right.

I guess the blame of who damaged it would be a question, so watch out for them trying to pin that on you.

 

[Yennz05]

OK, so you can't even boot with that drive in the system?

That looks pretty much like a confirmation... maybe if you have a USB dock, you can try that drive on a PC and see if it can be subjected to any kind of smart test that would help you to prove to the retailer or manufacturer that it's no good... but the physical damage sounds obvious enough that it would be assumed your claim is right.

I guess the blame of who damaged it would be a question, so watch out for them trying to pin that on you.

Correct, it stops with the smart error but then I can continue boot by pressing F1. I am sending the two bad drives back and ordered two new ones. I connected the other one to my PC and it won't initialize in disk management. Hopefully, these next two will work.

 

[Yennz05]

Now ada3, one of the "good" drives is giving me a failed SMART self-check. 2nd time it has done this. I did a long check and it immediately failed. Clearly I got some sketchy drives from an aftermarket seller. I'm going to pull all of them and order new drives directly from WD.

root@freenas:~ # smartctl -a /dev/ada3 -t short
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p14 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD120EFBX-68B0EN0
Serial Number: 5QJ1L4KB
LU WWN Device Id: 5 000cca 2b0dce780
Firmware Version: 85.00A85
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 9 09:05:24 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 73) The previous self-test completed having
a test element that failed and the test
element that failed is not known.
Total time to complete Offline
data collection: ( 87) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1299) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0004 127 127 054 Old_age Offline - 112
3 Spin_Up_Time 0x0007 167 167 024 Pre-fail Always - 379 (Average 411)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 22
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000a 100 100 067 Old_age Always - 0
8 Seek_Time_Performance 0x0004 128 128 020 Old_age Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 261
10 Spin_Retry_Count 0x0012 100 100 060 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22
22 Unknown_Attribute 0x0023 010 010 025 Pre-fail Always FAILING_NOW 10
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 40
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 40
194 Temperature_Celsius 0x0002 209 209 000 Old_age Always - 31 (Min/Max 25/33)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: unknown failure 90% 261 0
# 2 Short offline Completed: unknown failure 90% 261 0

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-linemode".