I've been a using FreeNAS for a little while now, embarrassingly this is my first post and it’s asking for help!
Hardware: HPE ProLiant MicroServer Gen8
Intel® Celeron® G1610T (2.3Hz/2-core/2MB/35W) Processor
2x 4GB-DDR3-ECC-Memory-RAM
4 x WD Red 2TB 3.5" SATA
Boot: Dual - Kingston 16GB DataTraveler SE9 USB Flash Drive
FreeNAS-11.1-U6
My setup has been 4 x WD Red 2TB 3.5" SATA drives as my pool in a raidz1-0 (which I have a couple of backups of) but not of the five jails or couple of VM's.
Current zpool status:
A week or two ago ada1 gave a few unreadable (pending) sectors, not surprising due to the age, I took the opportunity to order a replacement 4TB WD Red 3.5" drive and start the replacement and expansion on the existing pool from 2TB to 4TB disks.
This is the long smart test output before the disk was replaced.
Prior to my replacement 4TB arriving and swapping out the faulty ada1 it was inevitable that ada2 showed a unreadable (pending) sectors error. It was only a couple initially.
This is the long smart test of ada2.
I started the process of replacing ada1 via the GUI and following the manual, all went as planned and the resilver was under way.
During the resilver the errors from ada2 increased, there were a few tens of read errors but a few hundred checksum errors.
After the resilver had completed all data, jail’s and VM’s seemed okay. I was concerned about the checksum errors, reading around I saw the mention of running a scrub on the pool, once this was completed the pool started to resilver again. Once again this completed and there were a few more errors on ada2. I left the setup for 24 hours then rebooted the FreeNAS, on boot up the resilver started once again. I took the opportunity to shut down the FreeNAS and check as best as I could the internal cabling, restated the disks & RAM then powered it back up to allow the resilver to continue.
This is the current error on ada2: Device: /dev/ada2, 48 Currently unreadable (pending) sectors
After running zpool status -v
I saw that there were three files which had permanent errors, as these were not important I removed them, this are now listed as:
errors: Permanent errors have been detected in the following files:
colossus/pyramid:<0x218b5b>
colossus/pyramid:<0x1b18a7>
colossus/pyramid:<0x1727aa>
Should these disappear on reboot or after the current resilver has completed? Could these have been causing the resilver not to complete as they were corrupt?
I have a second 4TB disk ready to replace ada2 but I’d like some guidance if at all possible as to when this would be best to do, I feel the resilver may not be completing and I could introduce data lose if not done correctly?
Time for RAIDZ2 now i have bigger disks?! Data wise i'm okay, it's the jail's and VM's i'd need to work out how best to migrate.
Thanks
Hardware: HPE ProLiant MicroServer Gen8
Intel® Celeron® G1610T (2.3Hz/2-core/2MB/35W) Processor
2x 4GB-DDR3-ECC-Memory-RAM
4 x WD Red 2TB 3.5" SATA
Boot: Dual - Kingston 16GB DataTraveler SE9 USB Flash Drive
FreeNAS-11.1-U6
My setup has been 4 x WD Red 2TB 3.5" SATA drives as my pool in a raidz1-0 (which I have a couple of backups of) but not of the five jails or couple of VM's.
Current zpool status:
Code:
[root@ds_fnas /]# zpool status -v
pool: colossus
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Apr 29 18:46:43 2019
2.03T scanned at 422M/s, 1.45T issued at 301M/s, 4.96T total
353G resilvered, 29.16% done, 0 days 03:24:00 to go
config:
NAME STATE READ WRITE CKSUM
colossus ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/cce2612e-cd4d-11e7-b4fd-00fd45fd9c44 ONLINE 0 0 0
gptid/200908ae-6906-11e9-bf3b-00fd45fd9c44 ONLINE 0 0 0 (resilvering)
gptid/cc4efaf1-ceab-11e7-9303-00fd45fd9c44 ONLINE 0 0 0
gptid/6fff97c6-ceb0-11e7-895d-00fd45fd9c44 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
colossus/pyramid:<0x218b5b>
colossus/pyramid:<0x1b18a7>
colossus/pyramid:<0x1727aa>
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:55 with 0 errors on Mon Apr 22 03:45:55 2019
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da0p2 ONLINE 0 0 0
da1p2 ONLINE 0 0 0
errors: No known data errors
A week or two ago ada1 gave a few unreadable (pending) sectors, not surprising due to the age, I took the opportunity to order a replacement 4TB WD Red 3.5" drive and start the replacement and expansion on the existing pool from 2TB to 4TB disks.
This is the long smart test output before the disk was replaced.
Code:
smartctl -a /dev/ada1 | more smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD20EFRX-68EUZN0 Serial Number: WD-WCC4M7RFR068 LU WWN Device Id: 5 0014ee 264213f62 Firmware Version: 82.00A82 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Apr 24 22:01:21 2019 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (26760) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine rrecommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 270) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 86 3 Spin_Up_Time 0x0027 172 170 021 Pre-fail Always - 4400 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 18 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 12451 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 18 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 249 194 Temperature_Celsius 0x0022 120 102 000 Old_age Always - 27 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 2 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 2 SMART Error Log Version: 1 ATA Error Count: 3 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 3 occurred at disk power-on lifetime: 11688 hours (487 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 80 b8 96 41 Error: UNC at LBA = 0x0196b880 = 26654848 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 28 b8 96 41 08 38d+13:40:21.647 READ DMA c8 00 00 28 b8 96 41 08 38d+13:40:18.251 READ DMA c8 00 00 28 b8 96 41 08 38d+13:40:14.873 READ DMA Error 2 occurred at disk power-on lifetime: 11688 hours (487 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 80 b8 96 41 Error: UNC at LBA = 0x0196b880 = 26654848 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 28 b8 96 41 08 38d+13:40:18.251 READ DMA c8 00 00 28 b8 96 41 08 38d+13:40:14.873 READ DMA Error 1 occurred at disk power-on lifetime: 11688 hours (487 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 80 b8 96 41 Error: UNC at LBA = 0x0196b880 = 26654848 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 28 b8 96 41 08 38d+13:40:14.873 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 12451 26641600 # 2 Short offline Completed without error 00% 12434 - # 3 Extended offline Completed: read failure 90% 12403 26644912 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. [root@ds_fnas ~]#
Prior to my replacement 4TB arriving and swapping out the faulty ada1 it was inevitable that ada2 showed a unreadable (pending) sectors error. It was only a couple initially.
This is the long smart test of ada2.
Code:
smartctl -a /dev/ada2 | more smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD20EFRX-68EUZN0 Serial Number: WD-WCC4M5YA2KX7 LU WWN Device Id: 5 0014ee 2642140ed Firmware Version: 82.00A82 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Wed Apr 24 22:05:15 2019 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (27720) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 280) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 154 3 Spin_Up_Time 0x0027 175 173 021 Pre-fail Always - 4241 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 38 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 12543 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 280 194 Temperature_Celsius 0x0022 120 103 000 Old_age Always - 27 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 3 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 15 SMART Error Log Version: 1 ATA Error Count: 7 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 7 occurred at disk power-on lifetime: 11781 hours (490 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 20 26 a1 40 Error: UNC at LBA = 0x00a12620 = 10561056 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 48 25 a1 40 08 38d+16:07:59.958 READ DMA c8 00 00 48 25 a1 40 08 38d+16:07:56.586 READ DMA Error 6 occurred at disk power-on lifetime: 11781 hours (490 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 20 26 a1 40 Error: UNC at LBA = 0x00a12620 = 10561056 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 48 25 a1 40 08 38d+16:07:56.586 READ DMA Error 5 occurred at disk power-on lifetime: 7681 hours (320 days + 1 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 30 02 a1 40 Error: UNC at LBA = 0x00a10230 = 10551856 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 60 01 a1 40 08 7d+20:46:24.631 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:20.657 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:16.687 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:12.709 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:08.612 READ DMA Error 4 occurred at disk power-on lifetime: 7681 hours (320 days + 1 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 30 02 a1 40 Error: UNC at LBA = 0x00a10230 = 10551856 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 60 01 a1 40 08 7d+20:46:20.657 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:16.687 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:12.709 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:08.612 READ DMA Error 3 occurred at disk power-on lifetime: 7681 hours (320 days + 1 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 30 02 a1 40 Error: UNC at LBA = 0x00a10230 = 10551856 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 60 01 a1 40 08 7d+20:46:16.687 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:12.709 READ DMA c8 00 00 60 01 a1 40 08 7d+20:46:08.612 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 12524 - # 2 Extended offline Completed: read failure 90% 11894 10545296 # 3 Short offline Aborted by host 90% 90 - # 4 Short offline Aborted by host 90% 88 - # 5 Short offline Aborted by host 90% 68 - # 6 Extended offline Interrupted (host reset) 50% 68 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
I started the process of replacing ada1 via the GUI and following the manual, all went as planned and the resilver was under way.
During the resilver the errors from ada2 increased, there were a few tens of read errors but a few hundred checksum errors.
After the resilver had completed all data, jail’s and VM’s seemed okay. I was concerned about the checksum errors, reading around I saw the mention of running a scrub on the pool, once this was completed the pool started to resilver again. Once again this completed and there were a few more errors on ada2. I left the setup for 24 hours then rebooted the FreeNAS, on boot up the resilver started once again. I took the opportunity to shut down the FreeNAS and check as best as I could the internal cabling, restated the disks & RAM then powered it back up to allow the resilver to continue.
This is the current error on ada2: Device: /dev/ada2, 48 Currently unreadable (pending) sectors
After running zpool status -v
I saw that there were three files which had permanent errors, as these were not important I removed them, this are now listed as:
errors: Permanent errors have been detected in the following files:
colossus/pyramid:<0x218b5b>
colossus/pyramid:<0x1b18a7>
colossus/pyramid:<0x1727aa>
Should these disappear on reboot or after the current resilver has completed? Could these have been causing the resilver not to complete as they were corrupt?
I have a second 4TB disk ready to replace ada2 but I’d like some guidance if at all possible as to when this would be best to do, I feel the resilver may not be completing and I could introduce data lose if not done correctly?
Time for RAIDZ2 now i have bigger disks?! Data wise i'm okay, it's the jail's and VM's i'd need to work out how best to migrate.
Thanks
Last edited:
