coquinone
Dabbler
- Joined
- Mar 6, 2022
- Messages
- 14
Hi all,
I need some help on importing my zfs pool, I'm not an expert in FreeBSD, Truenas and zfs but I'm pretty familiar with Linux.
It's a bit of a long story, so I will try to explain the whole process and hopefully some of you can help me out.
Hardware:
HP N56L microserver, 8GB of memory
5 x 2TB disks
Running Truenas 12U3
I noticed my nfs and sambda shares were offline and not accessible so I rebooted the server a couple of times with no luck.
Hooked up a monitor and saw kernel panic and stack trace points to IO.
I was running Truenas on a usb stick so first thing I thought was a buggered usb stick, so I decided to reinstall on an SSD.
I plugged in the usb and mounted the zfs partition on another PC so I can copy off the config db.
With the db config copied over, I downloaded 12U8 and created an install usb.
During the installation I encountered checksum error on the copied over base-os*.tgz file. After a couple of retries I plugged all my data HDDs and started trying a few things including manually copy the file over. Throughout the process I delete the partitions that were automatically created during the start of the installation. My memory is a bit hazy here so I wasn't sure if I unplugged all the HDDs before I started messing around or after the first a few tries.
Half a day goes pass and after trying different SSDs, different version of Freenas and Truenas, I eventually worked out one of the two sticks of ram I had was faulty. So I replace both with a spare set and got 11.0 installed on the SSD.
After 11.0 got installed, I immediately created a new install usb with 12U8 and upgraded it without booting into 11.0.
Upgrade completed successfully and I booted into 12U8 (during the start process I saw entropy got regenerated). I then overwritten the config db with the one I previously copied and rebooted. After seeing some sqlite errors I realised it was from an older version so I used the webUI restored the config db.
12U8 is not running with my previously configure and zfs pool was not imported and offline
The above is how I got to my current situation and now I'll explain what troubleshooting I have done to the zfs pool
I saw the pool was not online so I went to shell and tried to import it
Mounting it in read only also results in same error
zdb result as follows
Also it seems two of my disks don't have a partition table and with the two that gpart can detect they both have the wrong kinds of partitions
ada4 is my 120GB SSD.
I'm also getting messages that says ada3 has unreadable sectors
Smartctl output for ada3 (looks like the disk is faulty)
I need some help on importing my zfs pool, I'm not an expert in FreeBSD, Truenas and zfs but I'm pretty familiar with Linux.
It's a bit of a long story, so I will try to explain the whole process and hopefully some of you can help me out.
Hardware:
HP N56L microserver, 8GB of memory
5 x 2TB disks
Running Truenas 12U3
I noticed my nfs and sambda shares were offline and not accessible so I rebooted the server a couple of times with no luck.
Hooked up a monitor and saw kernel panic and stack trace points to IO.
I was running Truenas on a usb stick so first thing I thought was a buggered usb stick, so I decided to reinstall on an SSD.
I plugged in the usb and mounted the zfs partition on another PC so I can copy off the config db.
With the db config copied over, I downloaded 12U8 and created an install usb.
During the installation I encountered checksum error on the copied over base-os*.tgz file. After a couple of retries I plugged all my data HDDs and started trying a few things including manually copy the file over. Throughout the process I delete the partitions that were automatically created during the start of the installation. My memory is a bit hazy here so I wasn't sure if I unplugged all the HDDs before I started messing around or after the first a few tries.
Half a day goes pass and after trying different SSDs, different version of Freenas and Truenas, I eventually worked out one of the two sticks of ram I had was faulty. So I replace both with a spare set and got 11.0 installed on the SSD.
After 11.0 got installed, I immediately created a new install usb with 12U8 and upgraded it without booting into 11.0.
Upgrade completed successfully and I booted into 12U8 (during the start process I saw entropy got regenerated). I then overwritten the config db with the one I previously copied and rebooted. After seeing some sqlite errors I realised it was from an older version so I used the webUI restored the config db.
12U8 is not running with my previously configure and zfs pool was not imported and offline
The above is how I got to my current situation and now I'll explain what troubleshooting I have done to the zfs pool
I saw the pool was not online so I went to shell and tried to import it
Code:
root@nas[~]# zpool import vpool cannot import 'vpool': I/O error Destroy and re-create the pool from a backup source. root@nas[~]# zpool status pool: boot-pool state: ONLINE config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 ada4p2 ONLINE 0 0 0 errors: No known data errors root@nas[~]# zpool import pool: vpool id: 12335856088955846037 state: ONLINE status: Some supported features are not enabled on the pool. action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: vpool ONLINE raidz1-0 ONLINE ada0 ONLINE ada2 ONLINE ada1 ONLINE ada3s1 ONLINE ada5 ONLINE root@nas[~]# zpool import 12335856088955846037 cannot import 'vpool': I/O error Destroy and re-create the pool from a backup source. root@nas[~]# zpool import -Fn 12335856088955846037 root@nas[~]# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT boot-pool 101G 1.20G 99.8G - - 0% 1% 1.00x ONLINE - root@nas[~]#
Mounting it in read only also results in same error
Code:
root@nas[~]# zpool import -o readonly=on -NR /mnt 12335856088955846037 cannot import 'vpool': I/O error Destroy and re-create the pool from a backup source.
zdb result as follows
Code:
root@nas[~]# zdb -e -d vpool Dataset mos [META], ID 0, cr_txg 4, 185M, 562 objects Dataset vpool/iocage/jails [ZPL], ID 105, cr_txg 32250219, 153K, 7 objects Dataset vpool/iocage/templates [ZPL], ID 218, cr_txg 32250225, 153K, 7 objects Dataset vpool/iocage/download [ZPL], ID 88, cr_txg 32250215, 153K, 7 objects Dataset vpool/iocage/releases [ZPL], ID 118, cr_txg 32250223, 153K, 7 objects Dataset vpool/iocage/log [ZPL], ID 111, cr_txg 32250221, 153K, 7 objects Dataset vpool/iocage/images [ZPL], ID 95, cr_txg 32250217, 153K, 7 objects Dataset vpool/iocage [ZPL], ID 82, cr_txg 32250213, 9.71M, 597 objects Dataset vpool/.system/services [ZPL], ID 72, cr_txg 32250130, 153K, 7 objects Dataset vpool/.system/rrd-e0dba3050c7945dbbaae28f6aa6a3cc9 [ZPL], ID 60, cr_txg 32250124, 105M, 419 objects Dataset vpool/.system/samba4 [ZPL], ID 54, cr_txg 32250120, 4.05M, 56 objects Dataset vpool/.system/configs-e0dba3050c7945dbbaae28f6aa6a3cc9 [ZPL], ID 66, cr_txg 32250126, 208M, 191 objects Dataset vpool/.system/cores [ZPL], ID 227, cr_txg 32250117, 477M, 8 objects Dataset vpool/.system/webui [ZPL], ID 277, cr_txg 32250128, 153K, 7 objects Dataset vpool/.system/syslog-e0dba3050c7945dbbaae28f6aa6a3cc9 [ZPL], ID 271, cr_txg 32250122, 42.4M, 73 objects Dataset vpool/.system [ZPL], ID 46, cr_txg 32250114, 785M, 23 objects Dataset vpool@auto-20210805-040000 [ZPL], ID 523, cr_txg 32999210, 2.11T, 956559 objects Dataset vpool@auto-20210813-040000 [ZPL], ID 585, cr_txg 33136519, 2.10T, 956509 objects Dataset vpool@auto-20210808-040000 [ZPL], ID 543, cr_txg 33050390, 2.10T, 956547 objects Dataset vpool@auto-20210824-040000 [ZPL], ID 412, cr_txg 33325440, 2.12T, 956514 objects Dataset vpool@auto-20210729-040000 [ZPL], ID 284, cr_txg 32871730, 1.97T, 956224 objects Dataset vpool@auto-20210807-040000 [ZPL], ID 538, cr_txg 33033333, 2.10T, 956547 objects Dataset vpool@auto-20210811-040000 [ZPL], ID 568, cr_txg 33102021, 2.10T, 956520 objects Dataset vpool@auto-20210826-040000 [ZPL], ID 444, cr_txg 33359509, 2.11T, 956451 objects Dataset vpool@auto-20210730-040000 [ZPL], ID 290, cr_txg 32888789, 1.97T, 956224 objects Dataset vpool@auto-20210817-040000 [ZPL], ID 611, cr_txg 33205512, 2.10T, 956509 objects Dataset vpool@auto-20210801-040000 [ZPL], ID 304, cr_txg 32922901, 1.97T, 956224 objects Dataset vpool@auto-20210820-040000 [ZPL], ID 635, cr_txg 33257124, 2.10T, 956509 objects Dataset vpool@auto-20210815-040000 [ZPL], ID 597, cr_txg 33171016, 2.10T, 956509 objects Dataset vpool@auto-20210818-040000 [ZPL], ID 618, cr_txg 33222759, 2.10T, 956509 objects Dataset vpool@auto-20210803-040000 [ZPL], ID 320, cr_txg 32957031, 1.97T, 956229 objects Dataset vpool@auto-20210822-040000 [ZPL], ID 383, cr_txg 33291192, 2.10T, 956492 objects Dataset vpool@auto-20210821-040000 [ZPL], ID 367, cr_txg 33273936, 2.10T, 956489 objects Dataset vpool@auto-20210816-040000 [ZPL], ID 602, cr_txg 33188265, 2.10T, 956509 objects Dataset vpool@auto-20210823-040000 [ZPL], ID 388, cr_txg 33308246, 2.10T, 956492 objects Dataset vpool@auto-20210819-040000 [ZPL], ID 629, cr_txg 33240005, 2.10T, 956509 objects Dataset vpool@auto-20210802-040000 [ZPL], ID 313, cr_txg 32939978, 1.97T, 956229 objects Dataset vpool@auto-20210627-040000 [ZPL], ID 324, cr_txg 32340941, 2.02T, 956646 objects Dataset vpool@auto-20210814-040000 [ZPL], ID 590, cr_txg 33153768, 2.10T, 956509 objects Dataset vpool@auto-20210825-040000 [ZPL], ID 437, cr_txg 33342417, 2.11T, 956427 objects Dataset vpool@auto-20210812-040000 [ZPL], ID 580, cr_txg 33119270, 2.10T, 956509 objects Dataset vpool@auto-20210809-040000 [ZPL], ID 549, cr_txg 33067534, 2.10T, 956544 objects Dataset vpool@auto-20210804-040000 [ZPL], ID 356, cr_txg 32975454, 2.00T, 956255 objects Dataset vpool@auto-20210731-040000 [ZPL], ID 293, cr_txg 32905846, 1.97T, 956224 objects Dataset vpool@auto-20210727-040000 [ZPL], ID 516, cr_txg 32858267, 1.97T, 956224 objects Dataset vpool@auto-20210810-040000 [ZPL], ID 560, cr_txg 33084776, 2.10T, 956520 objects Dataset vpool@auto-20210806-040000 [ZPL], ID 535, cr_txg 33016276, 2.10T, 956547 objects Dataset vpool [ZPL], ID 21, cr_txg 1, 2.27T, 957584 objects Verified large_blocks feature refcount of 0 is correct Verified sha512 feature refcount of 0 is correct Verified skein feature refcount of 0 is correct Verified device_removal feature refcount of 0 is correct Verified indirect_refcount feature refcount of 0 is correct
Also it seems two of my disks don't have a partition table and with the two that gpart can detect they both have the wrong kinds of partitions
ada4 is my 120GB SSD.
I'm also getting messages that says ada3 has unreadable sectors
Code:
root@nas[~]# gpart show => 63 3907029105 ada3 MBR (1.8T) 63 1985 - free - (993K) 2048 3907026944 1 linux-data (1.8T) 3907028992 176 - free - (88K) => 40 246162592 ada4 GPT (117G) 40 1024 1 freebsd-boot (512K) 1064 33554432 3 freebsd-swap (16G) 33555496 212598784 2 freebsd-zfs (101G) 246154280 8352 - free - (4.1M) => 63 3907029105 ada0 MBR (1.8T) 63 3907024002 1 ntfs (1.8T) 3907024065 5103 - free - (2.5M) Mar 7 14:46:03 nas 1 2022-03-07T14:46:03.634637+11:00 nas.localdomain smartd 978 - - Device: /dev/ada3, 256 Currently unreadable (pending) sectors Mar 7 14:46:03 nas 1 2022-03-07T14:46:03.634714+11:00 nas.localdomain smartd 978 - - Device: /dev/ada3, 256 Offline uncorrectable sectors
Smartctl output for ada3 (looks like the disk is faulty)
Code:
root@nas[~]# smartctl -a /dev/ada3 smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p12 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda Green (AF) Device Model: ST2000DL003-9VT166 Serial Number: 5YD4GDHL LU WWN Device Id: 5 000c50 03918a8d4 Firmware Version: CC32 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 5900 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Mon Mar 7 15:30:18 2022 AEDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 633) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 349) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x30b7) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 102 099 006 Pre-fail Always - 4037456 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 215 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 079 060 030 Pre-fail Always - 4382177005 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 22984 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 204 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 077 077 000 Old_age Always - 23 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 066 047 045 Old_age Always - 34 (Min/Max 31/34) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 159 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 225 194 Temperature_Celsius 0x0022 034 053 000 Old_age Always - 34 (0 11 0 0 0) 195 Hardware_ECC_Recovered 0x001a 014 004 000 Old_age Always - 4037456 197 Current_Pending_Sector 0x0012 097 097 000 Old_age Always - 256 198 Offline_Uncorrectable 0x0010 097 097 000 Old_age Offline - 256 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 22937 (189 75 0) 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3222260999 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1273268245 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 22635 - # 2 Short offline Completed without error 00% 22467 - # 3 Short offline Completed without error 00% 22299 - # 4 Short offline Completed without error 00% 22131 - # 5 Short offline Completed without error 00% 21963 - # 6 Short offline Completed without error 00% 21795 - # 7 Short offline Completed without error 00% 21627 - # 8 Short offline Completed without error 00% 21459 - # 9 Short offline Completed without error 00% 21292 - #10 Short offline Completed without error 00% 21124 - #11 Short offline Completed without error 00% 20956 - #12 Short offline Completed without error 00% 20788 - #13 Short offline Completed without error 00% 20620 - #14 Short offline Completed without error 00% 20452 - #15 Short offline Completed without error 00% 20284 - #16 Short offline Completed without error 00% 20116 - #17 Short offline Completed without error 00% 19948 - #18 Short offline Completed without error 00% 19781 - #19 Short offline Completed without error 00% 19613 - #20 Short offline Completed without error 00% 19445 - #21 Short offline Completed without error 00% 19277 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.