Added new disks to JBOD Tray and Wiped Out Zpool

[William Fulton]

Hi, here is my setup before the problem:

1 - IBM x3550 8GB Memory and LSI SAS Card
2 - SAS 12 Unit JBOD Tray
8 - 1.5 GB Hard Drives
2 - Raidz Arrays - 4 Drives in each array (one is raidz, one is raidz2)
FreeNAS-8.0.2-RELEASE-amd64 (8288)

The failed zpool is the raidz array. Here is what happened. I needed to add some new disks so I could add replication and more robust protection for my existing storage. So, I added 8 new drives to the JBOD trays and when I rebooted the FreeNAS server, somehow the new drives had duplicate names and two of the disks in my raidz zpool got their names wiped out or something like that.

I don't have a ton of experience with gpart or any of these tools, but I have many years of linux admin experience, so I can definitely take instruction. What I suspect occured is the label of the two drives got wiped out. The data is still there, I am sure, but the labels are gone. When I try to see a zpool status all I get is an error.

Any help would be greatly appreciated! Also, I will be moving all of the data to raidz2 after this is repaired. I see that raidz1 is dead.

Thanks!

 

[paleoN]

So, I added 8 new drives to the JBOD trays and when I rebooted the FreeNAS server, somehow the new drives had duplicate names and two of the disks in my raidz zpool got their names wiped out or something like that.

Did you create another pool using the new drives? Not sure why you think the labels/partitions were overwritten.

Code:

zpool import

zpool status

camcontrol devlist

gpart show

glabel status

 

[William Fulton]

zpool import:
pool: nfs
id: 4995856370439920777
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://www.sun.com/msg/ZFS-8000-3C
config:

nfs UNAVAIL insufficient replicas
raidz1 UNAVAIL insufficient replicas
gptid/b1565d09-02b0-11e1-9ad5-00145e160158 ONLINE
gptid/b198e56f-02b0-11e1-9ad5-00145e160158 UNAVAIL cannot open
gptid/b1db26b9-02b0-11e1-9ad5-00145e160158 UNAVAIL cannot open
gptid/b21f2ab3-02b0-11e1-9ad5-00145e160158 ONLINE
zpool status:
pool: slowraid
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
slowraid ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gptid/4e3a7f94-050d-11e1-accb-00145e160158 ONLINE 0 0 0
gptid/4eb54c1e-050d-11e1-accb-00145e160158 ONLINE 0 0 0
gptid/4f352de8-050d-11e1-accb-00145e160158 ONLINE 0 0 0
gptid/4fa474a5-050d-11e1-accb-00145e160158 ONLINE 0 0 0
errors: No known data errors
camcontrol devlist:
<nStor OneStor 5000AA 5.56> at scbus0 target 13 lun 0 (pass0,ses0)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 15 lun 0 (pass1,da0)
<ATA WDC WD1502FAEX-0 1D05> at scbus0 target 19 lun 0 (pass2,da1)
<ATA WDC WD1502FAEX-0 1D05> at scbus0 target 22 lun 0 (pass3,da2)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 24 lun 0 (pass4,da3)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 27 lun 0 (pass5,da4)
<ATA SAMSUNG HD204UI 0001> at scbus0 target 29 lun 0 (pass6,da5)
<ATA ST31500341AS CC1H> at scbus0 target 31 lun 0 (pass7,da6)
<ATA ST31500341AS CC1H> at scbus0 target 32 lun 0 (pass8,da7)
<ATA ST31500341AS CC1H> at scbus0 target 33 lun 0 (pass9,da8)
<ATA ST31500341AS CC1H> at scbus0 target 34 lun 0 (pass10,da9)
<HL-DT-ST RW/DVD GCC-4244N 1.02> at scbus1 target 0 lun 0 (pass11,cd0)
<Generic- SD/MMC 1.00> at scbus2 target 0 lun 0 (pass12,da10)
This appears to be a problem. I don't see the other two Western Digital Hard Drives, but the JBOD tray reads them as fine.

 

[paleoN]

This appears to be a problem. I don't see the other two Western Digital Hard Drives, but the JBOD tray reads them as fine.

Use [code][/code] tags for CLI output and you seem to be missing some commands.

Are the newer drives being enumerated first? What happens if you remove the other tray/new drives.

 

[paleoN]

FreeNAS-8.0.2-RELEASE-amd64 (8288)

Just noticed this. This version is ancient. Upgrade to something newer.

 

[William Fulton]

Should I upgrade before I try to fix the drive issue? I'll try removing everything again tonight and see what the results are.

 

[paleoN]

Should I upgrade before I try to fix the drive issue? I'll try removing everything again tonight and see what the results are.

By upgrade I actually mean a fresh install to a different USB stick. FreeNAS not enumerating all the drives may be a firmware/driver issue. Newer versions of FreeNAS have newer LSI drivers. Typically you should keep the firmware in sync with the driver version, but you can boot to see if it recognizes all the drives.

I would removing everything first so you are starting from a working base. Actually, I might not, but you should.

 

[William Fulton]

OK. I'll try to do that. I'll post my results tomorrow morning. I can only work on it during certain hours because I have some functions running on it at regular intervals throughout the night.

 

[William Fulton]

I tried rebooting the server with only the 4 disks in the zpool connected and saw this error on the screen. This is one of the reasons why I think the disks may be mislabeled. The system did not recognize the two missing disks after it was fully booted up.

 

[paleoN]

I tried rebooting the server with only the 4 disks in the zpool connected and saw this error on the screen. This is one of the reasons why I think the disks may be mislabeled.

That's likely the FreeNAS boot drive. Seeing as how you couldn't be bothered to supply all the output I requested earlier I can only guess. With the system off, have you completely reseated the drives, SATA connectors and power connectors or moved the problem drives to a bay?

 

[William Fulton]

I checked the connectors, reseated the drives in the bays, etc. They don't show up in the output you requested, that is the only reason I did not post it. I will repost everything again after I have some time to spend on it later today.

 

[William Fulton]

I managed to get my disks back online after much fiddling with my jbod tray, my cables, my server, etc. It looks like this might have been a connection issue between the jbod trays and the SAS controller.

I tried running a fresh install of Freenas 9 but it failed to boot up properly. So I am still running Freenas 8.0.2 for now. Perhaps I will have a chance to upgrade again in the near future.

Thanks for the help, paleoN!

 

[paleoN]

I managed to get my disks back online after much fiddling with my jbod tray, my cables, my server, etc.

Excellent.

I tried running a fresh install of Freenas 9 but it failed to boot up properly. So I am still running Freenas 8.0.2 for now. Perhaps I will have a chance to upgrade again in the near future.

You could also try the 8.3.2-RELEASE instead.