Added Disc, how bad did I screw up, and how can I fix?

[emptyBox]

Greetings,

I seem to be having a really bad day and having some serious brain lapse.

I'm looking for any ideas as to what I can possibly do to help save my data. My freenas consist of 4x4tb WD red drive, mirrored, which leaves me with roughly 8tb on freenas 9.2.1.7

The drives were working fine. Several months ago, I bought an EXTRA 4tb drive, which I put in an external USB3.0 case, mounted, and formatted the drive to zfs with my freenas server, and made an extra volume, I didn't add it to my original volume, as I knew that it would destroy all the data for my first volume.
My intention was to use it to back up all the data from my volume1, but after reading a lot of the forum feedback from Cyberjock and others about poor performance/stability of USB3, I decided to skip it.

Fast forward to today, I was fixing a few other things in my server closet, and I see my USB3 disconnected, and remember I need to do a backup. I took the HDD out of the external case, and put it into my freenas. I must've grabbed the wrong lines, as I thought I grabbed the my hot install lines. I figured I would just add the drive/volume2 and backup all the data from volume 1 via sata, since it would be faster than USB3, safer and more stable.

However, when I added the drive via sata cables, it seems to have "degraded" my original volume. And when I removed the drive, it said peripheral drive destroyed. I still have access to my volume1, but I have a flashing degraded going off up top. I still can access my data, but I need to figure out what to do. I'm off to a local electronics store to buy another drive enclosure. Obviously I"m an idiot. Anybody have any ideas? I really just need to figure out the best way to back up my data from my volume1, quickly, and I guess reinstall freenas. Any ideas or experience would be appreciated.

 

[danb35]

What is the output of 'zpool status'? In code tags, please.

 

[emptyBox]

action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 3h5m with 0 errors on Mon Jan 26 07:08:15 2015
config:

NAME STATE READ WRITE
CKSUM
volume1 DEGRADED 0 0
0
mirror-0 ONLINE 0 0
0
gptid/fa41874c-cf35-11e3-aa66-002590f2b088.eli ONLINE 0 0
0
gptid/fa984fd3-cf35-11e3-aa66-002590f2b088.eli ONLINE 0 0
0
mirror-1 DEGRADED 0 0
0
gptid/fb08c855-cf35-11e3-aa66-002590f2b088.eli ONLINE 0 0
0
5805876042820472370 REMOVED 0 0
0 was /dev/gptid/fb6284c7-cf35-11e3-aa66-002590f2b088.eli

errors: No known data errors

 

[emptyBox]

I'm unsure of the easiest method to backup the data on the drive now. I figure I probably have a limited time to get data off of the disks.

I fear shutting off the server to add the 4tb drive, and possibly use it as a backup. I am in the process of reformatting that 4tb drive as a USB and trying to drag as much over as I can, but I feel like that is a really bad option, and I worry about the data on the freenas which came from my mac, if it will have problems being put on a ntfs formatted drive.

Thanks danb35

 

[danb35]

Slow down and breathe. Your data is fine for the time being. Right now, your server thinks one of the 4 TB disks in your pool is disconnected. You do want to fix this, of course, and reasonably expeditiously, but this is not an immediately catastrophic situation. For future reference, when I or someone else asks for command output "in code tags", that means to click the insert button (just to the left of the disk icon, in the editor where you type the message), and select "code" from the menu that pops up. A window will pop up, into which you'll paste the requested information, then click the "Insert" button. This helps preserve formatting, which can be important with some command output.

So, your server has removed one of your disks from your pool. You've removed the new disk you installed, so you only have four disks installed, right? See if your server sees four disks--post the output of 'camcontrol devlist', in code tags, to see what your server is seeing.

 

[emptyBox]

Thanks again Danb35, I am pretty flustered. I just selected bbcode editor and pasted it into there without thinking about it. I re-read your post and realized what I did.

So yes, it appears that you hit it right on the head, I removed the extra disk I installed, and the server went to "degraded".

Code:

# camcontrol devlist
<WDC WD40EFRX-68WT0N0 80.00A80>    at scbus1 target 0 lun 0 (ada0,pass0)
<WDC WD40EFRX-68WT0N0 80.00A80>    at scbus2 target 0 lun 0 (ada1,pass1)
<WDC WD40EFRX-68WT0N0 80.00A80>    at scbus3 target 0 lun 0 (ada2,pass2)
<WDC WD40EFRX-68WT0N0 80.00A80>    at scbus4 target 0 lun 0 (pass3,ada3)
<SanDisk Cruzer Fit 1.27>          at scbus8 target 0 lun 0 (da0,pass4)

So it looks like the ada3 disk is mixed up possibly.

 

[danb35]

So whatever knocked the disk offline, it's back, but the server hasn't reconnected it to the pool. In the web GUI, go to Storage, click on your pool, then click Volume Status (the button that looks like a blank page at the bottom of the screen). That will pull up the volume status page, which should list your four member disks, with one showing "REMOVED" for the status. Click on that one, then click the "Replace" button at the bottom of the screen. A window will pop up, which should let you choose your disk to put it back into your array.

 

[emptyBox]

Wow, so as when I select "replace", it tells me that my key will be invalidated, then asks for a passphrase. This should be the same as the other mirror, or will this replace the passphrase for the entire volume?

 

[emptyBox]

Ok, I guess what I am trying to see is what I seem to be getting here. There is no member disk to select, and it is telling me to that the recovery key will be invalidated for the volume. Just trying to be sure that I don't do something which could be a bigger problem.

Thanks again Danb35 ,for helping me out thus far. I really appreciate it.

 

[Ericloewe]

Woah, woah woah! Encrypted pool?

STOP.

Take this moment to very carefully read the manual's instructions on how to replace disks in encrypted pools. Make sure you have backups of all relevant keys and passphrases, as well as of your most important data.

 

[emptyBox]

Yes, I was actually just looking at the manual, and from what I could tell, it seems I need to enter the correct passkey to get it to "resilver", in the "replacing a failed drive" section. However in section 6.3.12.1 of the manual, the photo clearly shows that it is a member disk of xxx12, and mine I have that red exclamation point.

I also tried to change my passphrase to something less lengthy, to be sure, but it was denied saying that it cannot open a gptid/xxxxxxx..

So I am pretty nervous at this point. What would be the fastest solution to back up the data? I have an empty 4tb disk, I suppose I could connect it to my windoze or mac box, and do it over USB. I haven't backed up in some time, so I am looking for a way to save most of my data.

 

[emptyBox]

What do you think is the best way to back up some of the data, before I attempt to "replace" the drive into the pool?

I have the 4tb drive, and an external enclosure for it. My thought is to put it back in the server, see if I can add another volume, and try to backup my volume1 to the 4tb drive.

My other options are to keep the drive in the external enclosure, format it for either windows or mac, and transfer as much as I can over the network. I'm guessing that this would take a long time though, even over usb3.

Any ideas?

 

[emptyBox]

Thanks to danb35 and ericloewe for the assistance. Just a follow up, should anyone else ever find this post by a web search.

Ironically, I couldn't seem to get the "member disk" to show up. After spending most of the day backing up my most recent data, I decided to upgrade from 9.2.1.7 to 9.3.latest. I had been plagued by sam-rids errors ever since upgrading from 9.2.1.3 to 9.2.1.9, I downgraded to 9.2.1.7, and seemingly had fewer errors. I was hoping to get rid of them completely with a completely fresh 9.3.

I rebooted the system, did a 9.3 firmware upgrade via the GUI, rebooted, and instead of unlocking the volume with a passphrase, I used my geli key. When I checked the disk status, ironically it was in "member disk" drop down box this time. I selected it, entered passphrase, and "replaced disk". I had to remove the other disk, which freenas had associated with my volume1, and my volume1 started to resilver.

I recreated the passphrase, reencrypted the drive and created a new recovery key per the manual, and am now waiting for the device to finish resilvering.

However the sam rids: deadlock seems to be a persistent error, that has followed me even to 9.3, looks like it will be a full reconfig after all.