Help/Advice needed rebuild ZFS pool

[DaveB_6958]

Hi guys, I'm been watching the forum for long time but never posted.

Trying to be proactive on solving a potential issue with my ZFS pool.
I have a homebrew FreeNAS server which has been running for several years and has survived motherboard replacement and other minor catastrophes.

Current with all the releases of FreeNAS, now at 11.2
The server was first built with three 3 TB WD red drives in a raidz1-0 pool, over the years I've added additional drives one at a time to a point where now there are 6 identical drives. Unfortunately the 3 newest are not part of the raidz-0 pool, which is what I want to correct. My understanding is if any of the newest three drives were to fail the entire pool would be lost. (true?)

Here's the existing configuration:

Code:

        NAME                                            STATE     READ WRITE CKSUM

        Pool                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/2f0850ed-879f-11e3-86a6-00012e31082b  ONLINE       0     0     0
            gptid/2f529b1d-879f-11e3-86a6-00012e31082b  ONLINE       0     0     0
            gptid/2ff23203-879f-11e3-86a6-00012e31082b  ONLINE       0     0     0
          gptid/f96b9943-49ab-11e4-a210-00012e31082b    ONLINE       0     0     0
          gptid/2bb8bb5a-43f8-11e8-85cb-00012e31082b    ONLINE       0     0     0
          gptid/88308c60-43f8-11e8-85cb-00012e31082b    ONLINE       0     0     0

I currently have about 9TB of data in large files (mostly movies)

What I think I need to do is back up the entire pool to a safe place, destroy the existing pool, reconfigure the drives and restore the data to the new pool.

So my question is how to do that, I have space for 2 more drives in my storage enclosure, so I could add 2 6TB drives at a cost of over $800 to back up to, or subscribe to a cloud storage service back up to it and restore from there using 150 Mb/ 10 Mb internet (slower but cheaper) or something else?

What would you do? Assuming I haven't won the lottery and need a solution that would be reliable.

Thank's in advance

-db-

 

[Chris Moore]

Hi guys, I'm been watching the forum for long time but never posted.

Welcome!

Unfortunately the 3 newest are not part of the raidz-0 pool, which is what I want to correct. My understanding is if any of the newest three drives were to fail the entire pool would be lost. (true?)

Yes. That is true and a scary place to be.
When you are posting a list like the one above, it makes it easier to read if you put it inside code tags like this:

Code:

[root@freenas ~]# zpool status -v|less                                                                                            
  pool: NAS                                                                                                                      
state: DEGRADED                                                                                                                  
status: One or more devices has experienced an unrecoverable error.  An                                                          
        attempt was made to correct the error.  Applications are unaffected.                                                      
action: Determine if the device needs to be replaced, and clear the errors                                                        
        using 'zpool clear' or replace the device with 'zpool replace'.                                                          
   see: http://illumos.org/msg/ZFS-8000-9P                                                                                        
  scan: scrub repaired 788K in 1h25m with 0 errors on Sat Jul  9 01:37:34 2016                                                    
config:                                                                                                                          
                                                                                                                                 
        NAME                                            STATE     READ WRITE CKSUM                                                
        NAS                                             DEGRADED     0     0     0                                                
          raidz2-0                                      DEGRADED     0     0     0                                                
            gptid/da4b033c-0ce0-11e6-b23c-d050997eebf5  ONLINE       0     0     0                                                
            gptid/db5b2629-0ce0-11e6-b23c-d050997eebf5  ONLINE       0     0     0                                                
            gptid/dc7f5445-0ce0-11e6-b23c-d050997eebf5  DEGRADED     0     0   196  too many errors                              
            gptid/dd99642d-0ce0-11e6-b23c-d050997eebf5  ONLINE       0     0     0                                                
            gptid/deb99316-0ce0-11e6-b23c-d050997eebf5  ONLINE       0     0     0                                                
            gptid/dfe5d181-0ce0-11e6-b23c-d050997eebf5  ONLINE       0     0     0                                                
                                                                                                                                 
errors: No known data errors                                                                                                      
                                                                                                                                 
  pool: freenas-boot                                                                                                              
state: ONLINE                                                                                                                    
  scan: scrub repaired 0 in 0h1m with 0 errors on Sat Jul  2 03:46:27 2016                                                        
config:                                                                                                                          
                                                                                                                                 
        NAME        STATE     READ WRITE CKSUM                                                                                    
        freenas-boot  ONLINE       0     0     0                                                                                  
          da0p2     ONLINE       0     0     0                                                                                    
                                                                                                                                 
errors: No known data errors 

Please, would you do that in the future.

What I think I need to do is back up the entire pool to a safe place, destroy the existing pool, reconfigure the drives and restore the data to the new pool.

Absolutely correct and It would be really a good idea to use RAIDz2, which is suggested when drive size is 2TB or greater. When you rebuild the pool, you could increase it from 6 drives to 8 which would keep the capacity about the same as it is now despite increasing redundancy.

So my question is how to do that,

Your idea is not invalid. What you might want to do, for a backup, is use a single large drive. You can get a 10TB drive for around $350 from NewEgg. I was just pricing them yesterday.

What would you do?

What I did do when I was in a somewhat similar situation a few years ago, I built a second NAS. I currently still run a primary and a backup at home. It is (in my mind) better than paying a monthly fee to a cloud service provider to backup my data.

 

[DaveB_6958]

Chris, thanks for the input. I see one problem - if I added a 10 TB drive to the mix then I'd only have 1 slot available for expansion and could not add the two 3TB disks as you suggest and would be investing $$ for a drive that I wouldn't use elsewhere.

Thanks for the tip on posting the configuration - forgot about that one.

 

[danb35]

add 2 6TB drives at a cost of over $800

...or, more reasonably, at a cost of under $400 for the two. Or better yet, two 8 TB disks for under $300 for the two.

Edit: The links above should say "under $400" and "under $300". Not sure why they aren't.

Edit 2: I edited the post to add the above note. When I saved it, the links displayed correctly. When I returned to this thread, the links are again saying "under 0". Forum's looking kind of broken here.

 

[Chris Moore]

Why are you saying "under 0". I don't see how? Is there a discount or rebate?

 

[danb35]

Why are you saying "under 0". I don't see how? Is there a discount or rebate?

Strange, that's not what I typed--I wrote "under $400" and "under $300", respectively. I'll edit and see if it takes.

 

[Chris Moore]

Chris, thanks for the input. I see one problem - if I added a 10 TB drive to the mix then I'd only have 1 slot available for expansion and could not add the two 3TB disks as you suggest and would be investing $$ for a drive that I wouldn't use elsewhere.

Thanks for the tip on posting the configuration - forgot about that one.

As a temporary backup while you reconfigure your pool, I had not intended that you consume a bay in your chassis. Move the data to another storage location temporarily, reconfigure the pool with extra disks and then move the data back. It takes time to move data, trust me, I just finished a server at work that had about 280 TB of data and it took 17 days to copy over to the new server.
I looked back up the thread and I don't see where you gave your full system specs but if you are lacking connectivity it is not a difficult thing to add more ports. There are plenty of people here willing to assist, just let us know what you are thinking. I have done this kind of thing more times than I can count between work and home.

 

[DaveB_6958]

Ok, good suggestions -- thank you.

On my configuration, I did mention I had only 2 empty drive bays left with 6 in use, sorry, thought that would have been evident.

So what I've decided to do is to attempt to resurrect an old HP Mediaserver EX495 that I have on the shelf, this is a server marketed to small businesses and consumers as an entry level server. It is a "headless" unit and doesn't have any video output port but can be mod to add one, it has 4 hot swap drive bays. I can gather a enough unused drives to achieve about 6TB. So I'll back-up most of the data to it, the rest will have to go onto an external drive (been thinking of getting one anyway for a Windows 2012 server). I'll have to divide up the backup into at least 2 parts, but it should ok.

After backing up I'll fill out the original server with 2 more WD 3TB Red drives and go with RAIDz2, then eventually fill up the repurposed HP server with a couple of larger drives to use for backups.

I hadn't considered using the HP before, but hopefully I can install FreeNAS on it if I can get video going so I can see what I'm doing, otherwise it'll be back to the drawing board.

 

[DaveB_6958]

More questions, assuming I destroy the old ZFS pool and create a new with all the same file structures and name just different RAIDz , will I need to reconfigure anything such as where jails point to for their storage or such? I'm assuming the internal drive identifier hash will change because of the reconfiguration.

Again, just trying to be proactive, any other gotcha items I should be aware of?

 

[Chris Moore]

will I need to reconfigure anything such as where jails point to for their storage or such? I'm assuming the internal drive identifier hash will change because of the reconfiguration.

When you destroy the pool, FreeNAS will helpfully offer to delete the share configuration. If you select no, when you recreate the pool with the same name, part of the work will already be done, but you will need to go back over the steps to ensure you don't miss anything. I did this recently and forgot to change the default user permissions of the filesystem so even though I had a share, I didn't have permission to write to it at the filesystem level. It was a fairly quick thing to change, but it is just an example of how you might miss a step by starting in the middle.
Here is a fairly comprehensive guide that might help:

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

 

[Chris Moore]

Forum's looking kind of broken here.

I got a laugh from that. Sorry.