Question on expanding zpool ( raidz1 )

[ninjaneer68]

I have a 12 bay server I put freenas on. I think I just messed up expanding my storage. I currently have 8 drives now which are 2 sets of raidz1-0 and raidz1-2

I just added 3 more drives and extended my previous volume with the three drives I just purchased. I am thinking I messed up and should have waited till I bought the 4th drive.

Can I remove these three drives and re-do the extension ? I am assuming I can't extend my raidz1-3 listed below and add the 4th drive now ? I have a 11TB in use and not really wanting to remove everything to re-do it.

I am experimenting with using this as my media server. Did I screw up by adding the three drives and not wait till I have 4 ? Now I have one spare empty bay just sitting there. I assumed I could just add the last drive to the raidz1-3 listed below.

Thank you in advance for any feedback

Code:

 ~# zpool status Storage
  pool: Storage
 state: ONLINE
  scan: scrub repaired 0 in 11h40m with 0 errors on Sun Jan 27 11:40:56 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Storage                                         ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/5500d87b-37b0-11e7-81d6-00e081c95102  ONLINE       0     0     0
            gptid/400b5b6d-3755-11e7-81d6-00e081c95102  ONLINE       0     0     0
            gptid/439eeca7-3755-11e7-81d6-00e081c95102  ONLINE       0     0     0
          gptid/6c889a45-3a4e-11e7-8bf6-00e081c95102    ONLINE       0     0     0
          raidz1-2                                      ONLINE       0     0     0
            gptid/7c725411-b37e-11e7-b7a2-00e081c95102  ONLINE       0     0     0
            gptid/80f51b84-b37e-11e7-b7a2-00e081c95102  ONLINE       0     0     0
            gptid/84a93aa9-b37e-11e7-b7a2-00e081c95102  ONLINE       0     0     0
            gptid/8838375e-b37e-11e7-b7a2-00e081c95102  ONLINE       0     0     0
          raidz1-3                                      ONLINE       0     0     0
            gptid/8db5cf0d-42b0-11e9-8072-00e081c95102  ONLINE       0     0     0
            gptid/915c2d0f-42b0-11e9-8072-00e081c95102  ONLINE       0     0     0
            gptid/94f8c8f5-42b0-11e9-8072-00e081c95102  ONLINE       0     0     0

 

[HoneyBadger]

Between the 3-drive RAIDZ1-0 and the 4-drive RAIDZ1-2, there's a single drive that seems to be on its own. I hope that's a formatting error and isn't indicative of a single-drive vdev.

But yes; you did, in a way, "screw up" - once a vdev has been created, you cannot add or remove drives from it.

 

[ninjaneer68]

That one drive raidz1-0 ,not sure why it's spaced like that.

The raidz1-3 , can i delete it then recreate or is that a no because all the storage is now stripping all it's data across the new drives ?

 

[PhiloEpisteme]

The raidz1-3 , can i delete it then recreate or is that a no because all the storage is now stripping all it's data across the new drives ?

If I'm understanding correctly you're correct, you cannot delete the vdev once it is added to the pool. Depending on how much data you've added you could backup the data, rebuild the pools using the vdev configuration you'd like, and then restore from backup. Take a look at the Slideshow explaining VDev, zpool, ZIL and L2ARC, it contains a ton of useful information about storage setups.

 

[ninjaneer68]

Since adding the new drives to the pool I haven't added new data, but there is currently 11Tb of existing videos. I would have to offload that then rebuild ?

I don't have a way to do that :-(

 

[PhiloEpisteme]

11TB may seem unmanageable but if you consider cheap very large HDDs, cloud storage, and only backing up data you cant stand to lose you may have options.

Consider too that no amount of array parity is a replacement for a backup. If your house suffers a disaster or your machine experiences a catastrophic failure you'll need backups to recover your data. Basically any data you want to keep safe require backups as part of your plan. :)

 

[Heracles]

Hey Ninjaneer,

Unfortunately, you did ruined the already low redundancy you had in your pool. From the most minimalistic of all, you dropped to none.

So now, you do have a single drive vDev in your pool. Should that drive fails or disappears, you will loose your entire pool and all your data. No matter what redundancy was in the pool before or how redundant is the next vDev. Should that single drive vDev fails, you loose everything.

Technically, to mitigate the emergency, you can add a drive to that single drive vDev and turn it into a mirror. The WebUI does not expose that option, but it is doable from the command line. You have more info here.

But the only real option is to rebuild your pool completely. So take the opportunity to rebuild it with a stronger protection. You should not use RaidZ1 anymore and go for at least RaidZ2.

So the complete solution seems to be :
--Turn that single drive vDev in a mirror, to recover some redundancy.
You need an extra drive plus the CLI commands referred to in the thread I linked above.

Then, get ready to rebuild the entire pool
--Buy a single 12 / 20 TB drive
--Connect it to a PC or your FreeNAS and move your data to that single drive. You will not have any redundancy on it but it is for a very short moment.
--Once all your data is transferred and confirmed in good condition, you destroy your actual pool.
--Once the pool is destroyed you can create a new one. Because you plan to go for 12 drives, I suggest you do 2x 6 drives RaidZ2. A single 12 drives RaidZ3 vDev would be another option but I prefer 2x RaidZ2. A single 11 drives RaidZ2 or RaidZ3 with a hot spare is another option. Know that you can add a hotspare any time...
--Once the pool is created, move back your data and then they will be much safer

Good luck,

 

[ninjaneer68]

Heracles,

Thanks for the advice, i might do that.

The 11TB , it's all data I can lose , prefer not to but it's not a big deal if i lose some or all of it.
It's just video media for entertainment.

How do i split the vdevs when i rebuild it. Is it just a matter of telling the volume manager how i want the volume build ?

The 12th drive is coming in the mail today. Here is the size of my drives ,

Eight 2 TB drives
Four 3 TB drives

Experimenting with cycling out drives for bigger drives.

 

[ninjaneer68]

11TB may seem unmanageable but if you consider cheap very large HDDs, cloud storage, and only backing up data you can't stand to lose you may have options.

Consider too that no amount of array parity is a replacement for a backup. If your house suffers a disaster or your machine experiences a catastrophic failure you'll need backups to recover your data. Basically any data you want to keep safe require backups as part of your plan. :)

I know, I don't have any data on here i can't afford to lose.

My important data like pictures and important documents is on cloud storage + plus 100% backed up to crashplan. For my back up solution in the house

The reason I am using free as is to get better at running it and some day might host pictures and such on it as well. Letting all this be a really life experience of getting better out building out my volumes.

I have small storage experience with NetApp, but I would consider myself an expert by any means.

I appreciate everyone's feedback on this question

 

[gpsguy]

I would create 2 -- 6 disk RAIDz2 vdev's. Better redundancy for larger drives. After you back up your data, you can wipe your existing disks and recreate the new volume (pool) using "volume manager".

In the vdev that contains 4x3TB + 2x2TB drives, the 3TB drives will be function as 2TB drives. Since you mention Experimenting with cycling out drives for bigger drives - you could replace the 2TB drives in this vdev and once the resilvering was complete, the pool size would expand automatically.

How do i split the vdevs when i rebuild it. Is it just a matter of telling the volume manager how i want the volume build ?

 

[ninjaneer68]

I would create 2 -- 6 disk RAIDz2 vdev's. Better redundancy for larger drives. After you back up your data, you can wipe your existing disks and recreate the new volume (pool) using "volume manager".

In the vdev that contains 4x3TB + 2x2TB drives, the 3TB drives will be function as 2TB drives. Since you mention Experimenting with cycling out drives for bigger drives - you could replace the 2TB drives in this vdev and once the resilvering was complete, the pool size would expand automatically.

Thank you for that, I just bought an 8TB external drive today to offload all the files.

I am currently on FREENAS 9.10. I think I will take this time to upgrade as well

So I go straight to 11.2 or stay down to 11.1 ?

 

[PhiloEpisteme]

The reason I am using free as is to get better at running it and some day might host pictures and such on it as well. Letting all this be a really life experience of getting better out building out my volumes.

This is a good approach. Before you use FreeNAS to store and protect important data you'll want to make sure understand vdevs, pools, disk replacement, mirroring, RAIDZ1|2|3, etc. I'm new myself and loved the learning process.

 

[ninjaneer68]

Between the 3-drive RAIDZ1-0 and the 4-drive RAIDZ1-2, there's a single drive that seems to be on its own. I hope that's a formatting error and isn't indicative of a single-drive vdev.

But yes; you did, in a way, "screw up" - once a vdev has been created, you cannot add or remove drives from it.

I never noticed it before, when I would check volume status on the command line, I just thought it was a clitch for that one drive

guess not, never looked at volume status from the gui before...... hahaha, well you live and learn... opps

 

[HoneyBadger]

I never noticed it before, when I would check volume status on the command line, I just thought it was a clitch for that one drive

guess not, never looked at volume status from the gui before...... hahaha, well you live and learn... opps

View attachment 29132

Yep, that one drive failing would cost you the pool. You absolutely need to migrate off and rebuild.

 

[ninjaneer68]

Yep, that one drive failing would cost you the pool. You absolutely need to migrate off and rebuild.

in progress as we speak, I mounted the 8TB drive. Working on backing up all the config files now..... Then will copy data to backup location. Going to spend all today working on this

 

[ninjaneer68]

I would create 2 -- 6 disk RAIDz2 vdev's. Better redundancy for larger drives. After you back up your data, you can wipe your existing disks and recreate the new volume (pool) using "volume manager".

In the vdev that contains 4x3TB + 2x2TB drives, the 3TB drives will be function as 2TB drives. Since you mention Experimenting with cycling out drives for bigger drives - you could replace the 2TB drives in this vdev and once the resilvering was complete, the pool size would expand automatically.

How do I do this, just create one RAIDz2 in the volume manager, then go back and extend the volume and do it again. is that how the two RAIDz are created ?

 

[ninjaneer68]

Does this look better everyone ?? One think I don't understand when you look at the screen shot. Why does it show a total of 21.tib then right below it shows 14.Tib of available space ?

I am going to take a guess why the size is off

Since I have total of 12 - 2 TB drives for a total of 24TB. and I have two raidz2 so 2 parity drives for raidz for a total of 8TB is that why the available space is 14TB ?

Code:

 pool: test
state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        test                                            ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/64747a3a-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/67eaa48b-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/6bbba11f-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/6f2563ff-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/7287b67d-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/761e216b-4361-11e9-9161-00e081c95102  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/84cb7790-4362-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/8843c825-4362-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/8bc4fd8e-4362-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/8f9d120a-4362-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/9326b5e5-4362-11e9-9161-00e081c95102  ONLINE       0     0     0
            gptid/993e7aff-4362-11e9-9161-00e081c95102  ONLINE       0     0     0

 

[Jessep]

14.0 TiB = 15.4 TB plus formatting loss
https://searchstorage.techtarget.com/definition/tebibyte-TiB

 

[gpsguy]

18TB is equivalent to ~16.2TiB. You'll loose a little extra to overhead.

Now, you could have created a wider, say 10 disk RAIDz2 vdev (I I wouldn't do 12) and you'd lose less (percentage wize to parity). But, with your 12 bay chassis and your inventory of disk sizes, having 6 disk vdev's allows you to grow you storage by upgrading 6 disks at a time. And, you're part way there with 4-x3TB drives.

Since I have total of 12 - 2 TB drives for a total of 24TB. and I have two raidz2 so 2 parity drives for raidz for a total of 8TB is that why the available space is 14TB ?

 

[ninjaneer68]

18TB is equivalent to ~16.2TiB. You'll loose a little extra to overhead.

Now, you could have created a wider, say 10 disk RAIDz2 vdev (I I wouldn't do 12) and you'd lose less (percentage wize to parity). But, with your 12 bay chassis and your inventory of disk sizes, having 6 disk vdev's allows you to grow you storage by upgrading 6 disks at a time. And, you're part way there with 4-x3TB drives.

AHHHHHH that makes sense ! I didn't understand why everyone keeps referrencing I should do 2 vdevs of 6 disk each........ its for the upgrade path of once one of the vdevs are all the same size, I will get the benefit of the size increase ?

ok, that is starting to make sense now.