Adding new disk to mirror existing zpool - stripe

[RlainTheFirst]

Hi!

After searching and trying multiple different approaches I've decided to ask the pros.

I have a newly installed freenas setup which contains a JBOD pool and in it a few datasets containing all my data.
This is called HomeVol and consists of 5 drives: 1 6TB and 4 500GB drives which comes to a total of almost 7.5TB effective storage.

Now I've purchased a second big drive that I wanted to add as a mirror to all the existing data, in case of a drive failiure. I have an offsite backup planned (all hardware ready) as soon as I figure this out.

Some outputs for reference:

Code:

root@freenas_home:~ # zpool status
  pool: HomeVol
state: ONLINE
  scan: none requested
config:

   NAME  STATE  READ WRITE CKSUM
   HomeVol  ONLINE  0  0  0
	gptid/9292f529-9d67-11e7-9fd1-68b599e30ec0  ONLINE  0  0  0
	gptid/93afa882-9d67-11e7-9fd1-68b599e30ec0  ONLINE  0  0  0
	gptid/94d0373e-9d67-11e7-9fd1-68b599e30ec0  ONLINE  0  0  0
	gptid/95f0255a-9d67-11e7-9fd1-68b599e30ec0  ONLINE  0  0  0
	gptid/96985d13-9d67-11e7-9fd1-68b599e30ec0  ONLINE  0  0  0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Wed Sep 27 03:46:00 2017
config:

   NAME  STATE  READ WRITE CKSUM
   freenas-boot  ONLINE  0  0  0
	mirror-0  ONLINE  0  0  0
	da0p2  ONLINE  0  0  0
	da1p2  ONLINE  0  0  0

errors: No known data errors

Images from gui:

What i've tried:

Code:

root@freenas_home:~ # zpool attach HomeVol /dev/ada5
missing <new_device> specification
usage:
   attach [-f] <pool> <device> <new-device>
root@freenas_home:~ # zpool attach HomeVol HomeVol /dev/ada5
cannot attach /dev/ada5 to HomeVol: no such device in pool

Now I understand that this wont work because I am not mirroring a specific disk but rather the entire zpool.
I thought this concept of mine was possible but am now having doubts.
All disks are connected directly to my MoBo which has 6 sata ports.

Any help is highly appreciated!

Kind regards
Robert / RlainTheFirst

 

[danb35]

Now I've purchased a second big drive that I wanted to add as a mirror to all the existing data, in case of a drive failiure.

That isn't possible. You can't mirror, in the ZFS sense, a pool of five striped disks, with a single disk.

 

[RlainTheFirst]

That isn't possible. You can't mirror, in the ZFS sense, a pool of five striped disks, with a single disk.

Not at all? I dont know what a vdev is but I've read it somewhere, cant that be used as medium step?

What is the closest I can get?
It does not have to be (if really not possible) a raid 1 setup, but it can be a local backup as well. I just dont want to have to go all the way to my offsite backup in case of disk failiure.

 

[garm]

Striping together those disks of yours is a really bad idea. ZFS will still be able to detect errors but can’t correct them sins there is no redundancy. Also if a single disk fails you will lose all your data.

This is what I would do..
Turn off the machine.
Finish your offsite with a proper ZFS schema.
Copy over your data to your new safer pool (okey to turn on your original machine for this step).
Destroy your pool on the original machine.
Buy a second 6 TB and 10 TB drive and rebuild your pool as mirror vdev (2*500+500,6+6,10+10) and enjoy ~ 13 TB of proper safe storage.

And also, spend the next few evening reading the FreeNAS manual, ZFS Primer and look at a few Allan Jude videos on YouTube. That will help raise the odds of you loosing you data.

 

[danb35]

That will help raise the odds of you loosing you data.

Hopefully you mean "reduce" the odds--I can't see why he'd want to raise the odds of losing his data.

 

[RlainTheFirst]

Striping together those disks of yours is a really bad idea. ZFS will still be able to detect errors but can’t correct them sins there is no redundancy. Also if a single disk fails you will lose all your data.

This is what I would do..
Turn off the machine.
Finish your offsite with a proper ZFS schema.
Copy over your data to your new safer pool (okey to turn on your original machine for this step).
Destroy your pool on the original machine.
Buy a second 6 TB and 10 TB drive and rebuild your pool as mirror vdev (2*500+500,6+6,10+10) and enjoy ~ 13 TB of proper safe storage.

And also, spend the next few evening reading the FreeNAS manual, ZFS Primer and look at a few Allan Jude videos on YouTube. That will help raise the odds of you loosing you data.

Thank you for your answer!
I want to make sure we understand each other, so just to be overly clear: I have the first 5 disks striped together, and would (just like you say) add redundancy by mirroring that stripe to a new disk.

I currently do not have the money for any more disks.

The offsite backup is not ready yet because the disks that will go into it hold the only backup of all my data outside freenas. Once the main one is up with redundancy I will erase those disks (exFat) and put them in the offsite for the nightly snapshots from my freenas.

I currently only have about 2.3TB of data, and I dont expect it to grow a lot in the next coming years. A cap, of say, 6Tb is acceptable for the next coming years.
If I understood you correctly, my conclusion is that it might be better to transfer everything onto the 6 tb drive and mirror that onto the 10tb one. If that means losing the extra 4 tb for now it is worth it. Is there a way to use that though?

Also: Wow, this forum is so far great and responsive! Any ways I can help out before I know anything?

 

[gpsguy]

@garm and @danb35 are so quick with replies that I can't get a word in edgewise today. ;-)

 

[RlainTheFirst]

@garm and @danb35 are so quick with replies that I can't get a word in edgewise today. ;-)

Haha I love it!

 

[gpsguy]

Also: Wow, this forum is so far great and responsive! Any ways I can help out before I know anything?

Read the docs. . If a PDF or ePUB file is more convenient, you can download one via link in my signature.

 

[garm]

Hopefully you mean "reduce" the odds--I can't see why he'd want to raise the odds of losing his data.

Okey “odds” is a snakepit :) I of course meant lowering the probability and by that increasing the payout on a bet on data loss ;)

 

[danb35]

What is the closest I can get?

The closest that I can think of is to set up a new volume (pool) on the new disk, and set up replication from the old pool to the new pool.

 

[RlainTheFirst]

The closest that I can think of is to set up a new volume (pool) on the new disk, and set up replication from the old pool to the new pool.

Question: Would something like this be possible:
Creating the new pool on 10tb drive, move all data there, destroy stripe. Set up new pool on 6tb drive, copy all data there, destroy 10tb drive, then make the 10tb drive mirror the 6tb drive/

I have a backup of all data.
Are the extra 4 TB left on the 10tb drive unusable or any way to use them productively?

Is it really not possible to have a vdev mirror another vdev without regard to the individual vdevs internal build up?

 

[garm]

Yes that’s doable, only issue is with a single drive pool a metadata corruption can destroy the entire pool. So only write absolute minimum (the initial backup). Mount the pool read only for your second step.

A mirror with a 6 and 10 TB drive will give you a net storage pool of ~ 6 TB until you replace the 6 TB drive with another 10 TB drive. But your not stuck in a corner, I usually replace bad drives with larger ones to slowly grow my pool outside adding new vdevs.

 

[danb35]

Is it really not possible to have a vdev mirror another vdev

No, it isn't. Multiple vdevs are striped, not mirrored.

Are the extra 4 TB left on the 10tb drive unusable or any way to use them productively?

You could partition the 10 TB disk into a 6 TB partition and a 4 TB partition, add the 6 TB partition as a mirror to the 6 TB disk, and create a separate pool on the 4 TB partition, but I wouldn't recommend this.

 

[danb35]

only issue is with a single drive pool a metadata corruption can destroy the entire pool.

It can, but it isn't easy. There are at least two copies of all metadata, and I believe up to six copies of the really critical stuff.

 

[RlainTheFirst]

So what do you guys think, does this sound like a good and valid approach?

First:
Make sure I have a backup outside of freenas for now (which I have on the disks that will be put into offsite backup server).

Then:

1. Create a new zpool on the 10tb drive with the gui. (Erase any datasets that might be on it)
   
2. Create a new recursive snapshot of the source dataset in the gui
3. SSH into the freenas box and do: zfs send -Rv HomeVol@snapshot_name | zfs receive -Fdu pool2 (I found that receive -Fdu was recommended, but I dont understand why -F, rollback, ideas?)
   
4. wait, it might take a lot of time apparently to finish above command.
   
5. Destroy the source volume in gui
6. Destroy zpool: zpool destroy HomeVol
   
7. Create new volume in gui. My plan is the 6tb disk by itself, then two mirrors of two 500GB disks each, ie 6tb+raid1(500GB,500GB)+raid1(500GB,500GB). A total storage of 7TB.
8. Copy data from 10tb into the new 7TB zpool: zfs send -Rv pool2@snapshot_name | zfs receive -Fdu newHomeVol
9. destroy Volume on 10tb drive in gui
10. Attach 10tb drive as a mirror to the 6tb disk, like so:
    zpool attach HomeVol gptid/9292f529-9d67-11e7-9fd1-68b599e30ec0 /dev/ada5
    and just live with the loss of 4tb. If I understand correctly that when the day comes I can replace the 6tb with a 10tb and gain back that storage and just rebuild mirror.
    

    A mirror with a 6 and 10 TB drive will give you a net storage pool of ~ 6 TB until you replace the 6 TB drive with another 10 TB drive. But your not stuck in a corner, I usually replace bad drives with larger ones to slowly grow my pool outside adding new vdevs.

    
    
11. Success, 7TB of redundant storage.
12. Destroy backup disks, setup offsite server with them and set up replication task on home server to it.

Does this sound feasible and like a good plan?

Many thanks for all the answers so far!

 

[MrToddsFriends]

3. SSH into the freenas box and do: zfs send -Rv HomeVol@snapshot_name | zfs receive -Fdu pool2 (I found that receive -Fdu was recommended, but I dont understand why -F, rollback, ideas?)

See also the zfs man page. I don't know if -F is simply ignored if you start with a fresh target pool.

7. Create new volume in gui. My plan is the 6tb disk by itself, then two mirrors of two 500GB disks each, ie 6tb+raid1(500GB,500GB)+raid1(500GB,500GB). A total storage of 7TB.

Personally I would leave out the four 500 GB disks. Four additional disks for an additional storage of 1 TB (800 GB if the 80% rule is taken into account)? I would prefer to keep the setup simple.

 

[Stux]

The closest that I can think of is to set up a new volume (pool) on the new disk, and set up replication from the old pool to the new pool.

The closest would be to set up a new pool on the 10TB, then use graid/geom or whatever to make a stripe device. Then mirror that.

Might even have to replicate a few time.

FWIW, I'd suggest just using the 6 and 10 in mirror.

 

[RlainTheFirst]

Personally I would leave out the four 500 GB disks. Four additional disks for an additional storage of 1 TB (800 GB if the 80% rule is taken into account)? I would prefer to keep the setup simple.

Allright, by now I am leaning toward just having the 6 and 10 mirror eachother and if I really want later I can expand that zpool with the 800GB mirrors if I want the experience of doing that.
Thats possible, right?

The closest would be to set up a new pool on the 10TB, then use graid/geom or whatever to make a stripe device. Then mirror that.

Might even have to replicate a few time.

FWIW, I'd suggest just using the 6 and 10 in mirror.

So a mirror of the 6 and 10 is what I will start with, but out of curiosity:
graid/geom, does that mean i could partition the 10tb into partitions similar to all the other disks involved an mirror them individually to the partitions, and not disks?
I see why this could be a bad idea, especially if a disk fails then one cannot simply insert a new disk but also has to partition it just right etc.

 

[MrToddsFriends]

Allright, by now I am leaning toward just having the 6 and 10 mirror eachother and if I really want later I can expand that zpool with the 800GB mirrors if I want the experience of doing that.
Thats possible, right?

That's right, you could add a second (and third) mirror vdev to your pool later but you cannot remove a vdev from a pool after adding it. So attempt to plan wisely.

A more beneficial experience would be to replace the 6 TB disk by a second 10 TB disk at some time in the future (expands usable storage distinctly) and then to buy a second 6 TB disk (adding a mirror of two 6TB disks to a then existing mirror of two 10 TB disks will expand usable storage distinctly as well).

And don't forget to read cyberjock's slideshow.