FreeNas - Add disks to a RaidZ2 Array - Backup and Restore Options

[mjk79]

All,

Forgive me if this has been addressed before, but I wanted to post my specific situation because I haven't found the info I needed to make an informed decision.

My current set up is..

Intel Xeon E3-1241v3 3.5ghz
Noctua NH-U12s Cooler
Asrock C226M WS Micro ATX board
32gb Crucial DDR3 1600 ECC unbuffered Ram
Samsung 850 Evo 250gb (Caching only)
4x 6tb WD Red Drives
RM1000 PSU (overkill but I got it cheap)

Last week I got my hands on 2 additional Seagate 6tb enterprise class drives. I wanted to add these to my existing Raid Z2 array, but quickly found out that it's not possible to add disks to it.

So the end goal here is to add these two drives to expand my storage and I'd love some advice on the best way to do it. If someone knows a way to do this without having to nuke my current volume, I'd love to hear it!

From what I've read on these forums, it looks like i'll have to backup my volume, destroy it, and recreate it with the new raid array. Here are the options I'm aware of.

1. Replication to a new FreeNas. This might not be possible. I'm in the middle of building a new server that was intended to run Ubuntu and act as a NVR for some video cameras. I've tried re-purposing it to create a backup FreeNas, but yet to get the freenas install to boot properly. Possibly because the hardware is pretty new on the H170 chipset and a skylake cpu. This box was never intended to be a long term freenas.

If by some miracle I can get it working, I'm not 100% clear on how replication would work from a restore standpoint. If it backs up my volume, jails and all configurations that I could just restore? That would be ideal, however I'm not clear on if this is the case.

2. Rsync. Ubuntu can run an Rsync server so that's one way I could backup my volume, but similarly to above, I'm not sure if i'd just be backing up my raw data, or my volume configuration/Jails/Settings, etc?

3. Network Transfer. I'd like to avoid this if possible. I'd be backing up my data but lose all my settings and jails would need to be set up again.

If I'm completely off-base on any of this, or if anyone has a better way to accomplish adding these drives, please let me know. I could use some expert advice.

 

[danb35]

If someone knows a way to do this without having to nuke my current volume, I'd love to hear it!

This isn't what you want to hear, but there's no way to do this without nuking your current volume.

How much data is on your pool right now? Do you have any spare drives you could use?

The best way to back up your data, if possible, is ZFS replication. Doing it locally (i.e., to another pool on the same machine) would be fastest, but would require the ability to set up another pool on a spare disk or two. You could also do it to another machine (FreeNAS or otherwise) over your network. Once your pool is rebuilt, replicate the data back. That will give you all your data, datasets, and other ZFS configuration. There's no need to back up your configuration, since you aren't changing it.

 

[mjk79]

You may have just saved my bacon :) it's about 8tb of data.

Do you know if 2x M1015 cards can function in one system? I have 2, both are pass through mode. One is in the NAS, the other in my half working server box.

If I can put them both in the same box, I have a bunch of 4tb drives I can use to fully populate the NAS, and backup to some extra drives on the second m1015 card. Would that work?

If I can get it fully populated now I can just replace the 4tb drives later with 6tb's.

 

[Bidule0hm]

Do you know if 2x M1015 cards can function in one system?

Yes, no problem.

I have a bunch of 4tb drives I can use to fully populate the NAS, and backup to some extra drives on the second m1015 card. Would that work?

Again, yes, no problem. Just be careful that the PSU is powerful enough if you add more drives ;)

 

[mjk79]

Anyone know if there's a guide or if someone can help me create a replication task locally? I'm having trouble getting the replication task to work, I keep getting errors.

 

[danb35]

I believe FreeNAS supports local replication from the GUI, but I've never done it. When I've had the need (not often), I've done it at the CLI. This post should get you started.

 

[depasseg]

I use 127.0.0.1 as the IP and disable the encryption cipher and compression.

 

[mjk79]

Ok so it looks like I was able to get past the host key error, but now i have a new one. Permission denied, public key/password.

I'm guessing i need to set permissions somewhere to make this work?

The Config.

 

[mjk79]

Made some progress, it seems to be at least recognizing the replication task now, but I'm getting a different error.

Hello,

The replication failed for the local ZFS vol1 because the remote system

has diverged snapshots with us and we were unable to remove them,

including:

['Vol_Backup@auto-20151112.1923-2w']

Replication vol1 -> 127.0.0.1:Vol_Backup failed: Failed: vol1/rnash_dataset (auto-20151112.1828-2w)

I'm really not sure what diverged snapshots means, can someone point me in the right direction?

 

[mjk79]

I think... it's working..

A combination of this from Cyberjock..

1. Go to box A's CLI (local or SSH, it doesn't matter)
2. Do "ssh root@boxB" from the CLI and accept the key.
3. Go to box B's CLI (local or SSH)
4. Do "ssh root@boxA" from the CLI and accept the key.
5. Reboot both boxes and try to do replication. (not required, but ensures the data is kept on reboot).

The troubleshooting section in the guide..

ssh -vv -i /data/ssh/replication hostname_or_ip

If this command asks for a password, ssh is not setup correctly.

Finally, this page had the rest of what i needed.
http://technology.mattrude.com/2010/01/enable-ssh-no-password-authorization-with-freenas/

Changing permissions on the authorized keys file seemed to do the trick.

ssh root@freenas
cd ~/.ssh/
chmod 600 authorized_keys

It's not really giving me a status on the WebGUI, but I can hear the external drives working hard and the volume is slowly filling up.

Thank you to everyone who posted here, I never would have found the info I needed without you guys pointing me in the right direction.

 

[depasseg]

Glad to hear it's going. The GUI won't give you status until a replication completes.

 

[mjk79]

Getting a few errors still. This error repeats.

Code:

Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/TS3) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/Ubuntu) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/btsync_1) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/couchpotato_1) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/customplugin_1) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/owncloud_1) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/plexmediaserver_2) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/plexmediaserver_3) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/sabnzbd_1) failed: No such file or directory
Nov 13 08:51:04 freenas collectd[3993]: statvfs(/mnt/Vol_Backup/jails/syncthing_1) failed: No such file or direct

Also the size of my Vol1 is 7.3tb, the size of the Vol_Backup is now 5.3tb. It appears to have stopped transferring also.

Current status is "Up to Date". Is this normal? I have compression totally off, why wouldn't the volume sizes match up?

 

[depasseg]

The statvfs error is a bug. The replication target becomes unmounted sometimes. Try running 'ls /mnt/Vol_Backup/jails/'. I bet it's empty. 'ZFS mount -a' fixes it temporarily.

 

[mjk79]

Thanks, what do you think about the difference in Vol sizes?

 

[mjk79]

Is this the same bug?

Hello,

The replication failed for the local ZFS vol1/jails/plexmediaserver_3 while attempting to

apply incremental send of snapshot auto-20151113.0828-2w -> auto-20151113.0928-2w to 127.0.0.1

 

[depasseg]

Thanks, what do you think about the difference in Vol sizes?

That's a little stranger. Could be different datasets, or replication points. Can you show the screenshots where those sizes are being reported?

 

[depasseg]

Is this the same bug?

I'm not sure if it's the same bug, but it's a bug.
https://bugs.freenas.org/issues/12143

 

[mjk79]

Here's where I'm seeing the volume sizes. I just don't want to blow away my vol1 unless I'm 100% sure all the data replicated over. Some of those numbers just don't add up, maybe I'm not understanding them properly.

I think Vol_Backup did become unmounted because I couldn't browse to it on the network. After rebooting, I was able to connect to it over the network again.

 

[mjk79]

It looks like the new volume (Vol_Backup) keeps disappearing. It appears to be mounted but I can't connect to it over cifs or ftp.

Code:

[root@freenas] ~# zfs list
NAME                                                    USED  AVAIL  REFER  MOUNTPOINT
Vol_Backup                                             3.54T  10.5T   272K  /mnt/Vol_Backup
Vol_Backup/Backup                                       223G  10.5T   223G  /mnt/Vol_Backup/Backup
Vol_Backup/btsync                                      66.6G  10.5T  66.6G  /mnt/Vol_Backup/btsync
Vol_Backup/jails                                        121G  10.5T  1.44M  /mnt/Vol_Backup/jails
Vol_Backup/jails/.warden-template-VirtualBox-4.3.12    1.37G  10.5T  1.37G  /mnt/Vol_Backup/jails/.warden-template-VirtualBox-4.3.12
Vol_Backup/jails/.warden-template-pluginjail            893M  10.5T   887M  /mnt/vol1/jails/.warden-template-pluginjail
Vol_Backup/jails/.warden-template-pluginjail--x64      1.38G  10.5T  1.37G  /mnt/Vol_Backup/jails/.warden-template-pluginjail--x64
Vol_Backup/jails/.warden-template-pluginjail-9.2-x64   1.37G  10.5T  1.35G  /mnt/Vol_Backup/jails/.warden-template-pluginjail-9.2-x64
Vol_Backup/jails/.warden-template-standard--x64        3.59G  10.5T  3.50G  /mnt/Vol_Backup/jails/.warden-template-standard--x64
Vol_Backup/jails/TS3                                   4.47G  10.5T  4.20G  /mnt/Vol_Backup/jails/TS3
Vol_Backup/jails/Ubuntu                                48.5G  10.5T  48.4G  /mnt/Vol_Backup/jails/Ubuntu
Vol_Backup/jails/btsync_1                              2.37G  10.5T  2.32G  /mnt/Vol_Backup/jails/btsync_1
Vol_Backup/jails/couchpotato_1                         7.78G  10.5T  7.66G  /mnt/Vol_Backup/jails/couchpotato_1
Vol_Backup/jails/customplugin_1                        33.5G  10.5T  33.5G  /mnt/Vol_Backup/jails/customplugin_1
Vol_Backup/jails/owncloud_1                            2.39G  10.5T  2.27G  /mnt/Vol_Backup/jails/owncloud_1
Vol_Backup/jails/plexmediaserver_2                     3.02G  10.5T  2.86G  /mnt/Vol_Backup/jails/plexmediaserver_2
Vol_Backup/jails/plexmediaserver_3                     6.46G  10.5T  6.16G  /mnt/Vol_Backup/jails/plexmediaserver_3
Vol_Backup/jails/sabnzbd_1                             1.89G  10.5T  1.80G  /mnt/Vol_Backup/jails/sabnzbd_1
Vol_Backup/jails/syncthing_1                           2.04G  10.5T  1.88G  /mnt/Vol_Backup/jails/syncthing_1
Vol_Backup/plex                                        3.14T  10.5T  3.14T  /mnt/Vol_Backup/plex
freenas-boot                                           4.21G  2.75G    31K  none
freenas-boot/ROOT                                      3.96G  2.75G    31K  none
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201508250051        44K  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509022158        44K  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509220011      12.2M  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509282017        63K  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201510290351      15.7M  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511020249      15.6M  2.75G  1.02G  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511040813      3.92G  2.75G  1.02G  /
freenas-boot/ROOT/default                              5.18M  2.75G   938M  legacy
freenas-boot/grub                                       240M  2.75G  11.2M  legacy
vol1                                                   3.52T  6.69T   430K  /mnt/vol1
vol1/.system                                            415M  6.69T   238K  legacy
vol1/.system/configs-adb946163d914f088dc14617dbc0bec3  19.6M  6.69T  19.5M  legacy
vol1/.system/cores                                      240M  6.69T  48.9M  legacy
vol1/.system/rrd-adb946163d914f088dc14617dbc0bec3       209K  6.69T   209K  legacy
vol1/.system/samba4                                    20.2M  6.69T  2.72M  legacy
vol1/.system/syslog-adb946163d914f088dc14617dbc0bec3    134M  6.69T  37.3M  legacy
vol1/Backup                                             222G  6.69T   222G  /mnt/vol1/Backup
vol1/btsync                                            66.2G  6.69T  66.2G  /mnt/vol1/btsync
vol1/jails                                              106G  6.69T  1.27M  /mnt/vol1/jails
vol1/jails/.warden-template-VirtualBox-4.3.12          1.31G  6.69T  1.31G  /mnt/vol1/jails/.warden-template-VirtualBox-4.3.12
vol1/jails/.warden-template-pluginjail                  834M  6.69T   830M  /mnt/vol1/jails/.warden-template-pluginjail
vol1/jails/.warden-template-pluginjail--x64            1.33G  6.69T  1.32G  /mnt/vol1/jails/.warden-template-pluginjail--x64
vol1/jails/.warden-template-pluginjail-9.2-x64          816M  6.69T   800M  /mnt/vol1/jails/.warden-template-pluginjail-9.2-x64
vol1/jails/.warden-template-standard--x64              3.31G  6.69T  3.24G  /mnt/vol1/jails/.warden-template-standard--x64
vol1/jails/TS3                                         2.51G  6.69T  3.91G  /mnt/vol1/jails/TS3
vol1/jails/Ubuntu                                      47.1G  6.69T  48.4G  /mnt/vol1/jails/Ubuntu
vol1/jails/btsync_1                                    1023M  6.69T  2.27G  /mnt/vol1/jails/btsync_1
vol1/jails/couchpotato_1                               6.39G  6.69T  7.60G  /mnt/vol1/jails/couchpotato_1
vol1/jails/customplugin_1                              32.1G  6.69T  33.3G  /mnt/vol1/jails/customplugin_1
vol1/jails/owncloud_1                                  1.00G  6.69T  1.68G  /mnt/vol1/jails/owncloud_1
vol1/jails/plexmediaserver_2                           1.74G  6.69T  2.38G  /mnt/vol1/jails/plexmediaserver_2
vol1/jails/plexmediaserver_3                           5.07G  6.69T  6.14G  /mnt/vol1/jails/plexmediaserver_3
vol1/jails/sabnzbd_1                                    518M  6.69T  1.74G  /mnt/vol1/jails/sabnzbd_1
vol1/jails/syncthing_1                                  690M  6.69T  1.30G  /mnt/vol1/jails/syncthing_1
vol1/plex                                              3.14T  6.69T  3.14T  /mnt/vol1/plex
[root@freenas] ~#

 

[mjk79]

I created a new volume with two extra disks, deleted the replication task and attempted to create a new one. For some reason it's not letting me create a new replication task from Vol_Backup to my test volume. I just wanted to make sure it was going to work, but this mount issue seems to be preventing it. I've rebooted several times and Vol_Backup hasn't come back yet.