backing up freenas to external

[RadarG]

Is there anyway to plug in an external USB into my freenas and backup everything to it? Could I just format the USB drive to a Linux filesystem and copy the data from three CIFS shares over to it?

 

[anodos]

Is there anyway to plug in an external USB into my freenas and backup everything to it?

Yes, but that's not the best way to do backups. USB can be unreliable and ZFS scrubs on a large USB drive will be about as much fun as getting punched in the face.

Could I just format the USB drive to a Linux filesystem and copy the data from three CIFS shares over to it?

It depends on how you plan to integrate the USB drive into your backup scheme. You won't be able to connect them directly to the FreeNAS device, but you could connect them to a separate linux server.

In short you haven't provided enough information to give a practical answer to your question. To start with,
(1) What information are you planning to back up? What protocols do you use to share the information? Do the backups need to be offsite? What is an acceptable risk level?
(2) What available hardware do you have? Do you have a computer that can be a dedicated backup target?
(3) How is your network set up? Are there computers that are permanently located on the network (desktop computers)?
(4) Is this your data or your employer's data? In the former case lost data may cause you some pain and frustation. In the latter case it can cost you your job (and it's better to ask them to fork out money for a stable and reliable backup solution).

The best way of backing up a FreeNAS appliance is to build a second computer and use ZFS replication, but you will still need to design a way of doing offsite cold-storage (using tape backups or disk backups).
I've used microsoft's synctoy in the past to back up small CIFS shares for a little extra redundancy.

 

[RadarG]

Yes, but that's not the best way to do backups. USB can be unreliable and ZFS scrubs on a large USB drive will be about as much fun as getting punched in the face.


It depends on how you plan to integrate the USB drive into your backup scheme. You won't be able to connect them directly to the FreeNAS device, but you could connect them to a separate linux server.

In short you haven't provided enough information to give a practical answer to your question. To start with,
(1) What information are you planning to back up?
about 300gb of family videos and 2-3Tb of media, movies,tv shows, etc

What protocols do you use to share the information? CIFS to load the family videos and photos. right now the only streaming i do is via plex and roku

Do the backups need to be offsite?
I would love to backup everything up off site, if not just the family pics
What is an acceptable risk level?
I can afford to lose the media I would love to have a good backup. The family pics I want protected from corruption and backup up locally and offsite.

(2) What available hardware do you have?
my freenas is a Dell poweredge T110 with a M1015 installed the case right now has 4 500gb 2.5 drives installed into a 5.25 internal enclousure. and I have 4 bays that I can install 3.5 drives. I have three drives installed right now. 2 500gb and 1TB
I have the 5x 500gb placed in Z3 with the last 500 and 1TB setup as CIFS shares.

Do you have a computer that can be a dedicated backup target? I have a linux laptop that I can use with a 2TB external hooked up to it.

(3) How is your network set up? three vmware hosts with lots of cables. gigabit switches. Are there computers that are permanently located on the network (desktop computers)?yes plus VMs

(4) Is this your data or your employer's data?
no homeuse only
In the former case lost data may cause you some pain and frustation. In the latter case it can cost you your job (and it's better to ask them to fork out money for a stable and reliable backup solution).

The best way of backing up a FreeNAS appliance is to build a second computer and use ZFS replication, but you will still need to design a way of doing offsite cold-storage (using tape backups or disk backups).
I've used microsoft's synctoy in the past to back up small CIFS shares for a little extra redundancy.

 

[diskdiddler]

If I do the "save config" thing in FreeNAS does that get ALL the stuff (jail mount points and stuff?)
So if my FreeNAS server had a motherboard and USB key failure, I could buy the same server, configure a new 9.2.1.6 USB key, import my setttings and import my ZFS volume? Then it'd be identical?

 

[anodos]

It sounds like you don't have any redundancy for your CIFS shares, which isn't a good idea. You will want to mirror them at a minimum (otherwise you lose benefits from ZFS scrubs) A 5-disk raidz3 vdev is also less than ideal. With your setup, I would possibly consider either (1) putting the 6 500GB drives into a RAIDZ2 zpool, and add another 1TB drive and create a mirrored 1TB zpool or (2) add another 1TB drive and put all the drives in RAID10 or (3) create an 8 disk RAIDZ2 vdev.

Regarding backups, if you can't set up a second FreeNAS appliance as a ZFS replication target, then you could set up a linux server as an rsync target and use a service like crashplan to back up your family photos (thus having two layers of backups for the important information).

 

[Sunseeker]

Hello,

i´m totally new to all this.. i have never worked with Freenas / Freebsd before.

I´m running FN 9.2.1.6 having 2 x 4TB drives in Mirror. At the Moment after installing Freenas succesfully i´m restoring all my Data from an external 4 TB USB drive back to the NAS HDD´s (USB Drive is connected to my Mac and i´m moving all the Data to the CIFS Shares. When this is done i would like to use the USB for future Backups, saving all the content located in my shares weekly / daily / every month My idea: i connect the USW when necessary to the NAS let me say the first day in Month and then running a Cron Job which does the Backup Job. Can i realise this and in case that yes, how can i do it ?

thanks in advance

P.s. i think that is exactly what RadarG wants to do !? There are no other Computers in my Network, i have just on Laptop running.

 

[diskdiddler]

I've opted to use my Windows machine to do the backups over the LAN. The USB disk attached to your FreeNAS server can not be written to in NTFS.
So if your FreeNAS machine goes bam, you're going to need linux / freebsd, whatever to read the data.
Better keep it in it's format you intend to use it in. - with robocopy and servers on all night and stuff, the backup script might take 10 hours to run, once a week overnight - no big deal.

 

[cyberjock]

What diskdiddler said. You're best bet is to use robocopy or rsync to make it happen. Do realize that if you have more than a couple of TB that USB speeds are going to make doing backups *very* painful.

 

[diskdiddler]

With USB3 and gigabit ethernet, I would think you could exceed 70MB/s - I dunno about everyone else, but my FreeNAS is 18TB, but the actual SUPER important stuff is about 50gb at most...

 

[Sunseeker]

Thank you both...

Ok, i am not able to use rsync, because i don´t have a remote host where i could plugin my USB Drive.

Maybe there is a way to use xcopy in a script and run it over a cron job.. if this takes hours doesn´t matter because when i do this ones a month it could take 10 hours and more (that would be ok, but if the Freenas crashes, i need a Freebsd / Linux or ....in case that i use UFS Format on the USB drive a Unix System - i dont like this very much...)

Any recommendations how i could use this USB Drive for backups (not using rsync, xcopy over a script when drive is zfs, ufs formated, and i would also not like to run the USB drive connected do my Laptop copy all the data using the network connection) ?

 

[gpsguy]

You should be able to use rsync or an equivalent from your laptop. It looks like the MacOS has direct support for it, many Windows user's use DeltaCopy.

The first time you run it, it could take awhile to transfer the data over the network. Assuming most of your data doesn't change much during the month, subsequent runs shouldn't take as long, since you're only copying the data that hasn't changed.

With the USB connected to your laptop, using a native format like OS X or NTFS, if your FreeNAS server blew up, you'd still be able to access the backup data on your laptop.

Ok, i am not able to use rsync, because i don´t have a remote host where i could plugin my USB Drive.

btw, one can even use rsync without a remote host. i've used it to transfer data from one pool to another on the same FreeNAS server.

 

[Apollo]

You can connect your USB drive on your Freenas machine and create a zvol.
You can follow the procedure I am describing in the following post to do local replication:

http://forums.freenas.org/index.php?threads/no-ecdsa-host-key-is-known-for.21521/#post-128909

I don't know if you can manage writing at USB3.0 speed though.

 

[Sunseeker]

Hello and thanks again.

i followed Apollo´s procedure and have the FN configured for using a local replication but i still didn´t run it. I´ve also configured local rsync but didn´t use it until today.

I´ve mounted the external USB Drive and created a Backup Dataset on it. I wrote a normal copyscript which just copy all the Data on my NAS HDD´s to the external USB (this is still running (just for having a complete Backup once)). What i dont understand is the difference between a local replication and a local configured rsync !? For my understanding both services are doing the same or not ?

Data on Pool "A" has to be copied to the new Dataset (my external USB) Data: "B" When something changes in Data "A" it should be copied to data "B"

Isn´t this what both Services are doing normally ?

 

[Alvin]

If you just want a copy of the data, you can attach an USB drive on the FreeNAS server, and format it with UFS. You can then still read the UFS drive on another computer running FreeBSD or Linux (read-only, or experimental rw). Then use local rsync. (You can even combine that with snapshots on the USB drive.)

I would personally not use ZFS on an USB drive, but you can still send ZFS filesystems to (compressed) files, which is very fast. The downside is that you have to receive them on a ZFS capable system in order to read them back, which can be the same FreeNAS server.
(Hmm, that should be a good feature request for FreeNAS. The inclusion of archivers/pigz or archivers/pxz to speed up compression.)
Also, don't forget the classic tar(1).

Be careful with rsync from/to Windows (DeltaCopy). It does not support UNICODE filenames. Likewise, the default rsync on Mac has no support for extended attributes (which is especially important on a Mac.)

(I'm experimenting with writing to a fiber attached tape robot. The first try should run this night.)

 

[Sunseeker]

Hello Alvin,

thanks for responding.

I have 2 x 4 TB in Mirror running and just need a Backup in case that somthing happens with the HDD´s.

Right now the ext. USB drive is formated with ZFS. This is not a problem because i have a MAC and in general the MAC is able to read ZFS formated Volumes using a special Plugin from for example "GreenBytes". or using "Parted Magic".

As i have the drive connected to my NAS it doesn´t matter if the default rsync in MAC or windows supports "something" or not. I just want to run the rsyn or replic. or whatever on the NAS locally, thatfor I still need to know what the difference is between the rsync and the replic. Service on FN.

 

[Apollo]

Sunseeker, the difference I see between Rsync and replication is that Rsync works directly with the available data within the Dataset, it read the files, convert them and send then to the destination. As Alvin mentioned, it doesn't support UNICODE, so if you have Latin based text in the filename it will screw them up. However, Replication works with the snapshots directly. and handle the blocks irelevent of the contents. If you use replication and you have more than one snapshot you could also replicate these snapshot as well, which Rsync is not going to do unless the snapshot is being clones and it's content copied over.

It is in essence similar to a ghosting versus a backup software.
I do the same for my backups, and I rely more on the replication process than Rsync. By the way, my backup drive is set as ZFS and I benefit from data integrity which RSync only support when files have been written. If the transfer gets interrupted, then the snapshot is not made available on the backup side and the allocated space become available again. With Rsync, you have to go through the entire comparaison process, hence roughly as much as twice the time to run the process again.
You know that when the snapshot is complete the data is available, no second guessing.

Beside, you can also connect the backup drive to pretty much any computer system as long as you can boot it from a USB FREENAS . So no worry about compatibility.
Also, as a last comment, you can scrub the backup drive to check wheither the data as stored and maintained properly because checksum are part of the blocks and will confirm the quality of the date against silent corruption. This is not the case with Rsync as the data stored cannot be compared unless you have access to the original fiels, but then which source to trust in the event of differences?
You can also replicate the backup to another backup and it will be identical to the original dataset from the source.

So in my opinion, common cense would dictate to only rely on ZFS as the backup media.

 

[cyberjock]

Right now the ext. USB drive is formated with ZFS. This is not a problem because i have a MAC and in general the MAC is able to read ZFS formated Volumes using a special Plugin from for example "GreenBytes". or using "Parted Magic".

Give these a read:

http://www.reddit.com/r/sysadmin/comments/2bk3ge/zfs_on_a_usb_drive/

http://www.reddit.com/r/freenas/comments/2cu7y5/zpool_corruption_twice_using_zfs_on_usb/ (7 hours old!)

As has been said, this whole idea just won't end well for you.

Secondly, Parted Magic and GreeByte CANNOT open ZFS partitions created by FreeNAS. If you created them elsewhere then you shouldn't be using them on FreeNAS either. Those two products can recognize them, but it CANNOT mount them at all as far as I know because of feature flags that are supported in FreeNAS but not in those products. Feel free to test this (and you should be before making the claim that it will work) and report back if you find otherwise. ;)

 

[Sunseeker]

Ok, understood !

In general it´s not a good idea to work with USB drives as a backup drive.

The only thing i have to backup is this USB backup drive... means, i have to use it for having a backup.

To make the best of the hole situation... i have to format the drive UFS (using the FN Vol. Manager) and then replicate the snapshots ? is it this what you suggest ?
but after this i will just have access to the snapshots on the drive and not access to file by file right ?

i´m confused :-(

 

[Alvin]

Ok, understood !
In general it´s not a good idea to work with USB drives as a backup drive.

I wouldn't say that. These days, using USB backup drives is more popular than tape. It's cheap. Yes, if you want protection against bit rot, you need another server with multiple drives and ZFS. It all depends on your needs.

To make the best of the whole situation, I have to format the drive UFS (using the FN Vol. Manager) and then replicate the snapshots ? is it this what you suggest?

You can use the UFS Volume Manager, but not ZFS replication. The GUI only allows remote replication, and besides, you need a zpool to replicate to! So, no. Something else will have to serve.

But after this I will just have access to the snapshots on the drive and not access to file by file right?

You have several choices. The first one is block based vs. file based. Block based would be taking ZFS snapshots, and sending them to the external drive. The GUI does not allow this, so you'll have to write a script for that. Basically, the command would be something like:
zfs send volume0/filesystem@tousb | gzip > /mnt/usbdisk/filesystem.gz

+ This is fast, and you will be sure that the backup contains all data.
- A restore requires restoring the whole filesystem.
- A restore requires a system with the same ZFS feature flags. Currently, this means FreeNAS only. (This might change. After all, that's why the OpenZFS project was founded.)

There's also file based. rsync, tar, cpio,.. The FreeNAS GUI does not allow you to rsync locally, so you'll have to create a script. This might be your best choice. You can easily add a cron job that uses tar to write data to the external USB drive.

- You also have to check if the backup is complete. Restrictive file permissions might cause rsync or tar to skip files.
+ You can restore files one at a time.
+ You can easily read the data on another computer, as long as it can read UFS.

If I were to use an USB disk that would be more or less permanently attached to a FreeNAS server, I would set up a jail with rsnapshot. Mount all filesystems you want to backup read-only in the jail, and the USB drive read-write. Configure rsnapshot and you will have an efficient backup system that does not require special software to recover.

 

[Sunseeker]

I can not believe that there is no way to backup your data on an easy way... that it could be handled from an typical enduser !?
I can not use ZFS Volumes on a Windows or MAC System.. even if i have snapshots on the external drive. What should i do with Snapshots on my Windows or MAC Computers ?

It´s a real shame for a FN system like this, that i have to connect my external drive to my computer, using a seperate backup software for backing up on a NTFS drive to be able to use the data in case that i have lost something on my NAS.

What, if i format this drive UFS using the own FN Vol. Manager and then local rsync on the NAS. would this make me able to access the data like on a NTFS system in case of a neccesary restore of some files / folders ?