Best way to copy my data to my FreeNAS? Import failed on day 3.

[CardinalPuffPuff]

My hardware is 4 x 4TB drives, and a HP microserver gen10 with 8GB of ram, AMD X3421 CPU, with freenas on a flash drive.

I managed to create a volume, of 8TB mirrored, in a few minutes. As a trial I started to copy my data over using "import disk" from the 8TB drive I keep my files on.

Q: Will that be a ZFS volume?

The import reported failure on day 3.

I would like to clear the NAS and then copy data over selectively using USB. How do I do that with a process that can verify copied data and start where it left off when there is som problem?

 

[Chris Moore]

What kind of connecting did this 8TB drive have before the attempt to connect it to the FreeNAS? What is the filesystem of the drive and how much data was on it?
Are you still able to get into the drive in the way you did before?

 

[CardinalPuffPuff]

Hi Chris,

The import drive is an external 8TB drive with 6.4TB of data. I connected it to the NAS by USB port. The drive is NTFS formatted.

I can read this drive fine from my laptop now.

 

[Bozon]

My hardware is 4 x 4TB drives, and a HP microserver gen10 with 8GB of ram, AMD X3421 CPU, with freenas on a flash drive.

I managed to create a volume, of 8TB mirrored, in a few minutes. As a trial I started to copy my data over using "import disk" from the 8TB drive I keep my files on.

Q: Will that be a ZFS volume?

The import reported failure on day 3.

I would like to clear the NAS and then copy data over selectively using USB. How do I do that with a process that can verify copied data and start where it left off when there is som problem?

rsync is the answer.

https://thesolving.com/storage/how-to-sync-two-freenas-storage-using-rsync/

Edit: Here is a better article that shows some examples. The second example seems like what you want to do. "Copy/Sync a Directory on Local Computer"

https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

Here is an article that looks like it is doing what you want. I haven't read through everything, so I might be mistaken. If not just google "rsync freenas" or just "rsync", it is a super easy to use tool and it is designed to keep files in sync between machines or disks where you may have connectivity issues.

Also, make sure you have good USB 3 cables and that you are connected to the USB3 ports on the server, some of the ports are USB2.

Also, 8GB of ram is cutting it close on what you need. Get 16 GB.

 

[Chris Moore]

Alternatively, since your source is a Windows computer, you might want to download this tool from Microsoft:
https://www.microsoft.com/en-us/download/details.aspx?id=15155
It is a tool called SyncToy. They are not still actively developing the tool, but it allows you to synchronize two folders. The USB drive can be the source and the network drive can be the destination. Just set it up and let it run. If it gets interrupted, it can pickup where it left off.
Easy to use, and I use it myself both at work and at home.
Here is a video on how to use it.
https://www.youtube.com/watch?v=WdeWQkZDUVM

 

[Bozon]

Alternatively, since your source is a Windows computer, you might want to download this tool from Microsoft:
https://www.microsoft.com/en-us/download/details.aspx?id=15155
It is a tool called SyncToy. They are not still actively developing the tool, but it allows you to synchronize two folders. The USB drive can be the source and the network drive can be the destination. Just set it up and let it run. If it gets interrupted, it can pickup where it left off.
Easy to use, and I use it myself both at work and at home.
Here is a video on how to use it.
https://www.youtube.com/watch?v=WdeWQkZDUVM

But if he is connecting the drive to his case, he can run rsync from the command line in freeNAS, but you are right if he is uncomfortable on the Unix command line, it is probably best to use a graphical tool. Can FreeNAS mount a non-ZFS drive? Would this enable him to use the GUI rsync tool built into freeNAS to import the data to the ZFS pool? I guess it can since he is already doing the copy using conventional tools. Doh.

 

[Chris Moore]

But if he is connecting the drive to his case, he can run rsync from the command line in freeNAS, but you are right if he is uncomfortable on the Unix command line, it is probably best to use a graphical tool. Can FreeNAS mount a non-ZFS drive? Would this enable him to use the GUI rsync tool built into freeNAS to import the data to the ZFS pool? I guess it can since he is already doing the copy using conventional tools. Doh.

The connection to a USB drive is usually not reliable enough in FreeNAS to do a large transfer like that. It will drop at some point and that can even cause the NAS to reboot. I have had it happen to me and I have seen other people on the forum complain about it. If you can take it out of the USB enclosure and direct attach it to the NAS, sure, cp or whatever. No problem, except for the NTFS problems. The most reliable way to get data from a USB drive with NTFS format is to copy it while it is connected to a Windows computer. That is just down to the software support for USB that is built into the operating system. It may have improved in the latest version of FreeNAS, but last I tried it, it didn't work very well.

 

[Bozon]

The connection to a USB drive is usually not reliable enough in FreeNAS to do a large transfer like that. It will drop at some point and that can even cause the NAS to reboot. I have had it happen to me and I have seen other people on the forum complain about it. If you can take it out of the USB enclosure and direct attach it to the NAS, sure, cp or whatever. No problem, except for the NTFS problems. The most reliable way to get data from a USB drive with NTFS format is to copy it while it is connected to a Windows computer. That is just down to the software support for USB that is built into the operating system. It may have improved in the latest version of FreeNAS, but last I tried it, it didn't work very well.

Then that makes a ton of sense, you have to work around what is broken, but rsync should be restartable, here is a nice helpful post for future reference about restarting rsync's.

https://unix.stackexchange.com/questions/48298/can-rsync-resume-after-being-interrupted

 

[CardinalPuffPuff]

Thanks for the help. I am a Unix user.

The external drive, which my data is on right now, is a Seagate "backup plus hub." I would not take any HD out of an external enclosure, if I had a choice. These things are flimsy.

I was connecting to the USB 3.0 port on my HP microserver gen10 (according to the spec which I just checked). I will look into getting an extra stick of 8GB ram.

rsync sounds like it will do the job for me. I have my data in folders. I can write scripts to rsync folders onto the NAS, handling things like interruption, and making logs so I know what is going on.

Stupid question 1: Is my volume (comprising 4 x 4TB drives in a mirrored 8tb configuration) a ZFS volume? FreeNas set it all up in a couple of minutes.

Stupid question 2: Is FreeNAS automatically "doing it's thing" (mirroring etc) just by the act of my copying data onto the FreeNAS?

 

[IQless]

Stupid question 1: Is my volume (comprising 4 x 4TB drives in a mirrored 8tb configuration) a ZFS volume? FreeNas set it all up in a couple of minutes.

Stupid question 2: Is FreeNAS automatically "doing it's thing" (mirroring etc) just by the act of my copying data onto the FreeNAS?

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
Take a look at this, it answers both your questions.

Also, take a look in @Chris Moore signature Useful links, a lot of excellent stuff there :)

 

[Bozon]

Thanks for the help. I am a Unix user.

The external drive, which my data is on right now, is a Seagate "backup plus hub." I would not take any HD out of an external enclosure, if I had a choice. These things are flimsy.

I was connecting to the USB 3.0 port on my HP microserver gen10 (according to the spec which I just checked). I will look into getting an extra stick of 8GB ram.

rsync sounds like it will do the job for me. I have my data in folders. I can write scripts to rsync folders onto the NAS, handling things like interruption, and making logs so I know what is going on.

Stupid question 1: Is my volume (comprising 4 x 4TB drives in a mirrored 8tb configuration) a ZFS volume? FreeNas set it all up in a couple of minutes.

Stupid question 2: Is FreeNAS automatically "doing it's thing" (mirroring etc) just by the act of my copying data onto the FreeNAS?

All volumes created by freeNAS use ZFS ( even the boot volume now). ZFS only gives you a little bit of safety. It does things like transaction oriented writing and block verification. What you probably wanted to know was “is it a raidzn volume?” No it is a mirrored configuration but with 4 drives this might be your best setup. Unfortunately you can only suffer 1 drive failure in this configuration ( I know technically there is a possibility that you could survive 2 drive failure but you would have to be unluckily lucky, which isn’t going to happen ). I think you can run raidz2 with this many drives but I keep seeing people say that 6 drives is the minimum that makes sense. I haven’t seen any good articles evaluating 4 drive raidz2.

 

[SweetAndLow]

All volumes created by freeNAS use ZFS ( even the boot volume now). ZFS only gives you a little bit of safety. It does things like transaction oriented writing and block verification. What you probably wanted to know was “is it a raidzn volume?” No it is a mirrored configuration but with 4 drives this might be your best setup. Unfortunately you can only suffer 1 drive failure in this configuration ( I know technically there is a possibility that you could survive 2 drive failure but you would have to be unluckily lucky, which isn’t going to happen ). I think you can run raidz2 with this many drives but I keep seeing people say that 6 drives is the minimum that makes sense. I haven’t seen any good articles evaluating 4 drive raidz2.

Rebuild times with a mirror are significantly faster than a raidz pool. This has greater impact than the idea of losing 2 drivers at once.

 

[Bozon]

Rebuild times with a mirror are significantly faster than a raidz pool. This has greater impact than the idea of losing 2 drivers at once.

You can't loose 2 drives at once with a RAIDZ pool, and not lose data. You can if it is a RAIDZ2 pool. It was probably a typo on your part. Just clarifying this. Sure I think it is swell to run a mirrored pool with only 4 drives. I just wanted to point out the pit falls.

 

[SweetAndLow]

You can't loose 2 drives at once with a RAIDZ pool, and not lose data. You can if it is a RAIDZ2 pool. It was probably a typo on your part. Just clarifying this. Sure I think it is swell to run a mirrored pool with only 4 drives. I just wanted to point out the pit falls.

What are you talking about? I didn't typo and I also didn't say anything about losing 2 disks in a raidz-1 vdev. I said that mirrors rebuild faster than a vdev that is RaidZ. (Note RaidZ is not refering to raidz-1, it's just a generic word for raidZ-X).

 

[Bozon]

What are you talking about? I didn't typo and I also didn't say anything about losing 2 disks in a raidz-1 vdev. I said that mirrors rebuild faster than a vdev that is RaidZ. (Note RaidZ is not refering to raidz-1, it's just a generic word for raidZ-X).

Rebuild times with a mirror are significantly faster than a raidz pool. This has greater impact than the idea of losing 2 drivers at once.

But you replied to my post where I was specifically talking about losing 2 drives, and you had a typo later in your post ( drivers instead of drives.) This caused me to incorrectly extrapolate the meaning from your post. Also, raidz is an often used shorthand for raidz1, I find very few people using RAIDZ when they mean RAIDZ2 or RAIDZ3.

What is significantly faster rebuild times, do you have an article with rebuild time comparisons, I would like to see that. Thanks.

 

[StoreMore]

If you want to copy it from a Windows mounted drive to the FreeNAS share you can use robocopy. This command line utility is built into windows and is fantastic at accurately transferring data. The tool allows for mirroring data, just copying data without permissions, avoids file path length issues, etc. I have used this tool for years to automate routine data transfer and have transferred 100s of TBs.

FROM THE WINDOWS CMD
Mirror Example:

Code:

robocopy "C:\Temp" "S:\ShareName" /MIR

Data Copy Example (Copies all files & in subdirectories but not permissions):

Code:

robocopy "C:\Temp" "S:\ShareName" /COPY:D /S

 

[Chris Moore]

What is significantly faster rebuild times, do you have an article with rebuild time comparisons, I would like to see that. Thanks.

This is highly dependent on a number of factors including the speed of the drive, processor resources available, other IO load on the pool, the amount of data involved; and it is not always true.

 

[Chris Moore]

What is significantly faster rebuild times

I talk about this in the following forum post:
https://forums.freenas.org/index.php?threads/recommendation-for-4-bay-drive-setup.73863/post-512418

 

[SaraNobi1]

to copy or transfer data from windows to another or freeNAS , you have to robocopy your data as a free tool , so check this link https://adamtheautomator.com/robocopy-the-ultimate/
or you can solve your problem with robocopy alternatives which I tried before like secure copy quest and gs richcopy360 that can copy all files and permissions .