Migrating to FreeNAS from Windows server 2008 R2.

[paleoN]

Warning: I only skimmed some of the posts.

I assume you are using rsync version 3 or higher? Version 2 is slow, waits to start sending, but I assume it's quite old as well.

Your seem to be overlooking running multiple rsyncs in parallel. You would want to run 3, or more?, to get concurrency & faster overall transfer anyway. This is even true if you use zfs send/receive, ideally from multiple datasets, to the final FreeNAS.

Since this entire drive is shared in Windows I wonder if I could mount it in a FreeNAS VM on the Windows server and then do an rsync from the FreeNAS VM to the FreeNAS physical machine. The VM is already setup and I use it to experiment and break FreeNAS.

Just mount it to the temp FreeNAS at that point. What is the VM getting you?

 

[cyberjock]

Deltacopy comes with rsync 3.0.4 and cwRsync uses 3.0.7.

I had thought about running multiple rsyncs at the same time. I was just hoping for a simple "set it up and click go". Unfortunately Deltacopy doesn't let you run multiple rsyncs simultaneously, which seemed to be the easiest option.

Right now we're looking at just letting it run until it finishes. I don't have the time pressure that I thought I would to finish this quickly. Especially since 8.3.1 adds encryption and we're hoping that 8.3.1 will be released before we finish all of this stuff so we can use encryption for one of the zpools. I asked some questions about 8.3.1 here.

 

[cyberjock]

Deltacopy comes with rsync 3.0.4 and cwRsync uses 3.0.7.

I had thought about running multiple rsyncs at the same time. I was just hoping for a simple "set it up and click go". Unfortunately Deltacopy doesn't let you run multiple rsyncs simultaneously, which seemed to be the easiest option.

Right now we're looking at just letting it run until it finishes. I don't have the time pressure that I thought I would to finish this quickly. Especially since 8.3.1 adds encryption and we're hoping that 8.3.1 will be released before we finish all of this stuff so we can use encryption for one of the zpools. I asked some questions about 8.3.1 here.

I was thinking that with both ends being FreeNAS I could use replication. It's all good though since we're going to let rsync run until it finishes.

By the way, overall speed average is 25.3MB/sec.

 

[cyberjock]

Ok, I have an update.

rsync is going too slow. Average speed is just 25.3MB/sec. I was thinking of using CIFS to copy from the server to FreeNAS, then doing an rsync to make sure I have all of the files. But if I copy them from Windows the dates change, so rsync rejects the files thinking they are out of date. But, if I use the 'cp -a' then it MIGHT keep the dates.

So, is there a way to mount the windows file share into FreeNAS? I tried searching but anything I search for always gives me the opposite. I figure if I can map //server/stuff to /mnt/servershare or something like that maybe I can do a 'cp -a' and get the faster CIFS speeds.

Edit: Sure enough, I changed my search and the first post got me the command.

mount_smbfs -I 192.168.1.2 //admin@server/m_drive /mnt/server

Doing a small 'cp -a' copy right now to see how rsync handles the files. It's currently copying at 50-100MB/sec(fluctuates).

Another edit: So the 'cp -a' did "fool" rsync. Since smb has its own checksums to identify bad packets I'm going to do a 'cp -a' of the whole server, then do an rsync to verify that all of the files are there. This has the advantage of being more than 3x faster than just rsync. But it has the downside that files 'may' be corrupted if the packet checksums fail me(I don't think thats very likely). I'm still not sure how much better rsync is than just doing a copy using smb/cifs.

 

[paleoN]

rsync is going too slow. Average speed is just 25.3MB/sec.

I thought you weren't worried about speed? I did suggest running a few in parallel. To get the most speed copying from the temp FreeNAS to the final one you would want to have the files in a few different datasets anyway.

I'm still not sure how much better rsync is than just doing a copy using smb/cifs.

For one it's restartable.

Your cp -a will also be faster to run a few in parallel.

 

[bollar]

Have you tried running rsync from the FreeNAS side? I wonder if that might give you options that the Windows client isn't handling in the way you expect.

 

[cyberjock]

I thought you weren't worried about speed? I did suggest running a few in parallel. To get the most speed copying from the temp FreeNAS to the final one you would want to have the files in a few different datasets anyway.

For one it's restartable.

Your cp -a will also be faster to run a few in parallel.

My cp command is somewhat restartable with cp -a -v -n. I also plan to do an rsync after I copy all of the data just to make sure I have everything and its up to date.

I did try running 2 rsyncs in parallel, it didn't go faster. I don't know why exactly. I know deltacopy didn't like me trying to do 2 at the same time so I had to do some stuff to make it play.

I also tried doing 2 CPs at the same time, it did go faster, but only marginally. I think that the current system performs more efficiently with less head seeking with 1 CP running.

Have you tried running rsync from the FreeNAS side? I wonder if that might give you options that the Windows client isn't handling in the way you expect.

I did try going the other way. The issue seems to be that rsync is CPU intensive and single threaded so the direction doesn't seem to matter as much as your single core max performance. In this case both machines are old chips(at least, by my standards). The current server is an E5606 2.13Ghz Xeon with 12GB of RAM(soon to be upgraded to 48GB of RAM). The temp server is an i7 920 with 20GB of RAM.



I have also found a limitation of this whole process. Some of the files have Chinese or Japanese characters in them. rsync has a heart attack when it sees them and converts the characters to a bunch of question marks, then pukes on itself with I/O errors. The cp command has the same issue without the I/O error messages. If I go to the mounted location of the files and do an 'ls -l' the file names listed have question marks. The filenames also have question marks if I browse to the location from within Windows. But if I copy the files to the temp server through Windows using drag-and-drop the file names are correct in Windows. But from the CLI they are still just question marks. I'm not sure how I want to deal with this yet.

The issue seems to be that rsync, cp, and the CLI don't handle the character set to properly identify the file names. But the odd characters are capable of being properly stored by the file system, accessible by smbd, and shared properly. I'm thinking I'll just make sure to go the roundabout way and always copy those files from within Windows when the time comes.

 

[bollar]

The issue seems to be that rsync is CPU intensive and single threaded so the direction doesn't seem to matter as much as your single core max performance.

Ick. In my applications, I usually use NFS followed by rsync for the same reason, but didn't think that would help in a Windows environment.

I have also found a limitation of this whole process. Some of the files have Chinese or Japanese characters in them. rsync has a heart attack when it sees them and converts the characters to a bunch of question marks, then pukes on itself with I/O errors. The cp command has the same issue without the I/O error messages. If I go to the mounted location of the files and do an 'ls -l' the file names listed have question marks. The filenames also have question marks if I browse to the location from within Windows. But if I copy the files to the temp server through Windows using drag-and-drop the file names are correct in Windows. But from the CLI they are still just question marks. I'm not sure how I want to deal with this yet.

The issue seems to be that rsync, cp, and the CLI don't handle the character set to properly identify the file names. But the odd characters are capable of being properly stored by the file system, accessible by smbd, and shared properly. I'm thinking I'll just make sure to go the roundabout way and always copy those files from within Windows when the time comes.

This doesn't fix your problem, but I would guess that the issue is that the file names aren't in Unicode (maybe UTF-8?) and rsync doesn't know what to do with them. In theory, rsync doesn't care about file names, but it's also not able to convert them. Maybe changing the code page on the Windows side can fix this, but I don't have experience, so don't know for sure.

 

[cyberjock]

This doesn't fix your problem, but I would guess that the issue is that the file names aren't in Unicode (maybe UTF-8?) and rsync doesn't know what to do with them. In theory, rsync doesn't care about file names, but it's also not able to convert them. Maybe changing the code page on the Windows side can fix this, but I don't have experience, so don't know for sure.

Actually, I think the problem is that the CLI doesn't recognize UTF-8(or whatever it is). That means that other programs that run through the CLI may the same limitations. In my case, cp and rsyncd seem to have these limits while smbd does not. Windows seems to properly support them throughout, but I bet if i tried to use the DOS copy command I'd run into the same problems as cp.

Edit: The DOS copy command does have the same problem that cp has.

 

[bollar]

Actually, I think the problem is that the CLI doesn't recognize UTF-8(or whatever it is). That means that other programs that run through the CLI may the same limitations. In my case, cp and rsyncd seem to have these limits while smbd does not. Windows seems to properly support them throughout, but I bet if i tried to use the DOS copy command I'd run into the same problems as cp.

Edit: The DOS copy command does have the same problem that cp has.

If the codepages aren't the same on both machines, rsync needs to be told how to make the conversion with the --iconv switch. I would guess that Samba doesn't have problems because of the way CIFS has been configured. If you look at your sharing settings, I think you'll find that CIFS has Dos=CP437 and Unix=UTF-8 and that's what's doing the conversion.

 

[cyberjock]

Well, I started migrating data back from my temporary FreeNAS server to the new permanent one. The permanent server has more storage space from the temp, but I have what I think is an issue. The snapshot is taken and then the replication starts. I get 850Mb/sec solid. Woohoo! I love those speeds! I have also verified that if you are using odd characters like Chinese/Japanese characters then your file names will be preserved.

I'm having an odd concern though. Since its 20TB of data the replication will take a few days. Not that big of a deal, but I am not sure if the task is broken or this behavior is normal.

I know.. you aren't supposed to fill a zpool over 80% for various reasons, but this is a one time copy and nobody is accessing the data on this temp server.

Source machine: (data from "View Volumes" under GUI)

Used: 19.3TiB(94%)
Available: 1.1TiB
Size: 20.4TiB
Status: HEALTHY

Destination machine: (before replication begins)

Used: 384Kib(0%)
Available: 28.4TiB
Size: 28.4TiB
Status: HEALTHY

Destination 1 minute after the replication starts:

Used: 2KiB(8%)
Available: 756KiB
Size: 823.5KiB
STATUS: Healthy

Is this normal? I did a zpool status and all sorts of stuff and I don't seem to recognize anything as being wrong except these sizes are completely bogus. Even if you assume that all of the space from the data is immediately allocated I end up with many TiB free. If I do a 'zpool iostat 1' the zpool writes 515MB every 5 seconds, so its doing something.

I tried recreating the zpool with 1 less disk. I get the exact same odd KiB size afterwards. Is this normal while the replication is in progress? Should I wait the 4 days for it to finish and see if it actually is ok? I get the distinct feeling I've somehow screwed something up badly.

I guess I just kind of expected that the files would start appearing as the replication task progressed and not have zero files and a zpool that magically shrunk to less than 1MiB.

 

[paleoN]

Source machine: (data from "View Volumes" under GUI)

[snip]

Is this normal?

It's inaccurate and always has been, but I can't say I understand what it's doing in your case. Are those numbers for the base zpool dataset?

Output of:

Code:

zpool list

zfs list -ro space -t all

May tell use more useful things.

 

[bollar]

I agree with paleoN and say that's normal until the replication is done. I run:

Code:

 zfs list -o space 

on both machines to check progress.

 

[cyberjock]

Well, running zfs list -o space shows that available space is slowly decreasing and used is slowly increasing. Both values are consistent with the size of my zpool. So I guess the GUI sizes are just bogus and I should ignore them.

Also a zpool list showed the same accurate data- allocated going up and free going down.

Strangely, I tried to create a 1MB file using the dd command when the zpool appeared to be less than 1MB. I got a file system full error! LOL. One thing I did notice when I was playing around was that on the source system I got a message at one point that /mnt/tank could not be unmounted. /mnt/tank only exists on the destination so it sounds like the zpool may actually be unmounted to some extent during the actual replication process.

I'm trying to figure out how the replication works exactly. The source zpool is over 90% full and I'd hate to replicate the poor file placement from having such a full zpool. In essence, I'm wondering if the "fragmentation" of my files on the source will be duplicated on the destination or if the destination doesn't quite work like that. I'm thinking the latter but I'd like to verify this.

 

[bollar]

I don't think the underpinnings of replication are well documented, but there are a couple of blogs that I found helpful:

http://cuddletech.com/blog/pivot/entry.php?id=984
http://www.brentrjones.com/2009/03/29/opensolaris-zfs-replication/


And some Sun documentation:

http://docs.oracle.com/cd/E22471_01/html/820-4167/shares__projects__replication.html

 

[cyberjock]

So I have finally completed the migration. Here's what I've learned from this voyage. Keep in mind that I had to copy 20TB of data from a Windows Server 2008 R2 machine to a temporary FreeNAS machine, then back to the same hardware as the Windows Server. The temporary FreeNAS server had brand new(read: untested and I consider unreliable until I've ran them through some paces).

Moving data from the Windows Server to FreeNAS.

1. The server ran Symantec Antivirus and was significantly hindering performance. Despite having an Xeon e5620 I could not get above about 40MB/sec consistently. Uninstalling the antivirus solved the problem(Who cares about antivirus when the operating system is about to go bye-bye anyway).
2. If you have files that have non-English characters you have extra pain and misery to deal with. Note that at this point I always copied data from one server to the other since I'd rather have 2 copies of my data than 1 if something goes wrong with those shiny new hard drives in the temporary server.

If you do NOT have any files with non-English characters then you can either copy them from within Windows Server by mapping a CIFS share from FreeNAS and copying to your zpool or mounting the Windows share(assuming you have one) and copying the data from the command line. For anyone not familiar with the command line check out the mount_cifs command via Google. It will explain how to use it. In both cases I got 60-80MB/sec for big files. A program for Windows that I use often called "Total Commander" was pretty helpful for copying all files, but Windows Explorer could have worked just as well. Alternately you can choose to use rsync. Rsync worked fine and I got speeds of about 25MB/sec and was CPU bound the entire time. As soon as I realized that non-English characters do not play well with rsync(not sure if this is a limitation of rsync or the rsync server or Windows rsync client used) I had to find a better method for copying data. I wouldn't recommend rsync for this particular situation as I can't see any value added with regard to using rsync versus just copying the files.

If you DO have files with non-English characters and you wish to retain those characters you cannot use any of the above steps. Any files that have non-English names will have those characters replaced with question marks. Guess what OS can't use question marks in file names. That's right, Windows. All hell broke loose when I tried to access folders that had filenames containing question marks. Can't access the file, delete it, move it, anything. The only fix I could find was to delete the file from the command line in FreeNAS(yuk!). I used "Total Commander" as mentioned above(but the Windows drag-and-drop copy could have worked also). Total Commander is my recommendation(or another program like it) because it supports resuming copying versus having to click "skip" a bunch of times to skip the files that are already copied. Total Commander also lets you do a copy from source to destination and only copy files that are bigger or smaller than the destination file.

3. Once the data was on the temporary FreeNAS server it was time to make sure my data was safe. For that I did a zpool scrub. It took just under 16 hours to do the complete scrub. After the scrub completed it was time to retire Windows and go to FreeNAS.

4. Installed FreeNAS and did the setup. In particular I made sure the SMART and email functions were properly setup since this was going to be the intial fill of the zpool. If a disk was going to start acting up I wanted to know while it was copying. Lots of writing would be going on so now is a good time to find any potential problems. I had no problems with the drives but I wasn't about to take any chances.

5. The ZFS replication was flawless. I setup ZFS replication and dumped the snapshot to my new FreeNAS server. It ran at a solid 800Mb/sec(100MB/sec) the entire time. I figure there is some hard limitation with exceeding that speed with replication. Perhaps it is coded to not use up 100% of the network bandwidth to prevent the server from DoSing itself while replicating. ZFS replication can properly handle non-English characters and performed faster(and easier!) than the previous data copying. I have only praises for the ZFS replication. After the replication finished I deleted the snapshot at the destination server(yes.. your data does stay after the snapshot is removed). Note that you have no way that I could find that would tell you exactly what % of complete the replication is at but if you know you are copying 10GB of data you can do a 'zfs list -o' at the destination server's command line and see how "full" the zpool is. The FreeNAS GUI will be all messed up with regards to the size of the zpool and how full it is. My zpool was less than 1MB while the replication was in progress. You just have to wait it out. Remember that if you make changes to your files after the snapshot is take those changes will not replicate to the server unless you do another snapshot and replication.

6. Run a final scrub. I ran a good ol' scrub on the zpool where the data is now at just to see if any drives were going to start acting up. If any drive was on its way out I figured I'd get lots of errors listed under 'zpool status' or emails about SMART problems. Since I had 2 full copies of the data at this step if a bunch of drives were going to fail I'd really like to know right now. Fortunately, I had no problems at all.

Now for some other tidbits:

1. My RAID controller(Areca 1280ML) was converted from RAID mode to JBOD mode for FreeNAS. It has a 2GB cache on-controller with battery backup and the entire machine is on an UPS. Strangely, I found that performance of the zpool increased by more than 3 fold by disabling the read-ahead and write cache on the RAID controller itself using the Areca CLI. I also saw a similar performance boost on the temporary FreeNAS server when disabling all cache and read-ahead options on the Highpoint RocketRAID 3560. This really only matters for scrubs since leaving the cache enabled still gave me almost 290MB/sec which exceeds the dual Gb NICs the machine has. Scrubs will now take less than a day versus 2-3 days which is always a good thing.

2. ZFS replication is awesome. It was very easy to setup despite never using it before just by following the FreeNAS manual. Now that I've seen how great snapshots are I think I'm going to start implementing them on my new server.

3. Those non-English characters can screw you up! Be careful of them. If you even think you might have one you should assume you do. If you have a movie or TV collection you almost certainly have a movie with one of those odd characters that will bite you in the booty later. If this is a business environment I would definitely assume you have files with non-English characters since a lot of businesses deal with companies that are non-English as a primary language and may have slipped in a bad character here or there.

4. Doing direct cable link from server to server seems to be faster than going through a 24 port enterprise-class switch. So feel free to use a direct link and manual IPs if you are transferring very large amounts of data. It can shave off quite a bit of time. Use Intel NICs if possible. I tried using the Realtek onboard NIC on the temporary FreeNAS server and it performed at less than 60MB/sec until I plugged in the Intel NIC. No, LACP won't help you transfer data faster so don't even think about it.

5. Both of the FreeNAS servers had only 12GB of RAM despite having over 20TB of data. This would normally not be recommended but did not appear to ever pose a problem. I do have more RAM on order and will be upgrading the final server to 24GB in the next few days. Even on the final FreeNAS server I get over 100MB/sec copying data to and from my server despite being relatively starved for RAM considering the quantity of data.


I think that is about all I can come up with for info and advice. If anyone has any questions about this whole process post your question in this thread and I'll try to help you if I remember how I did something specific.

 

[JaimieV]

Thanks for the writeup - always interesting to see how things go. I may move my NAS pair over from rsync to replication now, too...

One thing that I don't think got mentioned: your experiments with running cp's (and other things excepting rsync) in parallel probably didn't show a speed up because you were using CIFS for the mount, which is itself single-threaded and CPU-bound.

I've used rsync and mount+cp to copy files with non-ASCII characters from Macs to Linux and FreeBSD, AIX to HPUX, and various other combinations and never had any issues. I've either been incredibly lucky or this is primarily an "idiot Windows doing things proprietorily" problem.

Memory won't have been a problem because you've been reading files linearly. RAM helps as cache more than anything else, and linearly reading 20Tb is continuous flushthrough. TBH I'd be very surprised if 12gig wasn't enough to handle the 20Tb, there's usually a non-linear relationship between cache size and cached data size where increasing cache fades in utility over time. And the gig per Tb rule of thumb will have been written back when gigs of RAM were expensive, relative to Tb's. And we can only hope that ZFS's cache use is considerably better than either of your RAID cards! Might be worth a test.