NTFS disk import - no progress?

[ninfanti]

Hello everyone, just created a TrueNAS server and am importing the contents of an NTFS disk which contained my Windows backups (about 1.5TB). After an hour task manager shows progress as 0%.

However, am seeing the used space in my pool continue to climb. I'm assuming I should ignore the task manager and that the import is proceeding properly?

Thank you,
Nick

 

[gadgetbazza]

Same here. New TrueNas setup and currently importing 8TB with no idea of progress other than refreshing the available space on the destination drive nor any event that I can see when it started. Best thing I've found is to use the "Reporting" to look at the disk throughputs so that I can see that something is still going on! I've googled and found a number of closed threads reporting the same thing.

Also had the issue with docker pulls timing out on anything of any size including many of the included chart apps. I've moved away from a QNAP NAS because of it's awful software and had seen good feedback on TrueNAS, but in my first 24 hours, I'm not that impressed, fingers crossed my experiences will improve!

 

[ChrisRJ]

If we are strictly talking about a pool import here, this cannot be done with NTFS in my understanding. A pool import is about making an existing ZFS pool (usually created on another machine) known to your TrueNAS.

@gadgetbazza , sorry to hear that your disappointed. May I ask what feedback you are referring to? If you are looking for a 1:1 replace for QNAP, then TrueNAS may indeed not be your cup of tea. There is, unfortunately, a lot of shall we say misleading information out there, esp. on YouTube. For more information I suggest you have a look at the "Recommend readings" (blue button) in my signature.

 

[gadgetbazza]

If we are strictly talking about a pool import here, this cannot be done with NTFS in my understanding. A pool import is about making an existing ZFS pool (usually created on another machine) known to your TrueNAS.

@gadgetbazza , sorry to hear that your disappointed. May I ask what feedback you are referring to? If you are looking for a 1:1 replace for QNAP, then TrueNAS may indeed not be your cup of tea. There is, unfortunately, a lot of shall we say misleading information out there, esp. on YouTube. For more information I suggest you have a look at the "Recommend readings" (blue button) in my signature.

Thanks for the reply Chris.

So it's not a pool import, but using the "Import Data" button in the Datasets page. This allows you to pick a disk, in my case a USB3 NTFS drive attached with the content from the old NAS backup.

I'll take a read of your recommended readings, many thanks for that. The feedback I've been using has been largely from Home Automation forums and YouTube, so I maybe have been misled. I was not expecting TrueNas to be a 1:1 replacement, I understand that the QNAP has a much more user friendly front end, but it has many shortcomings and I ma hoping TrueNas to be a more professional grade product, even if it is harder to get to grips with. I'm a software programmer by trade (although getting a bit long in the tooth from a coding perspective now!) and have had many dealings with infrastructure etc over the years, but not really much in the linux/unix arena, so my experience in that area is more hobby grade (I'm running a few Raspberry PI and Docker type solutions).

Thanks again for the reply, I will read the documents you've linked.

 

[anodos]

If the goal is to ingest NTFS data into a zpool, I think the most sure-fire way to ensure that it gets copied correctly is to create an SMB share and then use a tool like robocopy to migrate it over via a Windows SMB client. This ensures that some more obscure features like alternate datastreams are written to local ZFS storage in a way that will be accessible to windows clients.

 

[gadgetbazza]

Can’t help but feel this is backwards…. Can’t plug a drive into a nas device and copy data off of it, recommendation is to copy it across the network???

It doesn’t matter now, it’s finished copying, I can tell by the disk I/o, but at the time no idea how long it was going to take or if it was running.

Of course I also have no way of knowing for sure if everything copied safely so I may well have to do yet another pass of transferring this data.

I’m just surprised that such a basic feature isn’t more well catered for on a Nas platform?

 

[jgreco]

Can’t help but feel this is backwards…. Can’t plug a drive into a nas device and copy data off of it, recommendation is to copy it across the network???

Yes. I believe that's the standard advice for at least the NetApp, EMC, and EqualLogic gear I've worked with in the past. Unofficially I believe you can plug an external USB disk into Synology or QNAP gear, mostly because they are based on Linux, and support a variety of commonly used filesystems such as EXT3 and NTFS. However, for the Windows stuff, on-host copies do not work well if you're copying more than just plain files.

If you have a better suggestion as to how to properly seed obscure NTFS stuff into TrueNAS without using Samba, we'd all be interested in hearing about it.

I’m just surprised that such a basic feature isn’t more well catered for on a Nas platform?

In order for it to be deemed a "basic feature", I'd expect it to be present on most other NAS platforms. It isn't.

 

[ChrisRJ]

I’m just surprised that such a basic feature isn’t more well catered for on a Nas platform?

Apart from other arguments, one main reason from my perspective is the target group for TrueNAS/ZFS. ZFS was developed in the early to mid 2000s for mission-critical filers for large enterprises. And TrueNAS' target group is more or less similar. Prices have come down over the last 15-20 years. But in general we are still talking about companies that spend at least a high 5-digit amount of money, up to 7 digits.

I must admit that, completely irrespective of all the arguments made, I have always seen the SMB path as the easiest and somewhat obvious approach. Just like I use large USB drives connected to a Win10 machine for backups (in addition to other things), rather than connecting those drives to TrueNAS.

 

[gadgetbazza]

Why are you focusing on the NTFS part? Attaching a USB drive and importing data is a feature of TrueNAS, my post is pointing out that the user experience of doing this is very poor.

Instead of trying to defend the product, why not bear in mind it’s something that is advertised as open source and free and therefore not only targeted at large commercial clients with tens of thousands to invest but also to home users with larger storage requirements and that my open point that this *feature* is somewhat lacking.

I just don’t accept that the only way to get data efficiently onto and off the Nas device is via the network port? I can’t be the only one?

 

[gadgetbazza]

Yes. I believe that's the standard advice for at least the NetApp, EMC, and EqualLogic gear I've worked with in the past. Unofficially I believe you can plug an external USB disk into Synology or QNAP gear, mostly because they are based on Linux, and support a variety of commonly used filesystems such as EXT3 and NTFS. However, for the Windows stuff, on-host copies do not work well if you're copying more than just plain files.

If you have a better suggestion as to how to properly seed obscure NTFS stuff into TrueNAS without using Samba, we'd all be interested in hearing about it.



In order for it to be deemed a "basic feature", I'd expect it to be present on most other NAS platforms. It isn't.

Plugging an external usb isn’t *unofficially* supported, it’s even built in with a simple one touch button to copy the data!

My point is that this is a feature of TrueNAS, on the Dataset page, you click the Import Data button, select the disk to import from and the format (and there’s a number of them to choose from, not just NTFS) and you can import the data.

When you do this you get a pop up that shows you progress which in my case never moved from 0% on a 10Tb import. When the user interface times out you can retrieve the same pop up from the tasks window again it never moved from 0% and there is no way of me knowing if it worked fully or not other than to assume that the 12 hours of high disk I/o is representative of the disk copy?

If you can’t identify that I’ve done something wrong here other than try to use the feature that is present, then I stand by my post that this feature needs some work.

I will likely have to accept that the only way I will ever know if it worked properly is to run a robocopy or rsync task to check it and fill and gaps.

 

[jgreco]

Instead of trying to defend the product, why not bear in mind it’s something that is advertised as open source and free

Fine. Please feel free to submit your patches to implement this feature, in the finest tradition of free software projects. Having worked in the free software community for several decades, I feel that expecting other people to write the feature you want written is incredibly rude and selfish.

 

[Ericloewe]

Why are you focusing on the NTFS part?

Because NTFS is the only "problem" here, as the only filesystem in widespread use outside of SD/CF cards that has historically been challenging for operating systems not named "Windows" due to a combination of obscurity of some implementation details and licensing.
Beyond that, the import feature is expected to work.

 

[jgreco]

Plugging an external usb isn’t *unofficially* supported, it’s even built in with a simple one touch button to copy the data!

Simply not true. What we're talking about here is not just simply copying the data, but doing so in a manner that allows some of the more obscure stuff involved to work correctly. If @anodos suggests that this is best done via SMB with Robocopy so that Samba can properly serve that data back out, then that's a powerful bit of helpful advice. I think most people who have tried to do more than a trite import of basic files using SMB or AFP have discovered that the resulting copies are problematic because of the propensity of the protocols to support weird stuff like "dual-forked" files that do not have a straightforward UNIX analog, or SMB with its complicated AD and ACL interconnections.

 

[anodos]

Simply not true. What we're talking about here is not just simply copying the data, but doing so in a manner that allows some of the more obscure stuff involved to work correctly. If @anodos suggests that this is best done via SMB with Robocopy so that Samba can properly serve that data back out, then that's a powerful bit of helpful advice. I think most people who have tried to do more than a trite import of basic files using SMB or AFP have discovered that the resulting copies are problematic because of the propensity of the protocols to support weird stuff like "dual-forked" files that do not have a straightforward UNIX analog, or SMB with its complicated AD and ACL interconnections.

Right, the GUI import stuff just mounts and relies on ntfs-3g here to expose the NTFS metadata in a way that it is readable by the kernel. Then that info is rsynced to ZFS.

1) NTFS -> ntfs3g -> rsync -> ZFS

If you go through SMB share you get this:
2) NTFS -> Windows SMB client -> SMB protocol -> Samba -> ZFS

Since the data is now sitting on a NAS and you presumably want to read it back at some point, it will go through:
3) ZFS -> Samba -> SMB protocol -> Windows Client -> NTFS

The symmetry between (2) and (3) makes it more reliable. You're guaranteed to have everything written in a way that can be retrieved correctly by the client. This is, in general, the way data migrations should happen (protocol to protocol). If you're creating an NFS export, use an NFS client to do initial data transfer (though with NFS this is less of an issue I suppose).

The obvious exception would be ZFS replication (but even then planning needs to be done to ensure that you keep the same share configuration on the receiving side).

I'm trying to favor correctness over other factors (which is I think more in-line with generally what the community here prefers).

Though if you're planning to user AFP, stop that and switch to SMB. :)

 

[jgreco]

*barf*. (not meant in any way a criticism of your stuff, @anodos , you work in a horrifying field of weird Microsoft stuff, which is where I place the blame).

Let's standardize on NFSv3

 

[anodos]

*barf*. (not meant in any way a criticism of your stuff, @anodos , you work in a horrifying field of weird Microsoft stuff, which is where I place the blame).

Let's standardize on NFSv3

Well, it's not like we've never had compatibility issues between Linux and FreeBSD, and don't get me started on symlnks.

 

[gadgetbazza]

Fine. Please feel free to submit your patches to implement this feature, in the finest tradition of free software projects. Having worked in the free software community for several decades, I feel that expecting other people to write the feature you want written is incredibly rude and selfish.

Well I guess "Resident Grinch" seems appropriate. Nice welcome to a new user of a forum.... and from a moderator too!

Submitting code isn't the only part of software / product development, there are many facets. There are also many others with decades of experience.

Hopefully you can review again and read my feedback for what it is was (constructive criticism) and maybe it won't receive any further rude replies?

Because NTFS is the only "problem" here, as the only filesystem in widespread use outside of SD/CF cards that has historically been challenging for operating systems not named "Windows" due to a combination of obscurity of some implementation details and licensing.
Beyond that, the import feature is expected to work.

Are you saying the same experience doesn't occur when using the Disk import feature if the data on the USB drive was in MSDOS / UFS or EXT2? If so, then I would have been better choosing another format, the data backup only contained basic files without any need to transfer permissions. I don't have any Windows systems here, and therefore didn't use any Windows clients at any part of the process, maybe I just made a poor decision with the data transfer drive format, but it seems I've stirred up a hornets nest.

Simply not true. What we're talking about here is not just simply copying the data, but doing so in a manner that allows some of the more obscure stuff involved to work correctly. If @anodos suggests that this is best done via SMB with Robocopy so that Samba can properly serve that data back out, then that's a powerful bit of helpful advice. I think most people who have tried to do more than a trite import of basic files using SMB or AFP have discovered that the resulting copies are problematic because of the propensity of the protocols to support weird stuff like "dual-forked" files that do not have a straightforward UNIX analog, or SMB with its complicated AD and ACL interconnections.

Not sure I can agree on "Simply Not True", the QNAP NAS box does support USB host attached disks for data import / export, as I mentioned there is even a "one touch" hardware button on the front of the unit to facilitate this exact process. Maybe the QNAP isn't considered a NAS in this community?

I am only trying to copy around 10Tb volume of CCTV clips and snapshot images with no AD / ACL data involved. This data is used by a single host application to present the information, so I only need to add one user account to the new location to make this available, I therefore have no forced dependancies on protocols but needed to select a format for the data to go onto the USB drive.

Right, the GUI import stuff just mounts and relies on ntfs-3g here to expose the NTFS metadata in a way that it is readable by the kernel. Then that info is rsynced to ZFS.

1) NTFS -> ntfs3g -> rsync -> ZFS

If you go through SMB share you get this:
2) NTFS -> Windows SMB client -> SMB protocol -> Samba -> ZFS

Since the data is now sitting on a NAS and you presumably want to read it back at some point, it will go through:
3) ZFS -> Samba -> SMB protocol -> Windows Client -> NTFS

The symmetry between (2) and (3) makes it more reliable. You're guaranteed to have everything written in a way that can be retrieved correctly by the client. This is, in general, the way data migrations should happen (protocol to protocol). If you're creating an NFS export, use an NFS client to do initial data transfer (though with NFS this is less of an issue I suppose).

The obvious exception would be ZFS replication (but even then planning needs to be done to ensure that you keep the same share configuration on the receiving side).

I'm trying to favor correctness over other factors (which is I think more in-line with generally what the community here prefers).

Though if you're planning to user AFP, stop that and switch to SMB. :)

So that's a fair explanation, thanks for that. However, referring to my actual issues, they aren't with NTFS at all, in fact the import appears to have worked, although I still haven't performed a reconciliation check using an SMB client as yet - and I'd rather not have to. Oh and no AFP here.

If we can get back to the original issue, it's not that it doesn't necessarily work, more that the experience doesn't lend itself to allow a user to understand whether it works and the lack of status.

The feedback thus far it seems is that the approach I've taken isn't supported, or good practice and doesn't feature on NAS devices according to @jgreco. But the reason for my discussion is that this doesn't feel like a good experience for the following reasons (I'm trying to clarify my points here into less disputable items).

1. The feature is supported, if it wasn't, it wouldn't exist in the product, it's clearly there in the menus and buttons that I used within the TrueNas interface.

2. The problem is with the user experience. Simply put, once the copy is started, there is no user friendly way of knowing the status of the copy (the progress bar never changed for me even after going back into the jobs menu after a number of hours), either in terms of progress or completion with success or failure. If the process uses rsync, then maybe capturing and making available the log for this could be a good approach to closing this gap?

 

[Ericloewe]

Are you saying the same experience doesn't occur

Well, that was a comment from something of a misread of the thread - it looked like the import hadn't actually worked. Since NTFS has always been a bit on the tricky side, I wouldn't be surprised to see other filesystem behave better. But clearly that's not the problem here.

It's reasonable to want better feedback out of this and it does show that TrueNAS has improved quite a bit over the years. Not that long ago, pretty much all things were similarly light on feedback, with the added bonus of blocking the webGUI.
Unfortunately, since it's not a very popular feature, I don't see the topic gaining much traction - bigger fish to fry, return on investment, etc.

 

[gadgetbazza]

It's reasonable to want better feedback out of this and it does show that TrueNAS has improved quite a bit over the years. Not that long ago, pretty much all things were similarly light on feedback, with the added bonus of blocking the webGUI.
Unfortunately, since it's not a very popular feature, I don't see the topic gaining much traction - bigger fish to fry, return on investment, etc.

And that's fair enough. I wasn't expecting it all to be put into a pretty web portal presentation to be honest. I'm quite happy to dig into log files etc. but I felt that I was unable to ascertain what was going on and the only choice I had at the time was to leave the NAS alone for 12 hours and hope.

 

[jgreco]

Well I guess "Resident Grinch" seems appropriate. Nice welcome to a new user of a forum.... and from a moderator too!

My role as a moderator has little to do with this. Unlike that QNAP or Synology product that you paid hundreds or even thousands of dollars for, TrueNAS is giving you a product free to use, and I simply think your attitude seems a bit entitled and offensive. There are a lot of times in computer science when "one-shot use" features are not outfitted with maximal bells and whistles.