Chunk Exceeded Limit Error when Importing NTFS Drives

[Dizzy49]

Running TrueNAS Scale 22.12.3.3

I have several 3TB and 4TB NTFS drives that I am trying to import and so far only 1 of the 5 drives have worked. The other 4 I get an error of "Error: Separator is not found, and chunk exceeded the limit".

A previous post indicated that it was an issue with a 100GB limit. I hope not because a 100GB limit on imports is kind of pointless, however, the 1 drive that worked only had 45GB of data, and the ones that didn't had 103GB, 365GB, 395GB, 435GB so there may be something to that.

It looks like all of them imported the folder structure. The 103GB looks like it may have imported data, but I get "permission denied" when I try to access any of them via shell.

I thought I could use chmod and change the permissions. I tried:
chmod -R 777 Video

I receive the following error:
chmod: changing persmissions of 'Video': Operation not permitted
chmod: cannot read directory 'Video': Permission denied

I went to the shell and tried to mount the drive directly with:
mount -t ntfs /dev/sde2 /old

I received the following error:
Error opening ready only 'dev/sde2': Permission denied
Failed to mount 'dev/sde2' : Permission denied

The other 3 drives did not import any data, but I am unable to mount any of them either.

Yes, I CAN mount them in another machine and copy the data over. For these it isn't a big deal. The problem is that doing so isn't very practical for the remaining 100TB of data on the other drives I need to import. Doing so over the network will take like a week!

Anyone have some other suggestions? If I can atleast get them mounted I can rsync the data over.

Thanks!

 

[NickF]

I'm going to ask a relatively stupid question here. I'm sorry in advance.
I get that you have 5 drives, and I get why this might be annoying. But consider the time you have spent thus far, and the time it took you to reach out here...

Have you tried plugging them into a Windows computer? Do they work there? If so, just copy the data over to the TrueNAS over SMB?

If that's an option..do that. If for some reason that's not an option, or the drives ALSO don't work on native Windows...feel free to @ mention me here and I'll answer whenever I can.

 

[winnielinnie]

If so, just copy the data over to the TrueNAS over SMB?

I second this.

 

[Dizzy49]

I'm going to ask a relatively stupid question here. I'm sorry in advance.
I get that you have 5 drives, and I get why this might be annoying. But consider the time you have spent thus far, and the time it took you to reach out here...

Have you tried plugging them into a Windows computer? Do they work there? If so, just copy the data over to the TrueNAS over SMB?

If that's an option..do that. If for some reason that's not an option, or the drives ALSO don't work on native Windows...feel free to @ mention me here and I'll answer whenever I can.

@NickF @winnielinnie
I get that, and it's a fair question.

Yes, I CAN mount them in another machine and copy the data over. For these it isn't a big deal. The problem is that doing so isn't very practical for the remaining 100TB of data on the other drives I need to import. Doing so over the network will take like a week!

I will prob do just that for these little guys, but the other 100TB is the real issue. Importing is quite a bit faster, and I don't exactly have 2 weeks straight to copy the files across the network.

 

[NickF]

I guess I'm not understanding the workflow here. For a few drives kicking around is one thing...like I said I get your concern.

I also misread while skimming...sorry
To now clarify: SEPERATELY, You have X number of other disks with about ~100TiB of data?

That being the case now, I'm just plain confused by your workflow.
These disks were in system(s) which wrote that data to them. What happened to those systems? How did you get in this situation and what resources do you have? This isn't as much a TrueNAS question as it is a general sysadmin question, and I'd be happy to answer with my sysadmin hat on.

 

[winnielinnie]

You can saturate a 1 Gbe connection over SMB at around 110 - 115 MB/s.

How fast exactly is the local copy from the HDD directly? How much time are you practically saving? (The clock has been running as you try to figure this out.)

 

[Dizzy49]

@NickF
I have a Windows NAS that I have been using for many years now. I have 4x 18TB, 5x 16TB, 3x 10TB, and the handful of random 3-4TB drives and I used Drivepool for them. Across the 10-18TB drives there is 106.2TB of data. Most of it videos, prob 2tb of music, 75tb of porn :P (kidding). I run am Emby server, have a half dozen containers, etc on it.

I picked up a R740 and am migrating to TrueNAS Scale. I have 3x 5 wide RAIDZ1 with 1TB SSDs. I have our docs and photos on it as well as other important things. This will also be app data storage, vms, etc.
I have a 5 wide RAIDZ2 with 20TB drives that I want to migrate the data from 18TB drives, and then create a RAIDZ2 with the 18TBs and add it to the pool and finish copying the data over. I will then add the 16TBs to the pool.
The 10, 4 and 3 TBs will go into another shelf as some additional backup.

@winnielinnie
It took me 1hr 18min to copy 375GB. That's about 78 mb/s, so 106TB would take 15.7 days
Even if I get only 110 mb/s locally that knocks off 4.5 days.

 

[winnielinnie]

It took me 1hr 18min to copy 375GB. That's about 78 mb/s, so 106TB would take 15.7 days

Locally? If that's the case, then why are you insisting on doing a local transfer? You can hit those speeds over SMB anyways. Your bottleneck is the spinning HDD itself.

If you want on-the-fly integrity checks and resume support, you can opt for rsync over the network, instead of copying over SMB.

 

[Dizzy49]

Locally? If that's the case, then why are you insisting on doing a local transfer? You can hit those speeds over SMB anyways. Your bottleneck is the spinning HDD itself.

If you want on-the-fly integrity checks and resume support, you can opt for rsync over the network, instead of copying over SMB.

No, that was with the drive on my Windows machine and copying it across the network to the TrueNAS dataset

 

[winnielinnie]

Being that a 7200-RPM HDD yields from 80-120 MB/s sustained reads, I don't think you'll gain speed from a local copy. (Not to mention a 5400-RPM drive is even slower than that.)

I could see your reason for connecting it locally if this was an SSD or NVMe.

A proper 1 Gbe network, especially for sequential transfers, isn't limited to 78 MB/s. (You would likely see it capping around 110 - 115 MB/s if the network was the bottleneck.) That's why I believe it's your spinning HDDs themselves that are the bottleneck.

Spinning drives can only read your files as fast as they can, no matter if it's pure SATA or over a network.

 

[NickF]

Yeah - gotta say man. The amount of time we chatted here could have been better spent just firing up the old server and kicking off the migration. I don't see any real urgency in resolving the issue either. My genuine advise, The best things in life come with patience.

 

[Dizzy49]

Yeah - gotta say man. The amount of time we chatted here could have been better spent just firing up the old server and kicking off the migration. I don't see any real urgency in resolving the issue either. My genuine advise, The best things in life come with patience.

The urgency is that my wife is gone for a week so I have a small window to get stuff moved, set back up and tested so she has no idea what happened. We have an understanding, I can do whatever the hell I want as long as it doesn't mess with her stuff (ie apps, files, shows, movies, ebooks, music, audiobooks, etc)