Downloaded file is bigger than placed on the storage

[serafio]

Hi all,

I have TrueNas Core 13.0-U5.3
Working on not the servers hardware (i5-7500, ddr4 16 GB (not ecc)), intel x520 SFP adaptor)
My system nas two m2 ssd Crucial p3 into stripe (512GB*2 = 1 TB SSD raid)
I used it for ftp needs.
Usualy about 60-80 different IP parallel connection under one account (content distribution)

Issue
Some clients (2-3%) after downloading files (its about 150 GB, 5 files about 25 GB, 5 files about 1-2 GB, 4-6 files about 10 KB) have files bigger that files on the ftp
Usually it happens with biggest files
different is small - 1-10 KB
If they try to resume downloading - doesn't help, the same size
delete and download agin - doesn't help, the same size
BUT! If they try to download in a different folder - 98% to have correct file
Clients usually use Filezilla

I think it happens when they use "resume download"

I tried to add:
DefaultTransferMode binary

Let's see what happens

 

[winnielinnie]

If they try to download in a different folder - 98% to have correct file

Correct file? Do you mean that the previous "completed" downloads were corrupted?

 

[serafio]

for example
On the storage a file has 10100100 KB (~10GB)
Client after downloading has on his side 10100110 KB
System can not accept file - because size is incorrect (hash too)
But after this incident, client should download again file from the storage in a different place on his side - and a magic - file will has 10100100 KB

 

[Ericloewe]

I don't want to be a downer, but FTP really is one of those things best forgotten. For distribution purposes, HTTP is typically good enough (consider making it easier on your users than "download these 25 different directories with hundreds of tiny files each"). For more serious file transfers with authenticated users, consider using SFTP.

 

[Whattteva]

for example
On the storage a file has 10100100 KB (~10GB)
Client after downloading has on his side 10100110 KB
System can not accept file - because size is incorrect (hash too)
But after this incident, client should download again file from the storage in a different place on his side - and a magic - file will has 10100100 KB

Sounds like corruption happens in-flight. Possibly bad network card or RAM on the client side?
Your next step in figuring this out is, is it isolated just to this one client or are multiple clients experiencing this? If it's isolated to that one client, the problem is most likely on the client side.

 

[serafio]

If my knowledge is correct...
FTP is more efficient at transferring large files, whereas HTTP is better for transferring smaller files such as web pages
SFTP is slower than FTP(FTPS) - due to the security built into the protocol.

BUT I would like to change my distribution from FTP to something better
I use at work Aspera, Gofilex, Signiant - but it so expensive for our tasks

 

[serafio]

This mistake happening only with 5-6 clients
They even tried to redownload 4-5 times - no result
but when change destination directory - it worked right away

I thought about some sync solution, like Nextcloud
But I dont have enough experience
I have about 150 clients and each deliveries for some clients, not whole
I should do "Share group" for each deliveries
But it doesn't protect me from a similar mistake with the size difference

 

[Whattteva]

but when change destination directory - it worked right away

Can you elaborate further on what you mean by this?

 

[Patrick M. Hausen]

Use binary mode instead of ASCII mode for FTP. ASCII mode will e.g. convert Unix LF into Windows CR/LF if the server is Unix and the Client is Windows.

Even if the file is not a text file FTP in ASCII mode will replace every byte with the value decimal 10 with two bytes with the values decimal 13 and 10.

So, corruption in flight - most probably. But not in some curious way but intentional. And setting the server to default mode binary won't help if the client asks for ASCII - which is still the default for many FTP clients.

 

[Whattteva]

Use binary mode instead of ASCII mode for FTP. ASCII mode will e.g. convert Unix LF into Windows CR/LF if the server is Unix and the Client is Windows.

Ah didn't think of that, that would make sense why the files are bigger. It effectively adds an extra byte for every new line for Windows clients.

Even if the file is not a text file FTP in ASCII mode will replace every byte with the value decimal 10 with two bytes with the values decimal 13 and 10.

So, corruption in flight - most probably. But not in some curious way but intentional. And setting the server to default mode binary won't help if the client asks for ASCII - which is still the default for many FTP clients.

I'm not sure if that's the case though. He mentions the clients use FileZilla. I have never had to switch the modes when I use FileZilla and never had corruption, so I'd assume the default is binary mode. It's possible that OP's clients set this somehow unknowingly though.

 

[Ericloewe]

Use binary mode instead of ASCII mode for FTP. ASCII mode will e.g. convert Unix LF into Windows CR/LF if the server is Unix and the Client is Windows.

Even if the file is not a text file FTP in ASCII mode will replace every byte with the value decimal 10 with two bytes with the values decimal 13 and 10.

So, corruption in flight - most probably. But not in some curious way but intentional. And setting the server to default mode binary won't help if the client asks for ASCII - which is still the default for many FTP clients.

This sort of weirdness is why I hate FTP.

If my knowledge is correct...
FTP is more efficient at transferring large files, whereas HTTP is better for transferring smaller files such as web pages
SFTP is slower than FTP(FTPS) - due to the security built into the protocol.

BUT I would like to change my distribution from FTP to something better
I use at work Aspera, Gofilex, Signiant - but it so expensive for our tasks

HTTP has seen tremendous amounts of work over the past decades. I have not benchmarked the two head to head, but I have a hard time believing, in the absence of data, that HTTP is meaningfully slower than FTP.
Both have pain points (firewalls for FTP; the absurd amount of request types and their nuances when configuring clients and servers for HTTP), so it's hard for me to really agree that FTP is advantageous in some way.

SFTP is slower than plain FTP, on a theoretical basis. The ubiquity of encrypted communications, including HTTPS, strongly suggests that it's not a meaningful factor.

I have some experience dealing with weird proprietary transfer systems, and I'm not left longing for them.

 

[serafio]

Can you elaborate further on what you mean by this?

for example:
client is donloading files to volume/user/data/package_01
And of course downloaded files can be corrupt
All resume downloading will not successful, if client use the same folder
But if he use volume/user/data for downloading - file with 95% will correct
After this - client move "corercted" file with replacing to volume/user/data/package_01 - magic - package is correct

 

[serafio]

This sort of weirdness is why I hate FTP.

HTTP has seen tremendous amounts of work over the past decades. I have not benchmarked the two head to head, but I have a hard time believing, in the absence of data, that HTTP is meaningfully slower than FTP.
Both have pain points (firewalls for FTP, the absurd amount of request types and their nuances when configuring clients and servers), so it's hard for me to really agree that FTP is advantageous in some way.

SFTP is slower than plain FTP, on a theoretical basis. The ubiquity of encrypted communications, including HTTPS, strongly suggests that it's not a meaningful factor.

I have some experience dealing with weird proprietary transfer systems, and I'm not left longing for them.

Thank you for your opinion