Download speed from FreeNAS machine dropping steadily

[AndyB78]

Hello,

I have for more than one year now a FreeNAS machine. Each Friday I have to download through FTP/Gbit Ethernet about 1.5TB of data. Until last week, the download would be done at roughly 75MB/s. For some reason, since last week, the speed starts high (80MB/s) and it steadily/constantly decreases down to about 3.5-4MB/s in about 12 hours. No change has been made since 2 weeks ago when the speed was OK, not even an update, no configuration update. No usable information is present in /var/log/messages.

Configuration:
Build FreeNAS-9.3-STABLE-201602031011
Platform Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
Memory 16349MB
HDDs: 2 x WD RED (4GB and 3GB) and one 0.5GB consumer grade HDD mounted in a span (so in total almost 7TB of harddisk space in one span)

What could be the problem? Why did the problem occur without any change from my side? What can I check?

Thanks!

 

[m0nkey_]

one 0.5GB consumer grade HDD mounted in a span

Do you mean stripe? If so, that could be bad.

 

[pschatz100]

What about the machine you are downloading to? Did you check there were no problems at that end?

 

[AndyB78]

Do you mean stripe? If so, that could be bad.

Yes, sorry, I mean a stripe. But if it's bad why is the speed decreasing so steadily over such a long time? Shouldn't the speed be very bad all the time or fluctuate erratically?

What about the machine you are downloading to? Did you check there were no problems at that end?

I have tried downloading to 2 different machines both on external HDD (the same USB3 type of HDD, 3 different HDDs) and internal HDD. I really don't think it's the machines that I am downloading to.

 

[AndyB78]

I have looked into Disk reporting and while ada1 and ada2 manifest a similar behaviour, ada0 looks totally different. It doesn't even show any reads, it only shows writes and it's into tens of KB/s area. Does this mean that ada0 is bad?

By the way, ada0 is a 0.5GB ST Barracuda and ada1 and ada2 are WD REDs.

If so, can I replace it with another larger HDD (3-4TB) and use all the space of the new HDD?

Thanks!

 

[SweetAndLow]

Do a dd read test locally on the box and see what speeds you get. Something like dd of=/dev/null if=/some/big/file

 

[pirateghost]

Please use proper terminology.

0.5GB would be around 512 MEGABYTES...for a hard drive?? what year is this?

You have multi-TERABYTE drives...TERABYTE is abbreviated as TB

MB=megabytes
GB=gigabytes
TB=terabytes

 

[AndyB78]

Do a dd read test locally on the box and see what speeds you get. Something like dd of=/dev/null if=/some/big/file

I did and with surprising results. The smallest result I got, was about 124MB/s but normally around 170MB/s with tarballs ranging from 160MBs to nearly 10GBs:

165523945 bytes transferred in 0.857375 secs (193059003 bytes/sec)
534042169 bytes transferred in 3.525185 secs (151493379 bytes/sec)
3996674529 bytes transferred in 22.685053 secs (176180966 bytes/sec)
6396395785 bytes transferred in 34.842612 secs (183579686 bytes/sec)
9481771167 bytes transferred in 51.774419 secs (183136215 bytes/sec)

So does this mean that the HDDs are fine? Thanks!

 

[Jailer]

So does this mean that the HDDs are fine?

It's going to be impossible to tell without any information on them.

Do you have scrubs scheduled on your pool? Do you have smart testing scheduled for your drives? Have you ever run any smart tests on your drives or run a scrub on your pool?

 

[SweetAndLow]

After you show some information about your drives you should also do a iperf test from your client to freenas.

 

[nightshade00013]

I did and with surprising results. The smallest result I got, was about 124MB/s but normally around 170MB/s with tarballs ranging from 160MBs to nearly 10GBs:

Using DD can give skewed results since in all likelihood the pool has compression turned on. A file that is more compressible will read a lot faster than one that is not.

Right now your config is so confusing and out of whack due to having multiple different drives which also makes it hard to troubleshoot due to who knows what is setup. So not to step on any toes but lets see some output from:

smartctl -a /dev/ada0
smartctl -a /dev/ada1
smartctl -a /dev/ada2

zpool status

zpool list -v

That should give us quite a bit of info about the health of the drives and pool. Hopefully you have went through and checked cables (unplugged and replugged power and data cables not just looked.)