SOLVED Slow LAN speeds (via FTP) since upgrade

[joeschmuck]

So if I understand the results it takes longer to write the data than read it back. It sure does look low for writing data, about 52MB/sec. I would expect that to be a side affect of your RAID and hard drive combination. The reading speed is much better but still slow.

Just curious, have you tried to enable Autotune? If not then you are likely to see improvement since Autotune should reallocate your available RAM since you have 16GB of it. Make sure to reboot your NAS after enabling Autotune.

 

[BrianDMG]

Unfortunately, Autotune is already enabled. I run my 2 drives in a mirrored, striped ZFS setup. I'm kind of at a loss as to why the internal data would be moving so slow. Even the write is slower than I would have expected. My reports don't generally show a lot of stress on my system, either.

Also, I ran a second set of dd tests and came up with similarly uninspiring numbers.

 

[joeschmuck]

If you have a single or pair of hard drives you could install into your NAS for testing purposes, they don't need to be large or new, just something you could test different configurations with. I don't know why your internal speed is so slow either but I have to suspect the RAID you have established. You could also run a Smart test on your drives to ensure things are okay there as well. Just pick one thing to change, make the change and run the dd test.

Did you upgrade your Zpool to the latest version when you upgraded to 9.1.1? I have not and I doubt I will right now until I find out about comparability with other operating systems from a recovery of data standpoint. I'm curious if you made that change and it harmed you.

BTW, I did get iperf to run on a VM loaded with Ubuntu and it worked as I hoped, 939Mbits/sec throughput. I was unable to use the Ubuntu Live CD (Try) as it refused to let iperf install. I just loaded it on a new VM and installed iperf, she works great.

 

[BrianDMG]

I don't recall having changed anything about my Zpool when I upgraded to 9.1. Obviously, I had to re-import the volume, but that's all I did. I'll try the SMART tests, and if there's no problem there, I'll try it with some other hard drives. The only other awkward thing about my hard drive setup is that one of my drives is 3TB, and the other is 2TB (which I now know isn't ideal, but I didn't at the time) - however, I don't think that's the problem, because at some point not too long ago (ie right before I upgraded to 9.1) I was getting 60MB/s transfer speeds on exactly the same infrastructure, with nearly identical hardware.

 

[BrianDMG]

Alright, here's some casual afternoon reading for you (results of SMART tests):

smartctl -H
[root@DMGNAS ~]# smartctl -H /dev/ada0
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

[root@DMGNAS ~]# smartctl -H /dev/ada1
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

zpool status
[root@DMGNAS ~]# zpool status
pool: DMG state:
ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags.

scan: scrub repaired 0 in 6h38m with 0 errors on Sun Sep 1 13:38:23 2013 config:
NAME STATE READ WRITE CKSUM
DMG ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/be52c471-7995-11e2-8a74-001921167480 ONLINE 0 0 0
gptid/bee356f4-7995-11e2-8a74-001921167480 ONLINE 0 0 0

errors: No known data errors

zpool iostat -v
[root@DMGNAS ~]# zpool iostat -v
capacity operations bandwidth pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
DMG 1.24T 589G 17 28 1.97M 2.52M
mirror 1.24T 589G 17 28 1.97M 2.52M
gptid/be52c471-7995-11e2-8a74-001921167480 - - 8 22 1008 K 2.52M
gptid/bee356f4-7995-11e2-8a74-001921167480 - - 8 22 1005 K 2.52M

Interestingly, I also had this pop up (not the first time, but the first time I've noticed it with multiple article links) for my Seagate drive. Obviously, there is a firmware update available, but updating it (as I saw in another thread) seems like more trouble than it's worth.

smartd[38260]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Oct 2 16:47:07
DMGNAS smartd[38260]: see the following Seagate web pages:
Oct 2 16:47:07 DMGNAS smartd[38260]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
Oct 2 16:47:07 DMGNAS smartd[38260]: http://knowledge.seagate.com/articles/en_US/FAQ/223651en

Lastly, while looking for hard drive testing commands in the forum search, I came upon this:

Also, you should be aware that once the array starts to fill -- some say 80% full, performance will start to suffer.

If you look at the zpool iostat results, you'll notice I have a scant 589GB of free space left on my volume. Could this potentially be part of the problem?

ETA: Completely destroyed my formatting after I posted. Fixed for coherency.

 

[cyberjock]

Try doing smartctl -a /dev/ada0 and then again for ada1. The "-H" results don't tell you the whole story. ;)

 

[paleoN]

ETA: Completely destroyed my formatting after I posted. Fixed for coherency.

Do everyone a favor: stay away from the font colors and use [code][/code] tags which preserves CLI formatting.

Also:

Code:

zpool list

 

[BrianDMG]

My apologies about the font coloring. Here are the results of the tests you guys asked for, although the smartctl -a /dev/ada# tests seemed to scroll more information up-screen that I couldn't get back to (using GUI shell).

smartctl -a /dev/ada0

Code:

SCT capabilities:              (0x3085) SCT Status supported.                                                                   
                                                                                                                                 
SMART Attributes Data Structure revision number: 10                                                                             
Vendor Specific SMART Attributes with Thresholds:                                                                               
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                 
  1 Raw_Read_Error_Rate    0x000f  115  099  006    Pre-fail  Always      -      94976648                                 
  3 Spin_Up_Time            0x0003  094  093  000    Pre-fail  Always      -      0                                         
  4 Start_Stop_Count        0x0032  095  095  020    Old_age  Always      -      5672                                     
  5 Reallocated_Sector_Ct  0x0033  100  100  010    Pre-fail  Always      -      0                                         
  7 Seek_Error_Rate        0x000f  078  060  030    Pre-fail  Always      -      60482537                                 
  9 Power_On_Hours          0x0032  095  095  000    Old_age  Always      -      4633                                     
10 Spin_Retry_Count        0x0013  100  100  097    Pre-fail  Always      -      0                                         
12 Power_Cycle_Count      0x0032  100  100  020    Old_age  Always      -      93                                       
183 Runtime_Bad_Block      0x0032  100  100  000    Old_age  Always      -      0                                         
184 End-to-End_Error        0x0032  100  100  099    Old_age  Always      -      0                                         
187 Reported_Uncorrect      0x0032  100  100  000    Old_age  Always      -      0                                         
188 Command_Timeout        0x0032  100  100  000    Old_age  Always      -      0 0 0                                     
189 High_Fly_Writes        0x003a  094  094  000    Old_age  Always      -      6                                         
190 Airflow_Temperature_Cel 0x0022  072  052  045    Old_age  Always      -      28 (Min/Max 24/28)                       
191 G-Sense_Error_Rate      0x0032  100  100  000    Old_age  Always      -      0                                         
192 Power-Off_Retract_Count 0x0032  100  100  000    Old_age  Always      -      61                                       
193 Load_Cycle_Count        0x0032  097  097  000    Old_age  Always      -      6876                                     
194 Temperature_Celsius    0x0022  028  048  000    Old_age  Always      -      28 (0 15 0 0 0)                           
197 Current_Pending_Sector  0x0012  100  100  000    Old_age  Always      -      0                                         
198 Offline_Uncorrectable  0x0010  100  100  000    Old_age  Offline      -      0                                         
199 UDMA_CRC_Error_Count    0x003e  200  200  000    Old_age  Always      -      0                                         
240 Head_Flying_Hours      0x0000  100  253  000    Old_age  Offline      -      4364h+12m+49.083s                         
241 Total_LBAs_Written      0x0000  100  253  000    Old_age  Offline      -      194743832812                             
242 Total_LBAs_Read        0x0000  100  253  000    Old_age  Offline      -      28065542440                               
                                                                                                                                 
SMART Error Log Version: 1                                                                                                       
No Errors Logged                                                                                                                 
                                                                                                                                 
SMART Self-test log structure revision number 1                                                                                 
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                 
# 1  Extended offline    Interrupted (host reset)      00%      4632        -                                                   
# 2  Extended offline    Completed without error      00%      4602        -                                                   
                                                                                                                                 
SMART Selective self-test log data structure revision number 1                                                                   
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                     
    1        0        0  Not_testing                                                                                             
    2        0        0  Not_testing                                                                                             
    3        0        0  Not_testing                                                                                             
    4        0        0  Not_testing                                                                                             
    5        0        0  Not_testing                                                                                             
Selective self-test flags (0x0):                                                                                                 
  After scanning selected spans, do NOT read-scan remainder of disk.                                                             
If Selective self-test is pending on power-up, resume after 0 minute delay.

smartctl -a /dev/ada1

Code:

Short self-test routine                                                                                                         
recommended polling time:        (  2) minutes.                                                                                 
Extended self-test routine                                                                                                       
recommended polling time:        ( 401) minutes.                                                                                 
Conveyance self-test routine                                                                                                     
recommended polling time:        (  5) minutes.                                                                                 
SCT capabilities:              (0x3035) SCT Status supported.                                                                   
                                        SCT Feature Control supported.                                                           
                                        SCT Data Table supported.                                                               
                                                                                                                                 
SMART Attributes Data Structure revision number: 16                                                                             
Vendor Specific SMART Attributes with Thresholds:                                                                               
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                 
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0                                         
  3 Spin_Up_Time            0x0027  171  171  021    Pre-fail  Always      -      6416                                     
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      126                                       
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0                                         
  7 Seek_Error_Rate        0x002e  200  200  000    Old_age  Always      -      0                                         
  9 Power_On_Hours          0x0032  093  093  000    Old_age  Always      -      5149                                     
10 Spin_Retry_Count        0x0032  100  100  000    Old_age  Always      -      0                                         
11 Calibration_Retry_Count 0x0032  100  100  000    Old_age  Always      -      0                                         
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      126                                       
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      78                                       
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      47                                       
194 Temperature_Celsius    0x0022  123  105  000    Old_age  Always      -      27                                       
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0                                         
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0                                         
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0                                         
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0                                         
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0                                         
                                                                                                                                 
SMART Error Log Version: 1                                                                                                       
No Errors Logged                                                                                                                 
                                                                                                                                 
SMART Self-test log structure revision number 1                                                                                 
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                 
# 1  Extended offline    Completed without error      00%      5119        -                                                   
                                                                                                                                 
SMART Selective self-test log data structure revision number 1                                                                   
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                     
    1        0        0  Not_testing                                                                                             
    2        0        0  Not_testing                                                                                             
    3        0        0  Not_testing                                                                                             
    4        0        0  Not_testing                                                                                             
    5        0        0  Not_testing                                                                                             
Selective self-test flags (0x0):                                                                                                 
  After scanning selected spans, do NOT read-scan remainder of disk.                                                             
If Selective self-test is pending on power-up, resume after 0 minute delay.

zpool list

Code:

[root@DMGNAS ~]# zpool list                                                                                                     
NAME  SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT                                                                         
DMG  1.81T  1.24T  589G    68%  1.00x  ONLINE  /mnt

 

[BrianDMG]

No one has any thoughts on this? If I'm reading these correctly, they're pretty flawless - is that correct?

 

[cyberjock]

They do seem pretty flawless. I'm sending you a PM...

 

[joeschmuck]

I must have started a reply but didn't finish it. Multitasking you know doesn't always work well for me. The drives do seam fine. Ada0 has some read/seek errors but some drives do report high numbers like that yet they are still fine. I still think your raid configuration is slowing you down but what bugs me is this was working fine when you were running 8.3.x.

 

[gpsguy]

What's different between the old and new hardware?

I was getting 60MB/s transfer speeds on exactly the same infrastructure, with nearly identical hardware.

 

[joeschmuck]

But I think based on your slow dd performance, the problem is still the RAIDZ configuration and mix of hard drives. I doubt the computer itself is the culprit (just a guess on my part).