Internet speeds

[Bhoot]

Hi. I have been using FreeNAS now for over a year and I am extremely happy with the performance and reliability. Everytime the scrub repairs some kilobits of data I breathe in a sigh of relief. I would like to thank all the people in the forums who have helped me make a regular idea of a NAS into a supercomputer of my own.

Coming to the problem at hand. Well it's not really a problem but I want to do something about it.
Background info. I have a 4Mbps connection which should give me a download speed of 512KBps. I am running multiple plugins including Transmission. About a month back I had transmission giving me upto 12MBps (yes a capital M not a small m) and my life couldn't be better. All things I put for downloads finished in minuted if not seconds. Now Transmission is kinda hanging around 450KBps which is quite normal for the subscription but other devices such as my laptop and PS4 are still getting super awesome speed. I downloaded a patch on PS4 of about 4GB in 15 minutes and nvidia driver of 350MB in under a minute.

Any way I can get the Transmission return to it's previous glory? I know this thread is a bit off topic but I couldn't find much help on google. Or maybe I was using the wrong keywords. Thanks a lot in advance.

quick edit:
FreeNAS bound to 192.168.1.99
Transmission bound to 192.168.1.4
Laptop and PS4 probably get DHCP IP.

 

[Ericloewe]

Everytime the scrub repairs some kilobits of data I breathe in a sigh of relief.

That should not happen often at all. It's a bit worrying that it's frequent enough for you to notice.

 

[Bhoot]

This is the current scrub.

Code:

# zpool status
  pool: bhoot
state: ONLINE
  scan: scrub in progress since Wed Jun  1 04:00:04 2016
        4.90T scanned out of 20.7T at 111M/s, 41h19m to go
        108K repaired, 23.66% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        bhoot                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/5663b940-bdde-11e5-9e00-f07959376c84  ONLINE       0     0     0
            gptid/cd427285-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0
            gptid/cda4f9a1-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0  (repairing)
            gptid/ce06b19f-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0
            gptid/ce69a75d-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0
            gptid/4cfc6072-b5e0-11e5-8742-f07959376c84  ONLINE       0     0     0
            gptid/cf2dd08e-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0
            gptid/cf91d6e8-e4d8-11e4-b39d-f07959376c84  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Tue May 24 03:46:18 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/40460acb-cf27-11e5-b12b-f07959376c84  ONLINE       0     0     0
            da1p2                                       ONLINE       0     0     0

errors: No known data errors

I have 2 disks currently with 1 and 6 unreadable/pending sectors. The pool is about 71% full. I just got a disk after RMA. Planning to resilver (most probably) when the scrub is over.

 

[Robert Trevellyan]

Your poor torrent performance might be a symptom of excessive fragmentation in the pool (check zpool list). You could test this by adding a single-disk scratch pool and telling Transmission to use that.

 

[Bhoot]

Your poor torrent performance might be a symptom of excessive fragmentation in the pool (check zpool list). You could test this by adding a single-disk scratch pool and telling Transmission to use that.

Code:

[root@freenas] ~/scripts# zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bhoot           29T       20.7T      8.25T         -                 33%        71%  1.00x     ONLINE    /mnt
freenas-boot  14.2G  1.04G    13.2G         -                   -              7%  1.00x      ONLINE     -

Frag does show 33%. I do believe there is no way to defrag the zpool except for transferring data to a new location and rewriting it to the zpool again? Any way I can ensure the fragmentation doesn't happen?? I did read somewhere (most probably this forum) that fragmentation is kinda something that's inherent in freenas and not worrysome. I mean TBH I'm getting the speed i should be getting from the ISP. I'm just being greedy and looking for more :p

Edit: Could it also be some kind of blocking done by the ISP seeing about 6TB traffic about 30 days on a 4Mbps connection? Like maybe restricting the speed to a particular mac address or a port?

 

[Robert Trevellyan]

Any way I can ensure the fragmentation doesn't happen?

There are pre-allocation strategies for torrents but I don't know how they interact with ZFS.

I'm getting the speed i should be getting from the ISP.

Yup, so the mystery is really centered on how you seem to get more than your promised speed in some situations.

Could it also be some kind of blocking done by the ISP

Could be, either due to volume of data or just traffic shaping. I assume you're using a random port and encryption for your torrents.

 

[Bhoot]

Port Randomised at launch. Port showing open. I was using preferred encryption. After your post I changed it to require encryption and changed the port to another random number. No noticeable changes.

 

[Robert Trevellyan]

Unfortunately I'm out of ideas. I'm certainly no expert as far as torrents go.

 

[Bhoot]

Anyone else??
This is the result of my latest speedtest.

Transmission D ~450KBps U ~450KBps

 

[joeschmuck]

Everytime the scrub repairs some kilobits of data I breathe in a sigh of relief.

I agree with @Ericloewe here, when reading your first post my eye was drawn to this more than the internet speed concern you have. The only time I have data being repaired is after I screwed around with my pool and that was only when I was originally building and testing it, and I put FreeNAS through hell trying to break it.

So, look at your SMART data and see if you can find something indicating you are having problems with one of your drives. And hopefully you are running frequent SMART long tests (I run mine once a week) to ensure you have no media (meaning the hard drive surface) issues. If you notice that the same drive is always being repaired, I'd check the cabling and if that doesn't fix it, replace the drive. But honestly, until looking at the SMART data, it just a guess right now.

 

[Bhoot]

I agree with @Ericloewe here, when reading your first post my eye was drawn to this more than the internet speed concern you have. The only time I have data being repaired is after I screwed around with my pool and that was only when I was originally building and testing it, and I put FreeNAS through hell trying to break it.

So, look at your SMART data and see if you can find something indicating you are having problems with one of your drives. And hopefully you are running frequent SMART long tests (I run mine once a week) to ensure you have no media (meaning the hard drive surface) issues. If you notice that the same drive is always being repaired, I'd check the cabling and if that doesn't fix it, replace the drive. But honestly, until looking at the SMART data, it just a guess right now.

Like I said there are 2 drives with unreadable pending sectors.
Here is a SMART report for them

Code:

########## SMART status report for ada6 drive (Western Digital Red: WD-WCC4E4KC79HK) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       29
  3 Spin_Up_Time            0x0027   178   178   021    Pre-fail  Always       -       8075
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       162
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8609
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       158
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       90
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       684
194 Temperature_Celsius     0x0022   117   099   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline       Completed without error       00%      8607         -


########## SMART status report for ada4 drive (Western Digital Red: WD-WCC4E5PL8DNS) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1848
  3 Spin_Up_Time            0x0027   181   181   021    Pre-fail  Always       -       7933
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       158
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8608
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       158
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       86
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       704
194 Temperature_Celsius     0x0022   116   098   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       5
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       280

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline       Completed without error       00%      8596         -

 

[Ericloewe]

Ouch, that second one is definitely a goner. Doesn't explain atrocious performance, though, since it supports TLER.

 

[Robert Trevellyan]

atrocious performance

 

[Bhoot]

I agree with @Ericloewe here, when reading your first post my eye was drawn to this more than the internet speed concern you have. The only time I have data being repaired is after I screwed around with my pool and that was only when I was originally building and testing it, and I put FreeNAS through hell trying to break it.

So, look at your SMART data and see if you can find something indicating you are having problems with one of your drives. And hopefully you are running frequent SMART long tests (I run mine once a week) to ensure you have no media (meaning the hard drive surface) issues. If you notice that the same drive is always being repaired, I'd check the cabling and if that doesn't fix it, replace the drive. But honestly, until looking at the SMART data, it just a guess right now.

Ouch, that second one is definitely a goner. Doesn't explain atrocious performance, though, since it supports TLER.

Like I always said, It's a pleasure being a part of this forums. Today when I woke up I was greeted with a lot of mails from my server. Most important being

Code:

Device: /dev/ada6, 1 Currently unreadable (pending) sectors
Device: /dev/ada4, 5 Currently unreadable (pending) sectors
Device: /dev/ada4, unable to open device
The volume bhoot (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

Luckily I had a cold spare. Through the GUI I couldn't "offline" the disk. It showed "UNAVAIL". Walked to the server, opened the baby up, replaced the disk physically, and started the resilver process.

Now it's resilvering

Code:

  pool: bhoot
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jun  8 09:26:19 2016
  494G scanned out of 20.4T at 124M/s, 46h48m to go
  60.3G resilvered, 2.37% done
config:

  NAME  STATE  READ WRITE CKSUM
  bhoot  ONLINE  0  0  0
  raidz2-0  ONLINE  0  0  0
  gptid/5663b940-bdde-11e5-9e00-f07959376c84  ONLINE  0  0  0
  gptid/cd427285-e4d8-11e4-b39d-f07959376c84  ONLINE  0  0  0
  gptid/ec0f7827-2d2c-11e6-b1de-f07959376c84  ONLINE  0  0  0  (resilvering)
  gptid/ce06b19f-e4d8-11e4-b39d-f07959376c84  ONLINE  0  0  0
  gptid/ce69a75d-e4d8-11e4-b39d-f07959376c84  ONLINE  0  0  0
  gptid/4cfc6072-b5e0-11e5-8742-f07959376c84  ONLINE  0  0  0
  gptid/cf2dd08e-e4d8-11e4-b39d-f07959376c84  ONLINE  0  0  0
  gptid/cf91d6e8-e4d8-11e4-b39d-f07959376c84  ONLINE  0  0  0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Tue May 24 03:46:18 2016
config:

  NAME  STATE  READ WRITE CKSUM
  freenas-boot  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  gptid/40460acb-cf27-11e5-b12b-f07959376c84  ONLINE  0  0  0
  da1p2  ONLINE  0  0  0

errors: No known data errors

@Ericloewe was it the RAW_READ_ERROR_RATE which is a sign of a failing disk? I just wanted to know what SMART parameter you used to determine the disk was dying?

Another question

Code:

  mirror-0  ONLINE  0  0  0
  gptid/40460acb-cf27-11e5-b12b-f07959376c84  ONLINE  0  0  0
  da1p2  ONLINE  0  0  0

Why does one of the boots show up as gptid... and the other as da1p2?? Is this normal? When I was using a single boot atmosphere it used to show up as da1p1

 

[joeschmuck]

@Ericloewe was it the RAW_READ_ERROR_RATE which is a sign of a failing disk? I just wanted to know what SMART parameter you used to determine the disk was dying?

No, that value is not reliable. Have you looked in the Troubleshooting Guides section of these forums, there I have a guide to diagnose hard drive failures and all your answers are there. But in short your specific failing indicators were the ID 197 Current pending sector count was above zero.

Why does one of the boots show up as gptid... and the other as da1p2?? Is this normal? When I was using a single boot atmosphere it used to show up as da1p1

It happens and it's normal, nothing to worry about. Typically happens when you replace a drive.

 

[Robert Trevellyan]

Why does one of the boots show up as gptid... and the other as da1p2?? Is this normal?

It's a good question. Mine shows up that way after a clean install to a matching pair of USB sticks, so I guess it's 'normal'.

 

[Bhoot]

No, that value is not reliable. Have you looked in the Troubleshooting Guides section of these forums, there I have a guide to diagnose hard drive failures and all your answers are there. But in short your specific failing indicators were the ID 197 Current pending sector count was above zero.

Sorry for the delay. Yes I have gone through that section and I just revised it again (I do tend to forget a few things). The unreadable (pending) sectors are always bad. But I have had a hard disk going upto 500+ unreadable (pending) sectors which was eventually replaced when I could find a suitable replacement. The disk did not drop out of the pool on its own. Why then a disk with just 5 "fall out" of the pool overnight? The value of sectors have been constant for a while.

This is the same disk from a May 19th report.

Code:

########## SMART status report for ada4 drive (Western Digital Red: WD-WCC4E5PL8DNS) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1518
  3 Spin_Up_Time            0x0027   190   182   021    Pre-fail  Always       -       7491
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       152
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8256
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       152
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       81
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       654
194 Temperature_Celsius     0x0022   108   098   000    Old_age   Always       -       44
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       6
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       280

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline       Completed without error       00%      8247         -

ID 197 here is 6 whereas later it was reduced to 5

The other hard disk has 1 bad sector as well.[/QUOTE]

 

[joeschmuck]

You have one good thing going for you right now, ID 5 is still zero. I suspect that ID 197 lowered because one of the sectors which had an issue worked. This can happen but I don't see it often. Another issue is ID 200, but to be honest, it isn't always a good indicator of a hardware failure.

Regardless the drive is failing and you have almost 2 years left on the warranty so I'd see if you can do an Advanced RMA (I think that is what they call it) where you give WD a credit card and they will ship you a replacement drive and then you ship back the failed one in the same box the new drive came in. It's pretty easy. This way you can leave your FreeNAS running until the new drive comes in.

BUT, lets say you want to keep the two drives in question because ID 5 is still a zero value, then I'd personally run the SMART Long Test twice a day on those two drives until either ID 197 drops to zero or for 2 weeks and all of the extended tests pass without issue. Unfortunately the SMART Long test is read-only so it's a limited test. You could pull the worst drive out of the system and perform a destructive test which would give you a better indicator of how good or bad the drive is, and then if you do that, I'd run the test at least a few times. These are just options you could do if you desire to.

Why then a disk with just 5 "fall out" of the pool overnight?

I don't know and could speculate but that speculation wouldn't fix the problem.

If you haven't done this you could try it but I don't think it will cause the problem to change... Move the SATA data cables around (or replace them) and see if the problem moves for the pool dropping offline, or maybe even stops dropping offline. Odds are that the problem will remain with the failing hard drive but if you are still debating on if you should RMA the drive...

Good luck and I await your next report.

 

[Ericloewe]

Here is what a Scrub does: A scrub reads all the data from the drive and writes it to another location in the storage pool.

Scrubs only read, they don't write anything (besides repair operations). That would need bock pointer rewrite, otherwise snapshots would end up not working across scrubs, in that they would end up containing every block on the pool, due to them having been rewritten by the scrub.

SMART long tests do test the disk surface, though.

 

[joeschmuck]

Scrubs only read, they don't write anything (besides repair operations). That would need bock pointer rewrite, otherwise snapshots would end up not working across scrubs, in that they would end up containing every block on the pool, due to them having been rewritten by the scrub.

SMART long tests do test the disk surface, though.

Don't ask me why I quoted something on the internet without verifying it. I went back to Solaris docs and yup, still read-only.

The SMART Long test is only a read operation, but it reads the entire surface.