Bad SMB READ performance on FreeNAS 9.1

[blackbinary]

Hi Guys,

I hope you can Help me. I need some help with my new NAS. I have very poor Performance with CIFS on FreeNAS 9.1 mostly if i try to read Data from a CIFS Share.

Code:

Running a 4000MB file write on drive H: 5 times...
Iteration 1:    73.34 MB/sec
Iteration 2:    71.39 MB/sec
Iteration 3:    74.51 MB/sec
Iteration 4:    73.59 MB/sec
Iteration 5:    74.40 MB/sec
------------------------------
Average (W):    73.45 MB/sec
------------------------------
Running a 4000MB file read on drive H: 5 times...
Iteration 1:    24.13 MB/sec
Iteration 2:    23.39 MB/sec
Iteration 3:    22.50 MB/sec
Iteration 4:    23.30 MB/sec
Iteration 5:    23.34 MB/sec
------------------------------
Average (R):    23.33 MB/sec
------------------------------

-> Windows or Apple as SMB Client makes no differences

This is iostat while i am Use NFS to read / write from NAS

Code:

[root@Lucifron] ~# zpool iostat zStorage 3 10
              capacity    operations    bandwidth
pool        alloc  free  read  write  read  write
----------  -----  -----  -----  -----  -----  -----
zStorage    9.43T  5.07T      2    10  43.2K  679K
zStorage    9.43T  5.07T    601    82  73.3M  572K
zStorage    9.43T  5.07T    601      0  73.5M      0
zStorage    9.43T  5.07T    687    82  85.2M  572K
zStorage    9.43T  5.07T    601    64  74.3M  303K
zStorage    9.43T  5.07T    681      0  83.9M      0
^C
[root@Lucifron] ~# zpool iostat zStorage 3 10
              capacity    operations    bandwidth
pool        alloc  free  read  write  read  write
----------  -----  -----  -----  -----  -----  -----
zStorage    9.43T  5.07T      3    10  214K  681K
zStorage    9.43T  5.07T      0    780      0  79.1M
zStorage    9.43T  5.07T      0    480      0  47.7M
zStorage    9.43T  5.07T      0    788      0  80.6M
zStorage    9.43T  5.07T      0    908      0  93.4M
zStorage    9.43T  5.07T      0    414      0  40.3M
zStorage    9.43T  5.07T      0    948      0  94.7M

My NAS is a INTEL Server Board DBS1200KPR based System with a i3-3220 3,33Ghz CPU and 16 GB DDr3 Ram.

The pool looks like this :

Code:

[root@Lucifron] ~# zpool list
NAME      SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT
zStorage  14.5T  9.43T  5.07T    65%  1.05x  ONLINE  /mnt
[root@Lucifron] ~# zpool status
  pool: zStorage
state: ONLINE
  scan: scrub repaired 0 in 31h13m with 0 errors on Mon Dec 30 07:13:46 2013
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        zStorage                                        ONLINE      0    0    0
          raidz2-0                                      ONLINE      0    0    0
            gptid/e9e81eaa-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/4c73f710-5b63-11e3-9070-54bef7383bcd  ONLINE      0    0    0
            gptid/ec471dc5-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/ed86b365-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/eec3ca65-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/eff04dda-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/f136afd4-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
            gptid/f28c5c62-4a27-11e3-beef-54bef7383bcd  ONLINE      0    0    0
        logs
          gptid/f320ff10-4a27-11e3-beef-54bef7383bcd    ONLINE      0    0    0
        cache
          gptid/f2f125bb-61c0-11e3-a304-54bef7383bcd    ONLINE      0    0    0
 
errors: No known data errors

(8x2TB WD2002FYPS, 60 GB Log SSD, 120 GB Cache SSD)

Can someone help me in this Case? Is there a Way to Tune SMB on FreeNAS?

Thanks
Black

 

[cyberjock]

First, get rid of the slog. Second, get rid of the cache. L2ARCs only work if you have lots of RAM. Otherwise you kill pool performance.

Then read this to understand what you are doing wrong: http://forums.freenas.org/threads/ssds-arc-and-zil.17191/#post-90640

slogs only matter for sync writes. l2arcs only work if you have lots of RAM to start. If you don't have lots of RAM(and we're talking a ratio of ARC to L2ARC not more than 1:5 which means you'd be crazy to have anything bigger than about 30GB for an L2ARC) then your L2ARC will actually hurt pool performance.

 

[cyberjock]

Oh, let me add a few things now that I've gone back and looked closely.

You've enabled dedup somewhere. You don't end up with a ratio of 1.05x for dedup with it disabled. Not sure how much data you have, but you are supposed to have 5GB of RAM per TB of disk storage(which you absolutely don't have) and you could need more as there is no upper limit for how much RAM you'll need. Dedup steals from the ARC(which means your L2ARC should be even smaller than the 30GB or so I quoted above). When dedup fails you, there's a good chance it'll take your pool with it. So I hope you have backups! You can't just turn off dedup either. You have to turn it off and remove all deduped blocks from the server to really get rid of dedup. Using dedup is like playing with fire. And with just 16GB of RAM, it's utterly crazy.

So while you complain about poor performance I think your bigger problem is you don't have a clue what you are doing and are throwing more and more hardware at the problem hoping it'll fix it. Reality check: it won't. And you've put your data at risk by taking this route.

FreeNAS should perform just fine with that hardware. If it's not, either your administration of the server is horrible, or you have some kind of hardware failure such as a failed disk.

No offense to you, this is just how I see it.

 

[jkh]

I hope you can Help me. I need some help with my new NAS. I have very poor Performance with CIFS on FreeNAS 9.1 mostly if i try to read Data from a CIFS Share.

With a 1GbE link to the machine, 95MB/sec should be more typical. The fact that you're seeing so much less strongly suggests that you have misconfigured this pool. As cyberjock asks, are you using dedup? If so, don't! You don't have anything close to the required hardware configuration for that.

 

[eraser]

What version of Windows are you running? Can you attach the output of running "arc_summary.py" ?

 

[blackbinary]

First of all, thank you very much for your answer. First of all, thank you very much for your answer.

You've enabled dedup somewhere.

you are
Yes, You are right. I Used dedup for 4 of my datasets. More than 3 TB of Data was writen with dedup enabled. I read about this 5GB per TB rule and tryed to Roll back what i'm done. I created eatch dataset new and made a Copy of the Old data with rsync. After that i deleted the dataset that had dedub enabled.
Question : is that the right way to get rid of the entire de-dedub data and dedub Table?? or must start over and use a Backup ?


About L2ARC:

You are better off doing a small partition so the SSD will use the unpartitioned space for allocations, thereby increasing performance AND longevity.

Thats a good point!, i know that mutch about SSD.
Question: is it a good Idea to use a 25GB partition for L2ARC ? Would that really bring a positive effect?

About ZIL
I realy dont know what to do with the ZIL SSD. leave unused? or mirror two Partition over both SSD or Stripe the two SSD for a small l2ARC partition ? or both ?

If using small partitions is worth a try, how can i create and add a partition to my Raid ?

With a 1GbE link to the machine, 95MB/sec should be more typical.

maybe but netio tells me that i only get 70-80MB/s i dont know why, i have new Switches here. (i tested from a Jail to Windows and Mac ) but this is a other case.

THANKS for your Help!

black

 

[blackbinary]

What version of Windows are you running? Can you attach the output of running "arc_summary.py" ?

a Updated W7 64bit

http://pastebin.com/RV3uSeHv

 

[cyberjock]

The best way to get rid of dedup is to move all of the data out of the dedup dataset and destroy the dataset.

Or even create a new dataset and do a local mv command from one dataset to the other.

About L2ARC:

Thats a good point!, i know that mutch about SSD.
Question: is it a good Idea to use a 25GB partition for L2ARC ? Would that really bring a positive effect?

About ZIL
I realy dont know what to do with the ZIL SSD. leave unused? or mirror two Partition over both SSD or Stripe the two SSD for a small l2ARC partition ? or both ?

If using small partitions is worth a try, how can i create and add a partition to my Raid ?


maybe but netio tells me that i only get 70-80MB/s i dont know why, i have new Switches here. (i tested from a Jail to Windows and Mac ) but this is a other case.

THANKS for your Help!

black

First, get rid of the l2arc and zil completely like I said in my first post. You don't have enough RAM or use your system in a way that either of those will benefit you. You should not be using EITHER one.

Second, as a general rule I consider most people to be at peak speeds after 70MB/sec. The reason being that there are many ways to eek out that extra speed if you really want it. In particular, more RAM and a faster CPU. You shot yourself in the foot in the RAM department choosing to go with a 2-RAM-slot motherboard. Now you know why I continue to tell people not to use them and they do it anyway.The manual makes it clear that the single fastest way to improve performance is more RAM, and there is no substitute. Your CPU is probably okay assuming you aren't running lots of jails that consume CPU power. Pool performance can also be affected by using some addons like transmission that does small random writes.

There are so many things that can affect your pool's performance including but not limited to hardware choices, tunables, etc. And everyone blindly(and stupidly) goes and starts buying unnecessary hardware and setting tunables/sysctls that shouldn't be set. Your FreeNAS box should work just fine with zero tunables, zero sysctls, and no customization for the protocols. I have ZERO customizing and I hit 110MB/sec almost every time.

 

[eraser]

http://pastebin.com/RV3uSeHv

I'm no expert on L2 ARC, but your arc_summary output indicates (DEGRADED)...

Code:

L2 ARC Summary: (DEGRADED)
    Passed Headroom:            5.21m
    Tried Lock Failures:            2.86m
    IO In Progress:                86
    Low Memory Aborts:            14
    Free on Write:                2.95k
    Writes While Full:            23.96k
    R/W Clashes:                31
    Bad Checksums:                3.16k
    IO Errors:                0
    SPA Mismatch:                0

A google search turned up a forum post that includes a patch. http://freebsd.1045724.n5.nabble.co...system-load-on-r255173-tp5849843p5854015.html

While you wait for the patch to eventually make it into a future release of FreeNAS I would try removing your L2 ARC and re-test things.

 

[blackbinary]

okay, i removed the l2ARC and tryed the same with the Log. But the NAS get stuck at this action andy finally was unresponsive. now, if i just reboot, the FreeNAS hang on "Mounting local filesystems". The Disks work very HArd to the begin. after 1-2 Houres i hear no Disks working. the nas just hangs.

I think that was not a good Idea to remove Cache and Log. I try to mount the Pool with openindiana.

 

[blackbinary]

okay, atm the NAS has overcame the Hang and is running again. I think this had to do with dedup and l2ARC removal. IMO i have made a Copy of all Datasets that had dedup enabled and removed the old Dataset. BUT i had Still a dedup of 1.05x Yesterday i saw 1.04x after some reboots and after some more reboots the NAS is back Online and i see a dedup of 1.00x.

The NAS is alive and work very hard like a running scrub. But Scrub is not running.
http://pastebin.com/0WZhWbd5

 

[cyberjock]

Not sure what its doing, but allocated and free space are changing. It could be your dataset delete is still in progress? I know some tasks are supposed to work in the background on v5000. I'm just not sure what those tasks are or if they've even been implemented yet.

In any case, I'd just keep an eye on it.

 

[Dusan]

I know some tasks are supposed to work in the background on v5000. I'm just not sure what those tasks are or if they've even been implemented yet.

http://open-zfs.org/wiki/Features#Asynchronous_Filesystem_and_Volume_Destruction
[PANEL]With asynchronous destruction the filesystem's data is immediately moved to a "to be freed" list, allowing the destroy operation to complete without traversing any of the filesystem's data. A background process reclaims blocks from this "to be freed" list and is capable of resuming this process after reboots without slowing the pool import process.[/PANEL]

 

[cyberjock]

http://open-zfs.org/wiki/Features#Asynchronous_Filesystem_and_Volume_Destruction
[PANEL]With asynchronous destruction the filesystem's data is immediately moved to a "to be freed" list, allowing the destroy operation to complete without traversing any of the filesystem's data. A background process reclaims blocks from this "to be freed" list and is capable of resuming this process after reboots without slowing the pool import process.[/PANEL]

Is there a way to prove that those features are in FreeNAS 9.2.0? From what I've heard in the admin section of the forums the ZFS code FreeNAS used is a mish-mash of available code from many sources put together to provide the best of all available worlds. There's no other OS anywhere that has similar code to what FreeNAS offers.

 

[blackbinary]

Does it make sense to Update?

Code:

[root@Lucifron] ~# uname -a
FreeBSD Lucifron.black.intra 9.1-STABLE FreeBSD 9.1-STABLE #0 r+16f6355: Tue Aug 27 00:38:40 PDT 2013     root@build.ixsystems.com:/tank/home/jkh/src/freenas/os-base/amd64/tank/home/jkh/src/freenas/FreeBSD/src/sys/FREENAS.amd64  amd64
[root@Lucifron] ~# zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zStorage  14.5T  5.58T  8.92T    38%  1.00x  ONLINE  /mnt
[root@Lucifron] ~# zpool status
  pool: zStorage
 state: ONLINE
  scan: scrub repaired 0 in 31h13m with 0 errors on Mon Dec 30 07:13:46 2013
config:
 
        NAME                                            STATE     READ WRITE CKSUM
        zStorage                                        ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/e9e81eaa-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/4c73f710-5b63-11e3-9070-54bef7383bcd  ONLINE       0     0     0
            gptid/ec471dc5-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/ed86b365-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/eec3ca65-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/eff04dda-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/f136afd4-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
            gptid/f28c5c62-4a27-11e3-beef-54bef7383bcd  ONLINE       0     0     0
 
errors: No known data errors

At this Point i can't enter the WebGUI. After Login the Webgui seems to hang.
SMB Test:

Code:

Running a 400MB file write on drive H: 5 times...
Iteration 1:     14.00 MB/sec
Iteration 2:     15.73 MB/sec
Iteration 3:     15.83 MB/sec
Iteration 4:     15.86 MB/sec
Iteration 5:     7.43 MB/sec
------------------------------
Average (W):     13.77 MB/sec
------------------------------
Running a 400MB file read on drive H: 5 times...
An error occured: The specified network name is no longer available.

that is realy the case, i cant write Data with SMB and can see it but if i Try to read windows Hangs

NFS test:

Code:

 [root@Lucifron] ~# zpool iostat zStorage 5 2000
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zStorage    5.58T  8.92T     40     13  3.82M  79.2K
zStorage    5.58T  8.92T    556     75  67.9M   521K
zStorage    5.58T  8.92T    599     74  73.7M   511K
zStorage    5.58T  8.92T    489     52  60.0M   226K
zStorage    5.58T  8.92T    501     70  61.8M   499K
zStorage    5.58T  8.92T    599     75  73.6M   527K
zStorage    5.58T  8.92T    548     46  67.4M   202K
zStorage    5.58T  8.92T    355     51  43.3M   221K
zStorage    5.58T  8.92T      0      0      0      0
zStorage    5.58T  8.92T      0      0      0      0
zStorage    5.58T  8.92T      0      0      0      0
zStorage    5.58T  8.92T      0      0      0      0
zStorage    5.58T  8.92T      0    353    819  38.6M
zStorage    5.58T  8.92T      0    784      0  90.1M
zStorage    5.58T  8.92T      0    860      0   102M
zStorage    5.58T  8.92T      0    937      0   111M
zStorage    5.58T  8.92T      0     1K  3.20K   122M
zStorage    5.58T  8.92T      0  1.06K      0   126M
zStorage    5.58T  8.92T      0  1.05K      0   125M
zStorage    5.58T  8.92T      0   1021      0   119M
zStorage    5.58T  8.92T      0    727      0  82.8M

copy some files to my Mac mini and some other files to the NAS while this was running.

black

 

[blackbinary]

that is realy the case, i cant write Data with SMB

My bad, i ment READ, can't Read.

 

[Dusan]

Is there a way to prove that those features are in FreeNAS 9.2.0? From what I've heard in the admin section of the forums the ZFS code FreeNAS used is a mish-mash of available code from many sources put together to provide the best of all available worlds. There's no other OS anywhere that has similar code to what FreeNAS offers.

Of course. Use the source, Luke! :) The code is present: https://github.com/trueos/trueos/bl...nsolaris/uts/common/fs/zfs/dsl_destroy.c#L746
You can check if the feature is enabled in your pool by running "zpool get feature@async_destroy <pool_name>". Output for one of my pools:

[PRE]# zpool get feature@async_destroy backup
NAME PROPERTY VALUE SOURCE
backup feature@async_destroy enabled local[/PRE]

 

[blackbinary]

Code:

[root@Lucifron] ~# zpool get feature@async_destroy
NAME      PROPERTY              VALUE                  SOURCE
zStorage  feature@async_destroy  enabled                local

it is

 

[blackbinary]

Okay, now my NAS is running again. With the latest stable 9.2 amd64. There is No Dedup, No Log, No cache. and the problem remains the same -.-... NFS is Fast and SMB has realy Read problems.. What can i try ?

Code:

Running warmup...
Running a 400MB file write on drive H: 5 times...
Iteration 1:    73.10 MB/sec
Iteration 2:    76.98 MB/sec
Iteration 3:    74.79 MB/sec
Iteration 4:    74.61 MB/sec
Iteration 5:    76.45 MB/sec
------------------------------
Average (W):    75.18 MB/sec
------------------------------
Running a 400MB file read on drive H: 5 times...
Iteration 1:    23.31 MB/sec
Iteration 2:    22.20 MB/sec
Iteration 3:    23.69 MB/sec
Iteration 4:    21.86 MB/sec
Iteration 5:    21.36 MB/sec
------------------------------
Average (R):    22.48 MB/sec
------------------------------

 

[eraser]

What tool are you using to perform your SMB read tests (so that I can run it on my system). Is it a Windows utility?