FreeNAS 9.1 and CIFS performance

[N00b]

I am currently running:
FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty) on
Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz on a
Intel INTEL DH55TC mobo with
12GB of Memory and
3x3TB Seagate ST3000DM001s RAIDZ1.
The mobo has Intel(R) PRO/1000 Network card.

These were LAN SpeedTest results on my LAN before the upgrade to 9.1.1.

Code:

Version 1.3.1
OS Version: Windows 7
Processor: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Date: 10/19/2013
Time: 18:18:40
Program Parameters: 0
High Performance Timer: 0.000000311
 
Test File: Z:\homes\freenas\tmp-lantest\NW_SpeedTest.dat
Write Time = 14.2610033 Seconds
Write Speed = 574.4336400 Mbps
Read Time = 21.6838406 Seconds
Read Speed = 377.7928560 Mbps

And these post upgrade to FreeNAS-9.1.1-RELEASE-x64 (a752d35)

Code:

Test File: Z:\homes\freenas\tmp-lantest\NW_SpeedTest.dat
Write Time = 347.7767947 Seconds
Write Speed = 23.5553360 Mbps
Read Time = 112.4263464 Seconds
Read Speed = 72.8654800 Mbps

And then back to 8.3.1-p2

Code:

Test File: Z:\homes\freenas\tmp-lantest\NW_SpeedTest.dat
Write Time = 15.7235326 Seconds
Write Speed = 521.0025120 Mbps
Read Time = 17.4478698 Seconds
Read Speed = 469.5129040 Mbps

all with 1024 MB of data. ( I did try with other sizes as well; all giving similar results so omitting)

It does seem that ZFS under 9.1.1 is less forgiving than under 8.3.1 . Since a have observed quite a few others who are facing similar issues , would appreciate if someone could advise a way to optimize ZFS for current hardware. I can at best add another 4GB to the current config but I suspect that it won't make too much difference.

 

[cyberjock]

And what happens if you do dd test on your pool locally? Your hardware should be able to smoke FreeNAS. But since you are doing network shares the limitation could be on the desktop too.

What you need to do is systematically test until you figure out where things are going wrong. My guess is its not the server. You've done everything right hardware-wise.

 

[N00b]

Cyberjock,
Here are the results I performed yesterday using. Sorry I haven't used dd before but was aware that of its reputation as data-destroyer :) so had to google a bit . Hopefully these two test should be sufficient - first without sync then with the sync conversion option.

Booting from my 8.3.1-p2 usb stick.

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 7.717004 secs (139139725 bytes/sec)   
 
Or Say 132.6939821    MB/s
           
[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test conv=sync           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 0.434258 secs (2472589712 bytes/sec)   
 
Or Say    2358.045303    MB/s

Then booting from the 9.1.1 stick. I have not upgraded pool.

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 15.926320 secs (67419330 bytes/sec)   
Or Say    64.29608345    MB/s
 
[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test conv=sync           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 70.332619 secs (15266626 bytes/sec)   
Or Say    14.55938911    MB/s

This really had me worried. I checked and found that a rsync job was also in progress at that time. I shut that down and tried again. Here are results from Run2.

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 11.515029 secs (93246993 bytes/sec)   
Or Say    88.92726231    MB/s
[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test conv=sync           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 41.943932 secs (25599456 bytes/sec)   
Or Say    24.4135437    MB/s

And another try with 'sync' as I couldn't believe the earlier results.

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero of=test conv=sync           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 30.535898 secs (35163263 bytes/sec)   
Or Say    33.5343008    MB/s

It seems that poor performance is not limited to data transfer. See the following test (from what I read it is supposed to be a pseudo test for cpu performance)
Under 8.3.1-p2

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero |md5           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 16.617271 secs (64616015 bytes/sec)   
cd573cfaace07e7949bc0c46028904ff           
Or Say    61.62263393    MB/s

Under 9.1.1 ( with rsync job running)

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero |md5           
1024+0 records in           
1024+0 records out           
107374182 bytes transferred in 139.584431 secs (7692418 bytes/sec)   
cd573cfaace07e7949bc0c46028904ff           
Or Say    7.336061478    MB/s

Without rsync (or anyother job running)

Code:

[pks@nbx ~/tmp-lantest]$ dd bs=1M count=1024 if=/dev/zero |md5           
1024+0 records in           
1024+0 records out           
1073741824 bytes transferred in 57.267038 secs (18749736 bytes/sec)   
cd573cfaace07e7949bc0c46028904ff           
Or Say    17.88114166    MB/s

This is the result of an rsync job itself.
First under 8.3.1 from a pcbsd box. Both are on 1G wired LAN.

Code:

[pks@pcbsd /usr/jails]$ sudo rsync -WhavP pcbsd3.wdn <nasbox ip>::bytepond
sending incremental file list
pcbsd3.wdn
    504.86M  5%  10.72MB/s    0:12:01

Frankly I am surprised and disappointed with this. I normally get ~ 15-20MB/s from this box when syncing movie rips.
Here with the nas box running 9.1.1

Code:

[pks@pcbsd /usr/jails]$ sudo rsync -WhavP pcbsd3.wdn <nasbox ip>::bytepond
sending incremental file list
pcbsd3.wdn
  296.35M  3%    4.53MB/s    0:29:12

It appears that either 9.1 is less forgiving than 8.3 or requires a different set of tuneables (I'm just SWAGing it here :) ). There seem to be a lot of similar reports of lesser performance on the forum.

 

[cyberjock]

Honestly, 9.1 caused a performance boost for many that use well supported hardware. I saw about 100MB/sec increase on my pool.

What hardware are you using? What SATA/SAS controller?

The rsync info isn't too useful as rsync is very CPU bound. I can do over 500MB/sec on my pool, but rsync doesn't even do 30MB/sec for me.

This does prove that the issue is on the server itself and not a networking issue. Your hardware config probably includes a SATA/SAS controller that isn't well supported in FreeBSD. At least, that's my first guess.

If you look at top, can you provide the line that says CPU: X% user, X% nice, X% system, X% interrupt, X% idle when the system is both idle and when you are doing a dd test? You may have hardware that is causing excessive interrupts in 9.x causing your performance to plummet.

 

[pirateghost]

wouldn't

Code:

~/tmp-lantest

be on the USB drive, or are you logged in via SSH with a user with a home mounted on a zvol/dataset?

 

[cyberjock]

wouldn't

Code:

~/tmp-lantest

be on the USB drive, or are you logged in via SSH with a user with a home mounted on a zvol/dataset?

Doesn't that depend on where his "home" directory is? I just figured that 130MB/sec from one of the test was enough to prove it wasn't the USB drive.

 

[N00b]

I am logged in via SSH with a user with a home mounted on a zfs dataset residing on a 3x3TB Seagate ST3000DM001s RAIDZ1 pool.

Code:

NAME                    USED  AVAIL  REFER  MOUNTPOINT
 
rock                    4.57T  776G  4.27T  /mnt/rock
 
[pks@nbx ~]$ pwd
 
/mnt/rock/homes/pks

The hard drives are directly on the mobo which is desktop grade ( Intel DH55TC) unfortunately. The current set up has served me well for the last 6 months (other than I've to Z1 and non-ECC components).

Here are the outputs : 9.1.1 idle

Code:

last pid:  3907;  load averages:  0.82,  0.95,  1.05                                                                                up 0+00:42:11  22:15:05
47 processes:  3 running, 44 sleeping
CPU: 58.7% user,  0.0% nice,  3.4% system,  0.5% interrupt, 37.4% idle
Mem: 63M Active, 72M Inact, 1367M Wired, 1168K Cache, 60M Buf, 10G Free
ARC: 1056M Total, 72M MFU, 967M MRU, 160K Anon, 13M Header, 4188K Other
Swap: 8192M Total, 8192M Free

9.1.1 dd running

Code:

last pid:  3935;  load averages:  1.27,  1.08,  1.09                                                                                up 0+00:43:02  22:15:56
44 processes:  1 running, 43 sleeping
CPU: 13.8% user,  0.0% nice,  2.0% system,  0.5% interrupt, 83.7% idle
Mem: 45M Active, 70M Inact, 1598M Wired, 1168K Cache, 60M Buf, 9969M Free
ARC: 1254M Total, 72M MFU, 967M MRU, 198M Anon, 13M Header, 4538K Other
Swap: 8192M Total, 8192M Free
 
  PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
3935 pks          1  52    0  9904K  2532K tx->tx  0  0:05 20.56% dd

Code:

last pid:  3969;  load averages:  1.21,  1.16,  1.13                                                                                up 0+00:44:37  22:17:31
45 processes:  2 running, 43 sleeping
CPU: 26.5% user,  0.0% nice,  0.3% system,  0.3% interrupt, 73.0% idle
Mem: 53M Active, 71M Inact, 2396M Wired, 1168K Cache, 60M Buf, 9162M Free
ARC: 2085M Total, 72M MFU, 1992M MRU, 160K Anon, 15M Header, 5880K Other
Swap: 8192M Total, 8192M Free
 
  PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
3956 root          1  87    0 83772K 15368K CPU2    2  0:18 48.88% python2.7
2434 root          7  20    0  122M 13968K uwait  1  3:47  5.91% collectd

Code:

last pid:  4096;  load averages:  1.11,  1.25,  1.17                                                                                up 0+00:49:52  22:22:46
45 processes:  2 running, 43 sleeping
CPU: 25.5% user,  0.0% nice, 26.0% system,  9.6% interrupt, 39.0% idle
Mem: 45M Active, 78M Inact, 1792M Wired, 1168K Cache, 61M Buf, 9767M Free
ARC: 1462M Total, 181M MFU, 1112M MRU, 144M Anon, 19M Header, 6244K Other
Swap: 8192M Total, 8192M Free
 
  PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
4092 pks          1  31    0  9904K  2532K CPU3    3  0:07 20.65% dd
2434 root          7  20    0  122M 13968K uwait  3  4:23 13.43% collectd
3897 pks          1  22    0 16552K  2500K CPU1    1  0:24  3.56% top

The last one show interrupt at 9.6% . I am trying to figure out a way to log the interrupt values to a text / csv file. Really appreciate your help.

 

[N00b]

I've logged the CPU process. It clearly shows three /four spikes as I ran the dd process (three/four times). :( So what do we do next?

 

[cyberjock]

And poor N00b has bad RAM.. sigh. His indication.. weird performance with 9.1. But 8.3 was fine. Go figure. Haven't gotten to the act of verifying his pool status, but we'll find out soon enough.

Figured this out in PMs.. I'll update when we have final disposition.

 

[N00b]

So poor N00b does have bad RAM. :(

Here is a screenshot of the test I ran with memtest86 overnight and there is one that I ran today with memtest86+ . From these screens is it possible to tell which RAM stick is bad? Or would I have to test each separately. A new stick that I bought worked fine yesterday but today is causing the box to reboot after a few seconds. Could there be another reason other than bad RAM? Help would be appreciated.

Memtest Results


Memtest86+ results

 

[N00b]

Update:
So this is what I am hoping happened. The RAM was probably ok but the 2 new sticks I added were probably incompatible with the 2 previous ones. Anyway I've traded in the old sticks for 4 new Transcend JM1333KLN 4x4GB sticks and run Memtest86+ again. Also ended up adding another 2GB to my BSD test box :).

Did 4+ passes without any error before I tripped the power supply. Currently running another 3 passes. So far so good. Now what next?

@cyberj : You mentioned something about some tests I need to do before I import the zpool.

 

[N00b]

Update :

1. Increased memory to 16GB and did 7 passes with memtest86+ without an error.
2. Booted from 8.3.1 stick and did a scrub on the pool. The scrub showed error in some files and snapshots.
3. Deleted all snapshots and rescrubbed the pool. No errors this time.:)

Noticed that the defaults on my mobo was loaded my disks as IDE disk and not as SATA with ACHI. Changed that to SATA and ACHI and reran tests with 9.1.1. The results are below:

The disks are set to idle so the first few writes take longer (termed "outliers"). Also reran Lantest (lite) : Here are the results:
Version 9.1.1 drives read as IDE.
Test File: Z:\homes\user\tmp-retest\NW_SpeedTest.dat Write Time = 0.2411796 Seconds Write Speed = 663.4059280 Mbps Read Time = 0.1935905 Seconds Read Speed = 826.4866960 Mbps Test File: Z:\homes\user\tmp-retest\NW_SpeedTest.dat Write Time = 12.6212540 Seconds Write Speed = 649.0638720 Mbps Read Time = 9.2783818 Seconds Read Speed = 882.9125760 Mbps

Version 9.1.1 Drives read as SATA
Test File: Z:\homes\user\NW_SpeedTest.dat Write Time = 0.2432111 Seconds Write Speed = 657.8646880 Mbps Read Time = 0.1889158 Seconds Read Speed = 846.9379680 Mbps Test File: Z:\homes\user\NW_SpeedTest.dat Write Time = 11.9194912 Seconds Write Speed = 687.2776560 Mbps Read Time = 9.9129474 Seconds Read Speed = 826.3939760 Mbps

@cj : Thanks a lot for your help and support.

 

[cyberjock]

Howdy Noob. Just got your PM.

I'm glad your system works, but I'd be interested to understand how the scrub fixed or didn't fix things. It's possible to report no errors but still have corruption if it "thinks" it fixed a corrupted block but the block was corrupted beforehand in RAM.

We're really in uncharted waters here because nobody has had a mountable pool after bad RAM. If you have files that were sourced from someplace else and then copied to your server it would be interesting to see of their MD5s changed. It's probably impossible to really know how bad/good your data is, but you are very lucky as you are the first to not lose their entire pool. Of course, you may find a lot of bad files in the future and not be too thrilled with that. But only time will tell.

 

[N00b]

CJ, here are two sample logs of the scrub process.
First Scrub

Code:

  pool: rock
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: scrub in progress since Sun Nov  3 06:38:19 2013
        841G scanned out of 6.99T at 309M/s, 5h48m to go
        0 repaired, 11.75% done
config:

    NAME                                            STATE     READ WRITE CKSUM
    rock                                            ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/61144d95-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0
        gptid/6204197b-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0
        gptid/62b13961-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        rock:<0x49a09>
        <0x6485>:<0x49a09>
        <0x6485>:<0x499df>

2nd Scrub

Code:

  pool: rock
 state: ONLINE
  scan: scrub repaired 0 in 5h19m with 0 errors on Sun Nov  3 13:19:14 2013
config:

    NAME                                            STATE     READ WRITE CKSUM
    rock                                            ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/61144d95-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0
        gptid/6204197b-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0
        gptid/62b13961-836b-11e2-8960-001aa0e32c82  ONLINE       0     0     0

errors: No known data errors

Let me figure out how to run MD5 on a lot of files as most of the me are scattered over multiple directories and machines (ie. on home pc , work laptop etc). They are all in one directroy on the nas. It wouldn't help to run Md5 on just a few files would it?

 

[cyberjock]

It's actually really complex. If the data was correct in RAM but the parity is trashed then I think a scrub will fix it. But if the data was corrupted in RAM then it's just trashed. There is no fixing it because ZFS doesn't know any better.

As for doing MD5 comparison you can just do md5sum * and get a list of sums for directories. But even if they match you have to consider things such as where the other non-server copy came from. If they came from the server originally they can be corrupted too since they have the same source and the md5sums will match giving the false impression they aren't corrupt. If you did rsync they could be corrupted for the same reason. To add to the confusion, if your RAM went bad just a day or two before and if you hadn't run a scrub or an rsync/zfs replication to your backup server since the RAM went bad then only your newest data has a chance to be corrupt.

So its one of those things where you can only make these two solid conclusions:

1. If the files from your server and the backup match the files may still be corrupt.
2. If the files from your server and backup don't match then at least one of them is corrupt. (Probably only 1 but there's no way to have a reasonable assuredness of this).

Notice I didn't provide a way to prove the files aren't corrupt. That will depend heavily on when and how you do backups and when your RAM went bad along with some random dumb luck. Since you don't really have much control of the those 3 things or their relationships its darn near impossible to be 100% sure(or even reasonably sure) of anything. :(

But if you have some "newer" files that total large quantities of disk space that might be an interesting comparison. Preferably a total number of bytes exceeding 5x your RAM to increase the likelyhood of the bad RAM being allocated.

You are welcome to do the md5sum(and it goes pretty fast from the CLI) but you really won't prove that files aren't corrupt but you might prove that some are. :p

If you are actually interested in trying this out I can help you via teamviewer. Just PM me with a skype name or some way to chat with you and I can help you test it out. I'm somewhat curious to see what the results are, but its more of a curiosity than a way to prove your files are safe from corruption.

 

[N00b]

So as it turns out I had some files on my desktop that I had copied and reorganized on the server when I first set this up at the beginning of the year. I am pretty sure that I didn't rsync them once I had them copied since they have now been grouped into folders etc and in some cases password protected. Other files on the server are from other sources like my personal files from my work laptop etc so the sample size was only 37. Files I could match using excel. See attached results. There are 3 differences but seem to be logical reasons for 2 of them (the files were password protected for whatever flimsy security a password on a file provides!!).

The third is odd but I have noticed this while copying files between my Windows machine and the server. (or maybe even 2 windows machines)!. The size on disk differs but the size of the file is shown to be the same:
Desktop file shows : Size : 2.42 MB (2,546,265 bytes) Size on Disk: 2.42 MB (2,547,712 bytes)
Server file shows : Size : 2.42 MB (2,546,265 bytes) Size on Disk: 2.42 MB (2,546,688 bytes)

I will nevertheless be saying a few extra prayers for some time to come.

(I have neat excel file with conditional format -red bold - for false but I am not allowed to upload an excel file) :)

 

[cyberjock]

For the 3 files that don't match, should they be identical? I realize that them opening is a good sign. Also when you open them do you know the password or do you just leave it blank and it lets the file open? Did you add the password?

As for the 3 files that didn't match, what file type are they? For example, if they are a video file and there is some corruption in the video you won't be able to identify it unless you actually watch through the whole video. The corruption could be audio and/or video. Even then, the corruption could only affect a few frames of the movie or a split second of audio might have no sound. Or your video program might crash every time you get to a certain spot on the video.

 

[N00b]

For the 3 files that don't match, should they be identical? I realize that them opening is a good sign. Also when you open them do you know the password or do you just leave it blank and it lets the file open? Did you add the password?

As for the 3 files that didn't match, what file type are they? For example, if they are a video file and there is some corruption in the video you won't be able to identify it unless you actually watch through the whole video. The corruption could be audio and/or video. Even then, the corruption could only affect a few frames of the movie or a split second of audio might have no sound. Or your video program might crash every time you get to a certain spot on the video.

CJ, all the three files were pdfs; scans of some documents. They open up but if I am missing something on the page /document I am unable to tell. I added passwords when I was carrying copies on thumb drive. Again of not much use or deterrent for someone wanting to break into one but ..... The password was exactly what I wanted it to be. I didn't try a different password but I did just now and it gives me an error. :D