Performance questions regarding my first build

[sqwob]

I've just completed my first tests with my first build, and was wondering if there might be a bit more performance in the config i have, that i'm not getting out of it.

Build FreeNAS-8.3.0-RELEASE-p1-x64 (r12825)
Realtek NIC
Platform Intel(R) Pentium(R) CPU G860 @ 3.00GHz
Memory 7856MB (I know, but since i al a single home user this shouldn't be the issue in this case)
Mobo ASRock B75 Pro3-M
Hard Drives 4 x 2Tb Seagate (ST2000DM001) in RAIDZ, forced 4K block size

Copying to CIFS (1 large file at a time, i get write speeds of 50MB/sec, and read speeds of 50MB/sec)
Copying to ISCSI I get a light increase to 54-67MB/sec, with an occasional 74MB/sec peak)

Here is a screenshot of some test transfers:

For some reason i haven't figured out yet, i don't see system load, cpu usage or memory usage in the reporting section.

I'd like to eliminate my network as a bottleneck, but am still figuring our how to benchmark the nas locally. I'll add results as soon as i've figured it out.

Local disk performance tested using DD:
42991616000 bytes transferred in 110.124729 secs (390390210 bytes/sec) or 372 MB/sec (sounds almost too fast? Cache?)
68052582400 bytes transferred in 152.006071 secs (447696476 bytes/sec) or 426 MB/sec (sounds almost too fast? Cache?)

I'm now in progress of running:
iozone -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -s 6g -i 0 -i 1 -i 2

Some incomplete results (it's still running and it looks like it's going to take a while)

Run began: Tue Jan 22 14:56:00 2013

Record Size 4 KB
Record Size 8 KB
Record Size 16 KB
Record Size 32 KB
Record Size 64 KB
Record Size 128 KB
File size set to 6291456 KB
Command line used: iozone -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -s 6g -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
6291456 4 369552 147531 424735 419544

If anyone has pointers (to relevant documentation) or tips, it's very much appreciated!

Kind regards
Rik

 

[cyberjock]

Id definitely get more RAM. You really have only 2GB for ZFS cache, which is VERY small for a ZFS pool of your size. I'd go with 12GB(16GB would be even better). Even my home system where I am a single guy and the only user I had to install more than 12GB to get even decent speeds.

Edit: Also, Intel NICs can give you pretty significant performance increases. If your NIC isn't an Intel I'd seriously consider spending $25 on one. It's money well spent! Realtek aren't optimized well and typically people with Realtek can't get anything above about 60-70MB/sec.

 

[sqwob]

The complete results from the iozone benchmark:



Am i interpreting them correctly that the raid has contininuous reads at speeds between 424.000kb/secand 596.104kb/sec?

 

[cyberjock]

Nope. Those tests are FAR more complex than you think. I tell people if you haven't read up on the test and how to use it then you are probably using it wrong and the values don't mean what you think.

You really want to know how it will work, build the zpool how you want and try to use it. Then work from there to find your bottleneck if you aren't happy.

Frankly, until you add more RAM you are really just wasting your time... I'll never understand why people even think you know better using only 8GB when the manual would recommend double the RAM.. If Windows says 1Ghz or faster CPU would you install a 500Mhz CPU?

Note: nobody else is posting.. are they? Us senior guys are so sick of the same problem we gave up. Use the search.. you'll see that you aren't alone with needing more RAM. They've likely dismissed you because you are ignoring the advice and getting more RAM which I'm not only recommending, but the manual recommends.. but whatever. I'm unsubscribing. Take care.

 

[sqwob]

Concerning the IOZone command, i spend over an hour reading up on the forum and references documentation before I executed any command, and this one came specifically for comparing performance of 4K cluser size systems, i'm having trouble relocating the topic though...

Since i'm getting opposite information from other senior members about the exact same topics i'm a bit sceptic that 8gb additonal RAM would solve my performance issue. I'm transferring files much larger than any piece of memory can hold thus exceeding the cache's abilities to enhance performance (unless i'm completely missing the point how the memory is being used?)

Quote Originally Posted by JaimieV View Post
Lightly loaded domestic NASes do *not* need as much as the folks are saying, I do wish they'd remember that... RAM scales with usage as well as disk.

Thanks for injecting some sanity into the discussion.

Regarding the Intel NIC (that i ordered anyway), i'm also reading extensive freenas tests where the performance difference is poor or nonexisting: http://forums.freenas.org/showthread.php?10325-Intel-NIC-vs-RealTek-NIC-Performance-Testing

disregarding the above, i can understand providing voluntary support here is tiresome in case of recurring questions.

 

[sqwob]

Reading up on this article a few things are cleared out:

#1: Add Enough RAM

A small amount of data on your disks is spent for storing ZFS metadata. This is the data that ZFS needs, so it knows where your actual data is. In a way, this is the roadmap that ZFS needs to find its way through your disks and the data structures there.

If your server doesn't have enough RAM to store metadata, then it will need to issue extra metadata read IOs for every data read IO to figure out where your data actually is on disk. This is slower than necessary, and you really want to avoid that. If you're really short on RAM, this could have a massive impact!

How much RAM do you need? As a rough rule of thumb, divide the size of your total storage by 1000, then add 1 GB so the OS has some extra RAM of its own to breathe. This means for every TB of data, you'll want at least 1GB of RAM for caching ZFS metadata, in addition to one GB for the OS to feel comfortable in.

Having enough RAM will benefit all of your reads, no matter if they're random or sequential, just because they'll be easier for ZFS to find on your disks, so make sure you have at least n/1000 + 1 GB of RAM, where n is the number of GB in your storage pool.

Still, this is assuming a multiple user load, and once it finds the data in the metacache, a long stream of a single file that exceeds whatever can be cached in memory. After reaching this point I assume i'm looking a raw transfer speeds that my nas & network allow me...

 

[sqwob]

come to think of it, my workstation has the same RAM modules, i'll shove in an additional 8gb later today and see what impact it has.

 

[William Grzybowski]

iozone came up with pretty good results, that would saturate gigabit link very easily.

Your next step is benchmark network transfer, use iperf tool for this.
Since it is an asrock board with a realtek NIC thats is probably the smoking gun in my opinion.
Note that CIFS is very CPU bound and single threaded (see top while transfering).
FTP is one of the simplest network protocol to test network speed.

EDIT: I would not take cyberjoke RAM advices too seriously :D Yes, RAM can help a lot but there are a few items to consider: 1. Home user, probably no multiple connections at a time 2. I doubt your pool is full, the math for ram usually involves used space data and not total space.

 

[sqwob]

I broke my main pc removing the ram (don't ask me what/how, still haven't figured it out), no benchmarking for a while i guess :(

 

[praecorloth]

Sad times. Sorry to hear about your desktop. :(

When testing iperf, make sure you take turns with FreeNAS acting as the server and as the client, and run multiple tests in each direction. If you see less than 900Mbit/s in either direction, you have a networking problem. When I did my testing, for the longest time I wrote off the iperf result. Since I wasn't even achieving 400Mbit/s speeds, it shouldn't matter that iperf saw only 640Mbit/s performance on the line. Once I hit the 600Mbit/s bottleneck, I'll troubleshoot the network. Mistaaaaaaaaake! Don't ignore it. It means that there are problems on your line. Fix them.

I will report, however, that I have seen gigabit speeds from Realtek based NICs. A better NIC can help mitigate the slowdowns you see from an over taxed system. For example I'm running,

Athlon 64 x2 5200+
5GB memory
3x 500GB drives in RAIDz

Moving from a Realtek NIC to an Intel NIC bumped me from ~30MB/s to ~50MB/s. My previous incarnation was

Pentium 4 3.2GHz
1.5GB memory
3x 500GB drives in RAIDZ
Realtek gigabit NIC

Moving from the Pentium 4 with 1.5GB memory to the Athlon 64 with 5GB memory bumped me up about 5MB/s.

Anyways, if you're wondering about caching issues, run iostat 1 on the FreeNAS box when you're testing. If your RAID drops to 0.00MB/s and you're still transferring at a steady speed, you've hit cached data.

 

[sqwob]

I managed to get my desktop back up and running, but was only able to add another 4gb to the Freenas machine (not at 12gb RAM with 8TB storage in it)

I ran Iperf and get these results:

[160] local 10.0.0.100 port 50299 connected with 10.0.0.6 port 5001
[ ID] Interval Transfer Bandwidth
[160] 0.0- 1.0 sec 114768 KBytes 940179 Kbits/sec
[160] 1.0- 2.0 sec 115192 KBytes 943653 Kbits/sec
[160] 2.0- 3.0 sec 112176 KBytes 918946 Kbits/sec
[160] 3.0- 4.0 sec 115120 KBytes 943063 Kbits/sec
[160] 4.0- 5.0 sec 114088 KBytes 934609 Kbits/sec
[160] 5.0- 6.0 sec 115200 KBytes 943718 Kbits/sec
[160] 6.0- 7.0 sec 113184 KBytes 927203 Kbits/sec
[160] 7.0- 8.0 sec 114520 KBytes 938148 Kbits/sec
[160] 8.0- 9.0 sec 114784 KBytes 940311 Kbits/sec
[160] 9.0-10.0 sec 115160 KBytes 943391 Kbits/sec

CIFS & iscsi copy speeds seem to have gone up minimally, averaging at 60MB/sec now

 

[William Grzybowski]

Did you test both ways? FreeNAS as a client and server?

Also, have you tried transferring over FTP?

You do not have autotune on, right? You shouldn't.

 

[sqwob]

I'm testing in the other direction at the moment (Freenas -> workstation), results are:

[184] 55.0-56.0 sec 92553 KBytes 758196 Kbits/sec
[184] 56.0-57.0 sec 92633 KBytes 758852 Kbits/sec
[184] 57.0-58.0 sec 91943 KBytes 753194 Kbits/sec
[184] 58.0-59.0 sec 91701 KBytes 751218 Kbits/sec
[184] 59.0-60.0 sec 93355 KBytes 764764 Kbits/sec
[ ID] Interval Transfer Bandwidth
[184] 60.0-61.0 sec 93643 KBytes 767127 Kbits/sec
[184] 61.0-62.0 sec 92355 KBytes 756574 Kbits/sec
[184] 62.0-63.0 sec 91635 KBytes 750678 Kbits/sec
[184] 63.0-64.0 sec 92310 KBytes 756204 Kbits/sec
[184] 64.0-65.0 sec 92753 KBytes 759829 Kbits/sec
[184] 65.0-66.0 sec 91520 KBytes 749735 Kbits/sec
[184] 66.0-67.0 sec 91284 KBytes 747801 Kbits/sec
[184] 67.0-68.0 sec 92764 KBytes 759922 Kbits/sec
[184] 68.0-69.0 sec 92683 KBytes 759262 Kbits/sec
[184] 69.0-70.0 sec 94033 KBytes 770320 Kbits/sec
[184] 70.0-71.0 sec 92845 KBytes 760582 Kbits/sec

I'll be setting up FTP for further testing now.

 

[sqwob]

FTP write speeds started out at a steady 75MB/sec but after 1GB it dropped to 30-50MB/sec (quite jumpy)

uploads did have an issue, halting 10 seonds when it "finished" completing the large file before uploading the next ones...

 

[sqwob]

You do not have autotune on, right? You shouldn't.

I've just checked and it's off

 

[William Grzybowski]

Where are you copying from or copying to? Are you sure the destination/source is capable of disk read/write superior to what you have been encountering ?

 

[sqwob]

Workstation = 256gb Crucial SSD on SATA2, it should be plenty fast ;)
Freenas = RaidZ over 4 drives

 

[William Grzybowski]

Well, I'm out of ideas. To be honest it looks like an intel card could be of some help although I can't be sure. The local tests with iozone went just fine.

 

[sqwob]

Well, I'm out of ideas. To be honest it looks like an intel card could be of some help although I can't be sure. The local tests with iozone went just fine.

there's a card being shipped to me as we speak, also some more ram (that i doubt make that much of a difference now, but will do when i start experimenting with jail / plugins). i'll post some more into this topic when it's installed.

 

[praecorloth]

I'd say be wary of those iperf results. Though maybe more testing on that can wait until the Intel NIC arrives.

In the mean time, redo your FTP test with iostat 1 running on the FreeNAS box. Check your transfer speeds vs how fast the data is being written to the drives. Just for grins.