Yet Another ZFS Tuning Thread

[TheSmoker]

Hi fokls!
Just upgraded from a FM1 dual core to a FM2 Quad core along with a healthy memory increase from 16G to 32G.
The thing here is that during file copying (around 91G of files) through Samba the mem usage is not going above 10G, like this:


I've ran the autotune script and the sysctls it added are:

Code:

net.inet.tcp.sendbuf_max 2097152
kern.ipc.maxsockbuf 2097152
net.inet.tcp.recvbuf_max 2097152

Also the tunable are:

Code:

vfs.zfs.arc_max 10411027752
vm.kmem_size_max 14459760768
vm.kmem_size 11567808614
kern.ipc.nmbclusters 5000

Any thoughts? Comments? Advices?

Thanks and regards!

Mihai

 

[TheSmoker]

Added some additional graphs in regards to CPU usage (single threaded samba using an max of 15-16% out of an CPU core) and the network bandwidth which does not go beyond 700Mbps. The network card is an onboard Realtek...

 

[cyberjock]

So what's the problem? Your performance seems to be really good for having a Realtek network card...

 

[cyberjock]

Also, you failed to mention your hardware specs, FreeNAS version, etc.

Can't do much good without some more info...

 

[TheSmoker]

Below the specs:
Build FreeNAS-8.3.0-RELEASE-x64 (r12701M)
Platform AMD A8-5600K APU with Radeon(tm) HD Graphics (quad core 3.9GHz)
Memory 32188MB
Motherboard Gigabyte GA-F2A85XM-D3H (8 SATA 6Gbps ports)
HDDs 6 x 2TB WD Green RAIDZ2

The initial question was: "How can I improve the memory usage for ZFS, as at the moment it ONLY uses 10GB out of 32GB?"

Thanks!

 

[cyberjock]

Why do you care about the memory usage? Not to mention the fact that "free" really means "not hard allocated" so it could be cache.

You shouldn't be caring about memory usage at all. As long as its performing and reliable, that's all that matters.

You are getting roughly the maximum speed you can get for your hardware. So what are you hoping to accomplish with higher memory usage?

I hope you aren't looking at numbers and wanting them higher "just because". But that's exactly what it sounds like...

 

[TheSmoker]

I am a FreeBSD addict: my redundant GWs runs FreeBSD, my mail server runs Freebsd, my web server runs FreeBSD ... you get the point ...
I also know what is the difference between FREE and SOFT ALLOCATED memory. :)

Code:

CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 177M Active, 77M Inact, 8810M Wired, 2460K Cache, 212M Buf, 22G Free

so we have ~8G of soft allocated cache and 22G FREE RAM :)

My wish is to start optimizing memory usage so I can squeeze everything bit of performance out of the HW.

So, how do I increase the ZFS memory buffers so the ram will be used as write cache? (i did not used ZFS before, so please be understanding :) )

 

[cyberjock]

So I'll ask again. What "performance" are you hoping to achieve? You said you get just under 700Mbit/sec. That's about all I would expect from your network card.

You keep saying you want more RAM usage, but have not given me any reason WHY you want more. The system may not be using it because your historical usage hasn't deemed it necessary.

 

[cyberjock]

Let me ask this question:

"If your RAM usage was 99% right now but your performance was the same as it is right now, would you STILL be asking for help?"

 

[TheSmoker]

"Unfortunately" that crappy card can do more. :)

Code:

C:\Users\PLEX\Downloads>iperf.exe -c 192.168.1.254 -w 64K -t 60
------------------------------------------------------------
Client connecting to 192.168.1.254, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[156] local 192.168.1.103 port 57696 connected with 192.168.1.254 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-60.0 sec  5.88 GBytes   842 Mbits/sec

I want the disk performance to match network performance and see numbers close to 800Mbps. I believe that can be achieved through RAM cache tuning for ZFS. Can you help out with that? Or at least point me into the right direction?

 

[cyberjock]

I wouldn't bet on getting much above 700Mbit/sec without an Intel NIC. It's in the manual, in my guide, in the FAQ that Intel NICs give the best performance. I haven't used a Realtek NIC, but I've gotten 120MB/sec from my Intel NICs...

I'm really not sure how you are going from "ram isn't fully allocated" to "must be causing low performance of network speeds". Just because stuff is in RAM doesn't necessarily make your network performance go faster.

 

[TheSmoker]

Let me ask this question:

"If your RAM usage was 99% right now but your performance was the same as it is right now, would you STILL be asking for help?"

YES I would have. Especially that I know it's not being limited by neither the network nor the disks.

 

[cyberjock]

Run a dd test of your drives. I'm betting your disk performance is over 200MB/sec.

 

[TheSmoker]

I wouldn't bet on getting much above 700Mbit/sec without an Intel NIC. It's in the manual, in my guide, in the FAQ that Intel NICs give the best performance. I haven't used a Realtek NIC, but I've gotten 120MB/sec from my Intel NICs...

I'm really not sure how you are going from "ram isn't fully allocated" to "must be causing low performance of network speeds". Just because stuff is in RAM doesn't necessarily make your network performance go faster.

I my case I am writting to FreeNAS. So it fills 10G of cache, does XOR calculation and then writes the date on disk. On write operations, experience show, that bigger caches usually helps.

 

[cyberjock]

Right, but you are missing my point. Realtek isn't exactly optimized to work well with FreeBSD. It's a well known fact, search the forums. You'll see LOTS of people complain about their network cards. Those that got Intel NICs suddenly had more performance. So are you REALLY going to tell me that increasing your cache size will fix this when replacing the NIC clearly increased the performance?

Edit: There's NO way it's filling 10G of RAM with a write cache, then flushing it to disk. In fact, if the write cache was over 1GB I'd be shocked. There's a calculation somewhere in the forum that will tell you roughly what your write cache size is. I want to say it was 6x the estimated performance of your zpool but also had a time limit too.

 

[TheSmoker]

Right, but you are missing my point. Realtek isn't exactly optimized to work well with FreeBSD. It's a well known fact, search the forums. You'll see LOTS of people complain about their network cards. Those that got Intel NICs suddenly had more performance. So are you REALLY going to tell me that increasing your cache size will fix this when replacing the NIC clearly increased the performance?

Edit: There's NO way it's filling 10G of RAM with a write cache, then flushing it to disk. In fact, if the write cache was over 1GB I'd be shocked. There's a calculation somewhere in the forum that will tell you roughly what your write cache size is. I want to say it was 6x the estimated performance of your zpool but also had a time limit too.

Still... if the limitation in my case is 845Mbps network, which I accept... RTL SUX ... that does not explain why the disk commit is only 65MBps which is roughly 700Mbps ... do you see the gap? Also the samba process if not the culprit either as it did not managed to max out 1 core ... it hangs at 15% ... so? Any not so aggresive oppinions? :)

 

[TheSmoker]

Here is the dd:

Code:

[root@jukebox] /mnt/tank0# dd if=/dev/zero of=/mnt/tank0/test48G.dd bs=1m count=48000
48000+0 records in
48000+0 records out
50331648000 bytes transferred in 197.302184 secs (255099295 bytes/sec)

 

[cyberjock]

65MB/sec comes to 520Mbit/sec.

You're asking for more network speed with a Realtek NIC. I can tell you from experience on this forum that an Intel NIC WILL give you more speed. If you have a performance issue with your zpool, that's totally different. It could be that your zpool just can't write data fast enough to give you good speeds. Green drives aren't the worlds fastest drives. I have them in 2 FreeNAS servers I work with and they are great. I hit almost 300MB/sec with a RAIDZ2 with a system far less powerful than yours.

Edit: I have never tuned any of my systems beyond the autotune feature of FreeNAS 8.2+.

 

[cyberjock]

So based on your dd, which is almost twice what a 1Gb NIC could theoretically put out, I'd say your limitation is either your zpool's latency(unlikely) or your NIC.

I'd say you have 2 options:

1. Buy 2 Intel NICs and connect a desktop to your server via Intel NICs and see if performance changes. They are cheap($25-30 each).
2. Create a RAMdrive on your FreeNAS server, share it and copy data to it. I'd bet money you'll hit a max, if you are lucky, of about 750Mb/sec. You might see a small improvement because you have a no-latency file system. But you may not see a performance increase over your zpool at all. ZFS's read ahead cache is quite sophisticated, so when you start reading a big file it will start caching far ahead of the file transfer(assuming the zpool can perform). That's why your NIC is most commonly your limiting factor.

Honestly, since you're familiar with FreeBSD I'd love to see comparisons on your system with a 16GB RAMdrive copying files to and from using Realtek, then Intel.

 

[TheSmoker]

So based on your dd, which is almost twice what a 1Gb NIC could theoretically put out, I'd say your limitation is either your zpool's latency(unlikely) or your NIC.

I'd say you have 2 options:

1. Buy 2 Intel NICs and connect a desktop to your server via Intel NICs and see if performance changes. They are cheap($25-30 each).
2. Create a RAMdrive on your FreeNAS server, share it and copy data to it. I'd bet money you'll hit a max, if you are lucky, of about 750Mb/sec. You might see a small improvement because you have a no-latency file system. But you may not see a performance increase over your zpool at all. ZFS's read ahead cache is quite sophisticated, so when you start reading a big file it will start caching far ahead of the file transfer(assuming the zpool can perform). That's why your NIC is most commonly your limiting factor.

Honestly, since you're familiar with FreeBSD I'd love to see comparisons on your system with a 16GB RAMdrive copying files to and from using Realtek, then Intel.

I have an Intel PCIExpress Desktop Adapter somewhere in storage, I will grab it later on tonight as the storage space it's on the other side of the city ...
I will also mount a ramdrive and do some testing ... will keep you posted.