ZFS Samba (CIFS) = abysmal performance in FREENAS 8.2

[TheWiz]

Hi,

I had a problem copying large file (>3GB). Copying wasn't flow and was slow.
It starting at 100MB/s for 15-20sec then drop to 0 for 3 sec and again. Like that:

My network is:
1. Giga network using Cat 5e & TP-LINK TL-WR1043ND
2. Desktop Computer - W7, Intel i5-2500 3.3GHz, 8GB RAM, WD Green WD20EARS 2GB 64MB 3GB/s.
3. NAS - FreeNAS-8.2.0-BETA2-x64 (r10638), AMD Liano A6-3500, 8GB RAM, 4X WD Green WD20EARS 2GB 64MB 3GB/s.


After i change Router Firmware, Re-installing FreeNAS and other staff i go the solution of Jeremy Chadwick.

I try to copy article here but i get forum error.strange. (Upload here)

Now my graph is like that:

I get stable 100MB/s (!)

Good day.

 

[drinking12many]

These setting helped with the bursty transfers I was seeing to I had 6GB of ram and set the vfs.zfs.txg.write_limit_override to about 756MB

 

[ProtoSD]

This seems to be a very popular topic and seems to affect only certain people, I am one of them. I saw that link along with at least a dozen others with similar tweaks and none of them seem to help. There doesn't seem to be any silver bullet for those of us experiencing this problem, but the symptoms appear to be the same.

@TheWiz, what is that app you're using to display that graph?

Anyway, with 8.04 and 8.2, there are 2 python utilities you can use from the command line (http://doc.freenas.org/index.php/Arcstat ). They haven't exactly helped me yet because I'm still trying to understand the output, but today I noticed during a read (copying from my NAS) that prefetch didn't seem to be occurring. I tried setting both 0 and 1 (sysctl vfs.zfs.prefetch_disable 0 or 1 from the command line). I don't know if that's a clue, but all the suggested tests for benchmarking read/write performance from ZFS say my disks can do 300+MB/s yet I'm only seeing it read a fraction of that.

AIO doesn't seem to make any difference, though some other tweaks have evened things out some I know it can be better. Part of the problem is CIFS/Samba. Supposedly Samba 4 is going to be better, but I don't think Samba is completely to blame.

 

[Hexland]

I agree that Samba isn't to blame.

I'm experiencing the same 'breathing' effect as illustrated above -- but I'm not using CIFS or NFS to transfer.

I have 6x2TB drives configured as Raid-Z2, and the entire ZFS volume is shared as an iSCSI device. I have a Windows machine at the other end with the iSCSI device mounted and formatted as NTFS. Reading/Writing to the iSCSI device shows the same on/off effect as show above.

If I didn't know any better, I'd say that ZFS stalls when flushing a write buffer... but I've found no way to prove it (except that if I use a 6x2TB UFS stripe set, there's no stalling)

 

[TheWiz]

I m using Bandwidth Monitor.

 

[praecorloth]

I agree that Samba isn't to blame.

I'm not so sure about this. A bunch of friends and I were having such a wide variety of performance issues, we decided to sit down for an entire afternoon and in to the evening with FreeNAS 8.0.4 and start benchmarking. The first thing we did was test the network using iperf. To and from the FreeNAS box from the proposed client, we were getting 900+megabits per second. So the network was clear.

The next thing was to test FTP transfer to the FreeNAS box at various levels of RAID. All tests came back around the 100 megabyte per second range. Theoretically on gigabit we should get up to 125 megabytes per second, but 100 megabytes seems acceptable to me.

The next thing was to test CIFS transfer to FreeNAS at various levels of RAID. All tests came back at about 30 megabytes per second. Holy crap.

Unfortunately all of the documentation we had regarding the tests has been lost to a re-OS of a laptop. With all the fun we were having, no one thought to put the results on the production FreeNAS fileserver. Oh sweet irony, open your arms! :) But since there is such an interest in this topic, I would be happy to run the tests again at some point and post the hardware, procedures, and results. Maybe if we can all put our heads together, we can figure this thing out.

 

[ProtoSD]

Praecorloth,

That sounds like a really great little hacking session, wish I could have participated. I've been done a lot of testing on my own but still haven't quite found a magic bullet. I think there are *some* things you can do to make CIFS run a little smoother, but I do think the bottom line is ZFS. It would be interesting to see those test results if you have a chance to run them again. I started to document some stuff, but gave up when I wasn't really getting any significant results.

If I didn't know any better, I'd say that ZFS stalls when flushing a write buffer..

I'm also quite certain this is what's happening. I found a video, I think it was Ben Rockland?, a ZFS guru from Oracle, he was talking about this (ZFS breathing, txg_timeout?) and said it was completely normal.... Hmmmm

Anyway, if that's the case there ought to be some buffer that can be tuned so it has enough data to send while ZFS is taking its nap.

There are some nice tools built in to FreeNAS for analyzing this stuff, but I haven't learned how to use them yet. The new Arcstat scripts that are now included in FreeNAS 8.04 are interesting, but don't tell me anything that really helps yet.

There's iperf, iozone, systat -ifstat, zpool iostat -v 1, gstat -f pattern (like p2), top -aSP, dtrace, diskinfo -tv /dev/disk (ie ada1), and tmux which will let you split up your terminal screen so you can watch multiple things running.

I've been able to get bursts of 125+MB/s, then dropping to 95MB/s for a bit longer, then dropping down to ~50MB. This was with a 20GB VMware vdmk file, and a 50GB zero filled file.

Unfortunately in 5 weeks I'll be leaving town for a few months and won't have direct access to my NAS.

 

[survive]

Hi protosd,

I think you mean ZFS ninja!

I think these are the videos you are talking about. They should be required viewing for any FreeNAS user.

https://blogs.oracle.com/video/entry/becoming_a_zfs_ninja

-Will

 

[ProtoSD]

Hey Will,

Yes, you're right! I must have been thinking about Ben's page on Prefetch.

 

[ProtoSD]

I've run into this infuriating problem as shown in the image below. I started using Bandwidth monitor, thanks to TheWiz. My transfers (reading from NAS) start out and go at whatever speed for a short bit, then drop to ZERO ("nap"), resume and drop to ZERO again (this repeats), BUT always after the second "nap" there is period of really slow transfer (9MB/s), then it jumps back up again after a bit. This happens consistently after that second dip/nap, if I run top, nothing is different, no higher CPU, no disk activity, nothing, the transfer just goes to shit for that period. The average transfer speed is somewhere between 50-60MB/s for a ~20GB file.

EDIT: Nevermind, CPU didn't jump up. The time between dips seems to average about 60 seconds.

 

[Gnome]

Have you considered that the NIC driver may be the issue?

I experienced the EXACT same symptoms on a RealTek NIC. Turned out the driver was crashing (0mb/s throughput) and the kernel would restart the driver.

Try an Intel Pro/1000, might just solve all your problems.

My personal recommendation if you have open PCI-Express slots Intel Pro/1000 CT or if you only have PCI slots, Intel Pro/1000 GT.

Remember all PCI slots share bandwidth, so if you take the PCI version you might find your performance quite bad if your motherboard has other devices on the PCI bus or you have other PCI cards installed. PCI-Express is recommended...

 

[ProtoSD]

Hmmm interesting, unfortunately my mobo has dual on board nics and I only have one pci-e slot which I'm hoping to add another disk controller to at some point. But, you have a good point that I hadn't considered and I'll definitely look into it some more.

Thanks Gnome!


EDIT: Both of my on board NICs are Intel 82574L

 

[Gnome]

You could even try Intel motherboards. Even Asus motherboards at the higher end are now shipping with Intel NIC chips standard (I read on Anandtech this was because of complaints by Asus customers)

The problem IMHO isn't the NIC hardware but the software and Intel has the best drivers (at least IMHO they do).

My personal setup is:
Intel DH67BL (Sandy Bridge, Socket 1055)
Intel 2100T (Low power, 35w TDP CPU)
4x4GB Kingston 1333Mhz Desktop RAM (16GB Total)
IBM M1015 (Popular 8x6GBit SAS Adapter which I flashed to IT mode with no BIOS)
RAID-z1 across 3x2TB WD20EARS

My motherboard has an Intel NIC onboard (Intel 82579V controller). It is fully supported by FreeNAS and is reported as Intel Pro/1000 (by the driver).

I also have an Intel Pro/1000 CT (Pci-Express version) with which I tested. Both the onboard NIC and PCI-Express version has exactly the same performance on the setup above:
105megabytes/s VIA Gigabit LAN on Samba.
My tests data is a 60GB Random Pattern File copied from an SSD (120GB OCZ Vertex 3) to my NAS (RAID-z1 across 3x2TB WD20EARS). I had to copy from my SSD on my desktop system because standard hard-drives couldn't maintain that kind of throughput. I used a 60GB Random pattern file because I felt it would not result in a cache hit (memory) and because it is incompressible. Small files cause around 60-100mb/s throughput depending on how many there are.

Speed never drops below 105megabytes/s on large files. I get the same speed on read/write. There are 2 switches (Layer 2 Gigabit) between the test computer and the NAS.

My daily job is software developer but I also work on many production servers (RHEL Linux and IBM AIX servers mostly but some FreeBSD and Solaris Sun-fire servers). In my experience (On Linux, FreeBSD and Solaris) using Intel NIC adapters have been the most stable by far!

Both Broadcom and Realtek have given me much grieve in the past. I wouldn't ever use them in an OS other than Windows (and actively avoid them even there).

My 2c.

 

[ProtoSD]

I don't know if you just saw the update to my post above, but both my onboard NICs are Intel 82574L. So according to your logic they shouldn't be the problem.

I know others with the same motherboard that had the performance you're talking about and it seems to have dropped around 8.0.4.

I'm still digging into driver changes or tuning that might help.

 

[Gnome]

No it shouldn't be the NIC then.

My performance hasn't changed last I tested. I'm currently running 8.0.4 p1 (latest stable in other words).

Maybe it has something to do with the amount of RAM I'm using?

 

[ProtoSD]

I don't think it's your RAM, I've seen others with 4 or 8GB get performance equal to yours.

My NICs also shows "em0: <Intel(R) PRO/1000 Network Connection 7.1.9>"

I have just found a couple of reports from people complaining of a "wedging" condition happening that is causing the driver to pause. I'm still looking into the details.

 

[Joshua Parker Ruehlig]

http://freebsd.1045724.n5.nabble.com/82574L-hangs-with-r233708-e1000-driver-td5624584.html
Hey Protosd, your intel nic, but the problem this guy and I have been experiencing is a complete stopping of the nic. The IP is unreachable and doesn't come back online (until you force it too).

 

[ProtoSD]

http://freebsd.1045724.n5.nabble.com/82574L-hangs-with-r233708-e1000-driver-td5624584.html
Hey Protosd, your intel nic, but the problem this guy and I have been experiencing is a complete stopping of the nic. The IP is unreachable and doesn't come back online (until you force it too).

Thanks, yeah I saw that. I'm just digging through them to eliminate the driver as a possibility, though it doesn't seem to be. There are some tunables you can change, and there are newer versions of the driver 7.32 vs. 7.19. I might give that a try just to see if there's any improvement.

My performance isn't horrific, but I know it can be better.

 

[Bever]

Looks like I have the same issue to deal with. I'll let you know of the outcome... I have 6GB memory, so I don't really know what settings I need to adjust for that...

Edit: that was CIFS, this is FTP and it's a whole lot worse (one big file copy):

 

[Bever]

OK, so tuning didn't go so well...

Next on the list: make the thing boot again...