Unable to break the 9MB/s barrier

[Crab Balls]

I'm at a loss on how to get more speed out of this thing. It is painfully slow, but other than that it is working like a charm. I'm not expecting much, but I feel it should be doing better. I would be thrilled with 50 or 60.

My setup is a Supermicro SYS-5015A-EHF-D525 1U Intel Atom D525 with 8GB RAM and a single 1TB drive. It is running ZFS. Only thing running is 1 Jail with Plex Media Server and a few Windows shares. I do realize better hardware and more RAM should help, but the CPU usage seems low, so I don't think it is that. I am sure I either did something wrong or am overlooking a something simple. My feeling is I should be getting better performance even with this hardware. Any ideas would be appreciated. Thanks.

 

[zambanini]

check the obvious..link speed.

search for iperf within the forum, with this tool you can meassure the interface speed.

also provide your testcase.


btw zfs and a single disk will prevent error checking on the filesystem

 

[Crab Balls]

iperf is showing around 453 Mbits/sec.

Sorry, I don't know what testcase is.

 

[zambanini]

iperf should be much higher. nic or switch problem...solve it. when you do not know your test case, nobody can help you...

 

[gpsguy]

You probably have a 100 Mbps component in your network. Possibly a switch/router connection, NIC, or bad cable.

I doubt you'll see 50-60 MB/s with that hardware. My guess, would be more like 30-40 MB/s. CIFS is single threaded - so a faster CPU's would be better.

You might want to try a direct network (cabled) connection between your client and the server. You'll need a static IP address on both devices.

If you don't/can't do the above, please provide detailed information as to how every thing is connected. Switch/router (including make/model numbers), client machine (spec's, OS, NIC/speed), cabling (CAT5e/CAT6,?)

 

[Crab Balls]

You probably have a 100 Mbps component in your network. Possibly a switch/router connection, NIC, or bad cable.

I doubt you'll see 50-60 MB/s with that hardware. My guess, would be more like 30-40 MB/s. CIFS is single threaded - so a faster CPU's would be better.

You might want to try a direct network (cabled) connection between your client and the server. You'll need a static IP address on both devices.

If you don't/can't do the above, please provide detailed information as to how every thing is connected. Switch/router (including make/model numbers), client machine (spec's, OS, NIC/speed), cabling (CAT5e/CAT6,?)

Thanks! I've been plugging away at this and now I'm getting consistent 30MB/s file transfers. This is very encouraging. I'm already using a static IP on this so I will plug the laptop directly into FreeNAS via crossover cable and see what I get.

All connections are gigabit. The router is another Supermicro SYS-5015A with pfSense and an Intel gigabit quad port nic. NAS is on its own vlan. Cabling is CAT5. Checking in pfSense I did notice a bunch of In/Out errors.

I got an i7 machine with 16BG and a few HDs I can use but I'm hoping to fine tune this as it uses so little power.

 

[Crab Balls]

Direct connection with laptop via crossover cable is transferring files at 46.9MB/s. I can live with that if I can solve the problem. I'm starting to think it is wiring.

 

[pirateghost]

All connections are gigabit. The router is another Supermicro SYS-5015A with pfSense and an Intel gigabit quad port nic. NAS is on its own vlan. Cabling is CAT5. Checking in pfSense I did notice a bunch of In/Out errors.

Being on it's own vlan means it has to traverse the router interfaces in addition to the rest of the network. Firewall and routing can indeed clog things up. You should put it on the same LAN as your clients.

 

[Crab Balls]

Being on it's own vlan means it has to traverse the router interfaces in addition to the rest of the network. Firewall and routing can indeed clog things up. You should put it on the same LAN as your clients.

I've been thinking that as well. I now ruled out any wiring issues. I plugged the laptop directly into the switch with a short cable and still getting around 36MB/s. I will have to plug directly into the router to confirm that I'm taking a performance hit with the vlan issue. If this is the case I will just live with the performance hit as I prefer to keep things isolated and contained. I do access this remotely through a VPN so having that extra layer of protection is important to me. I did a test of downloading the same 3GB file with two machines and the sum of both files was peaking in the mid-40s. I guess I can temporarily put it on the same lan for testing.

 

[cyberjock]

An Intel D525 is wholly inappropriate for FreeNAS. I've got one of those for a pfsense box, and despite having no overhead from things like jails, ZFS, etc it can only handle about 300Mb/sec of throughput. So I'm not surprised in the slightest that performance is terrible. It's gonna *be* terrible because you've got horribly underpowered hardware.

Consider the fact that the Pentium G2020 ($55 or s0) can score about 2800 (http://www.cpubenchmark.net/cpu.php?cpu=Intel+Pentium+G2020+@+2.90GHz) and is about the lowest-end CPU we recommend.

Consider the fact that the D525 can score only 700 (http://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+D525+@+1.80GHz).

So yeah, you have CPU problems. I wouldn't even *consider* a box with a CPU lower than the G2020 or that didn't have the same amount of CPU cache as the G2020. You're CPU does 1/4 of what a G2020 can do.

 

[Crab Balls]

An Intel D525 is wholly inappropriate for FreeNAS. I've got one of those for a pfsense box, and despite having no overhead from things like jails, ZFS, etc it can only handle about 300Mb/sec of throughput. So I'm not surprised in the slightest that performance is terrible. It's gonna *be* terrible because you've got horribly underpowered hardware.

Consider the fact that the Pentium G2020 ($55 or s0) can score about 2800 (http://www.cpubenchmark.net/cpu.php?cpu=Intel Pentium G2020 @ 2.90GHz) and is about the lowest-end CPU we recommend.

Consider the fact that the D525 can score only 700 (http://www.cpubenchmark.net/cpu.php?cpu=Intel Atom D525 @ 1.80GHz).

So yeah, you have CPU problems. I wouldn't even *consider* a box with a CPU lower than the G2020 or that didn't have the same amount of CPU cache as the G2020. You're CPU does 1/4 of what a G2020 can do.

Thanks for your help. Sorry I didn't get back to you sooner. I finally had a chance to get back to this today and found a few interesting things. I loaded the latest version of FreeNAS on an i7 Supermicro box (i7-2600K @ 3.4GHz) with 16GB RAM and am getting the same crappy transfer rate as the D525 box (peaking 451 Mbits/s -- 53.7 MB/s). I confirmed this using iperf. Hooked up an old i7 laptop (Ubuntu) on the same vlan and the thing is pushing a consistent 782 Mbits/s -- 93.2/MB/s transfer with iperf. Even with the slight performance hit using the vlan on the pfSense box, I am ruling the router out as the culprit and am very happy with the speed it can comfortably achieve. This leads me to believe my problem is in both FreeNAS boxes. I can accept the the D525 being a dog, but the i7 w/ 16GB should have given me a slight boost, it didn't.

Using FreeNAS-9.3-STABLE-201412240734

I'm open to suggestions on which way to go with this. Thanks again!!!

 

[jgreco]

An Intel D525 is wholly inappropriate for FreeNAS.

You know these used to be recommended as entry-level FreeNAS boxes back in the 8.0 days.... ;-)

 

[Crab Balls]

You know these used to be recommended as entry-level FreeNAS boxes back in the 8.0 days.... ;-)

I'm starting to think an i7 box is inappropriate for entry level FreeNAS. Might have to bump it up to a quad Xeon with 192GB RAM!:)

 

[jgreco]

Well, if you want to be like that about it, my first response to you was going to be along the lines of tie a brick to it and throw it out the window... heh

But yeah FreeNAS is resource heavy because the system is taking on the duties of a hardware RAID controller.

 

[Apollo]

btw zfs and a single disk will prevent error checking on the filesystem

You do get error checking at block level, however you do not have redundancy. Two different things.

 

[Crab Balls]

Well, if you want to be like that about it, my first response to you was going to be along the lines of tie a brick to it and throw it out the window... heh

But yeah FreeNAS is resource heavy because the system is taking on the duties of a hardware RAID controller.

Just trying to have a little levity in this situation, no harm meant.

 

[jgreco]

levity

Dude, I am talking gravity here, not levity! ;-)

 

[cyberjock]

You know these used to be recommended as entry-level FreeNAS boxes back in the 8.0 days.... ;-)

Yep. I remember. ;)

Well, you're using desktop hardware, so there's always the possibility that your desktop NIC isn't really a good NIC for throughput. For example, if you are using Realtek you can expect shit performance no matter what.

Thanks for your help. Sorry I didn't get back to you sooner. I finally had a chance to get back to this today and found a few interesting things. I loaded the latest version of FreeNAS on an i7 Supermicro box (i7-2600K @ 3.4GHz) with 16GB RAM and am getting the same crappy transfer rate as the D525 box (peaking 451 Mbits/s -- 53.7 MB/s). I confirmed this using iperf. Hooked up an old i7 laptop (Ubuntu) on the same vlan and the thing is pushing a consistent 782 Mbits/s -- 93.2/MB/s transfer with iperf. Even with the slight performance hit using the vlan on the pfSense box, I am ruling the router out as the culprit and am very happy with the speed it can comfortably achieve. This leads me to believe my problem is in both FreeNAS boxes. I can accept the the D525 being a dog, but the i7 w/ 16GB should have given me a slight boost, it didn't.

Using FreeNAS-9.3-STABLE-201412240734

I'm open to suggestions on which way to go with this. Thanks again!!!

There's lots of suggestions, starting with listing total hardware for the whole test. iperf is independent of the sharing protocol, so we can definitely rule out the protocol.

If iperf is giving poor throughput then you have problems that are almost certainly with the network infrastructure. I know iperf from my d525 when I first installed pfsense wasn't the greatest. But assuming both sides of the test have reasonable power, you should be able to hit 900Mbit/sec consistently every time. This is likely some bad NIC or cabling somewhere. You'll have to test and replace stuff to find the culprit though.

 

[Crab Balls]

Yep. I remember. ;)

Well, you're using desktop hardware, so there's always the possibility that your desktop NIC isn't really a good NIC for throughput. For example, if you are using Realtek you can expect shit performance no matter what.


There's lots of suggestions, starting with listing total hardware for the whole test. iperf is independent of the sharing protocol, so we can definitely rule out the protocol.

If iperf is giving poor throughput then you have problems that are almost certainly with the network infrastructure. I know iperf from my d525 when I first installed pfsense wasn't the greatest. But assuming both sides of the test have reasonable power, you should be able to hit 900Mbit/sec consistently every time. This is likely some bad NIC or cabling somewhere. You'll have to test and replace stuff to find the culprit though.

I've pretty much ruled out any cabling issues and NICs as being the problem. What I did was take my i7/16GB test box that had FreeNAS loaded to it and threw Manjaro on it and tested again on the same vlan and the speed jumped back up to a consistent 782 Mbits/s -- 93.2 MB/s transfer with iperf. All hardware and cabling were identical for the test, only thing different was switching from FreeNAS to Manjaro on the same box. Like I said, I realize I'm taking a slight performance hit running the NAS in a vlan, but getting speeds in the mid to upper 700 Mbits/s is more than acceptable for my needs. Too bad FreeNAS isn't cutting it.

At this point, if I have to buy enterprise hardware to get a software solution to work I might as well buy a dual Xeon box and a decent hardware RAID card and skip the whole FreeNAS ordeal.

 

[jgreco]

A key concept with ZFS has always been to replace the incredibly pricey high end RAID controllers you might use in a large server with dozens or hundreds of disks with a software solution; it is expected that you are instead throwing CPU at the problem because CPU is relatively cheap.

If you take a fully loaded LSI 3108 like the Supermicro AOC-S3108L-H8iR with the CacheVault and CacheCade options you're up around a thousand dollars just for the RAID controller - and you still need a halfway decent system to provide the grunt because CIFS is singlethreaded, so add on at least a low-end Pentium, motherboard, and memory for at least another three hundred dollars, so about $1300.

With FreeNAS we try to throw a low end Xeon, a quality mainboard with an HBA, some memory, and maybe a SLOG device, and that usually ends up somewhat less expensive. Costs can be cut there if you don't need the HBA or performance levels of a Xeon... an 8GB box with an i3 CPU and the non-HBA variant of that board is cheaper still.

"Enterprise hardware" is a red herring because what you actually need is hardware capable of doing what the software requires; we tend to find that no consumer grade hardware does this well, and so we typically suggest the Supermicro boards because they were designed for server use, and are actually cheaper than a lot of the prosumer kit people try to pick out on their own.