Any suggestions or opinons about performance optimization?

[escalibur]

I just wanted to make sure that my configuration is pretty much as good as it could be. Here is the graph (there are few more network clients (not PCs) connected to my HP switch which are missing from the graph)

- FreeNAS: Intel i3 4330, 16GB 1600MHz DDR3 ECC, AsRock E3C226D2I (latest bios), FreeNAS is running on SanDisk Cruzer Fit 8GB USB stick
- HP is 1810v2 GB 8 port switch (all ports are in the same VLAN (default configuration))
- Asus RT-N66U router with the latest Merlin firmware
- PC has Intel I218-V ethernet port with the latest Intel (19.5) drivers and Window's throttling off (http://www.thewindowsclub.com/configure-reservable-bandwidth-settings-windows)
- Jumbo frames are disabled on every device
- Transfering file was around 30GB video file from NAS to PC

I've disabled everything related to disk parking/disk power saving in mb's BIOS.


My question is: Is there something I should definitely change or consider within my configuration? LACP might be an overkill but I don't think that it should make anything worst at least. I know there is no magic to 'double' 1GB ethernet port's performance but I just wanted to make sure that I'm getting the maximum from it. Yes there are tons of articles on Google about FreeNAS otpimizations but most of them are pretty old and mostly written for non-home usage.


Thanks in advance!

 

[Joshua Parker Ruehlig]

What are you trying to do here? Get more than 83MB/s over the network?

• Theoretically you could get another ~25% to ~100MB/s with that network setup.

Did you test local..

• local reading/writing on FreeNAS?
• local reading/writing on your PC?
• iperf between your FreeNAS and PC?

 

[Apollo]

What happens when you write to Freenas, do you reach higher speed?
Can you transfer only about 4GB files, but repeat the process a few times in a row. The following transfers should have a throughput increase because it should be coming from cache.
Can you also experiment with ftp transfer with Filezilla or similar and run multithreaded.
These tests should allow you to see some pattern that would help you understand where the performance issue is coming from.

 

[escalibur]

What are you trying to do here? Get more than 83MB/s over the network?

• Theoretically you could get another ~25% to ~100MB/s with that network setup.

I just want to make sure that there is nothing wrong with my configuration. Surely if I can squeeze that another 20MB/s of bandwith then why not.

What happens when you write to Freenas, do you reach higher speed?
Can you transfer only about 4GB files, but repeat the process a few times in a row. The following transfers should have a throughput increase because it should be coming from cache.
Can you also experiment with ftp transfer with Filezilla or similar and run multithreaded.
These tests should allow you to see some pattern that would help you understand where the performance issue is coming from.

I need to run proper benchs soon and I will update my first post with the results. Regardless of the results my main goal is to maximize CIFS performance by best practice (without doing any 'risky tweaks').

 

[ABR]

If you're using only two NICs, you don't need to use LACP.
When you look at interfaces speed, you can see your network load does not divided between nics. Dealing with lacp can only help you in case of using more then one network segment for NAS clients and use L3 Switch for in-switch traffic routing.
In case of only one network segment, using LACP will give you worst network througoutput in favor of network fail-safety, but not more.

I recomend you use two separate IP addresses with two NICs, and make "manual distribution of network load" ;-). For example, some clients will use first NIC for access to NAS, and the second part of clients will use the second NIC. It looks worse, but work better.
I was unable to make network throughoutput increased by using LACP (compared with one NIC). That is why I prefer to use a 10-Gb NIC and 10Gbase-t switch, which is not so expensive now.

 

[ABR]

And... You've mentioned you're using WD Red HDD.
Try to disable TLER (ZFS dislike TLER) and disable WD Idle feature.
For disabling TLER you can use wdtler bootable image (14 Mb) ubcdfixes-tler-f3spinpoint.rar
For tweaking WD Idle parameter, you can use wd_idle bootable image (14 Mb) ubcd511custom.iso.zip
Sorry for my broken English

 

[Joshua Parker Ruehlig]

If you're using only two NICs, you don't need to use LACP.
When you look at interfaces speed, you can see your network load does not divided between nics. Dealing with lacp can only help you in case of using more then one network segment for NAS clients and use L3 Switch for in-switch traffic routing.
In case of only one network segment, using LACP will give you worst network througoutput in favor of network fail-safety, but not more.

I recomend you use two separate IP addresses with two NICs, and make "manual distribution of network load" ;-). For example, some clients will use first NIC for access to NAS, and the second part of clients will use the second NIC. It looks worse, but work better.
I was unable to make network throughoutput increased by using LACP (compared with one NIC). That is why I prefer to use a 10-Gb NIC and 10Gbase-t switch, which is not so expensive now.

If his switch supports LACP he should use that. It will allow 2gbs in and out (from multiple other clients or from another LACP client).

####

If you are using an unmanaged switch and your freenas will send more than receive you should use 'loadbalance'. If your freenas will receive more than send use multiple IPs and configure clients to use different IPs.
These were the results of my testing...

Multiple IPs same subnet
+ Can be selective in sending traffic to a single port.
+ Can receive theoretically 2gbps if you have an equal amount of traffic from clients
- Only sends traffic out a single NIC (freebsd limitation, it only has a single gateway per subnet) you may be able to get around this by creating routes but I didn't try/complicate my setup
- Can send 1gbps out
- Does not failover, if you disconnect the route for that subnet traffic ceased (including all TCP traffic which needs a 2-way handshake, didn't test with UDP)

Failover
+ Simplier setup in sending traffic, don't need to setup clients to use different IPs
+ Can send out 2gbps, uses header to chose which port to send from
+ failover works, when you disconnect a port freebsd knows to send a signal to the unmanaged switch to update which MAC the IP exists on
- Can only receive 1gbps

RoundRobin
+ Simplier setup in sending traffic, don't need to setup clients to use different IPs
+ Can send out 2gbps, uses algorithm to chose which port to send from.
+ failover works, when you disconnect a port freebsd knows to send a signal to the unmanaged switch to update which MAC the IP exists on
- Can only receive 1gbps
- In my testing this wasn't performing as well as loadbalance. Maybe because of the algorithm overhead my freenas doesn't has a modest CPU?

 

[ABR]

If his switch supports LACP he should use that. It will allow 2gbs in and out (from multiple other clients or from another LACP client).

####

These were the results of my testing...
.....
Failover
+ Simplier setup in sending traffic, don't need to setup clients to use different IPs
+ Can send out 2gbps, uses header to chose which port to send from
+ failover works, when you disconnect a port freebsd knows to send a signal to the unmanaged switch to update which MAC the IP exists on
- Can only receive 1gbps

Hi Joshua,
You wrote you have achieved 2 Gbps speed when sending data from Failover ports.
It is a very interesting for me, which setup did you use during this test.
What was the Switch model you have used during this test?
Also, you've written about LACP allows 2Gbps in and out from another LACP client. Is this setup have any hardware dependencies, or it can be realized using ordinary hardware like HP 1810 managed switch and two FreeNAS boxes with a pair of Realtek 8111 NICs in each?
I'm unable to implement such setup which would allow 2 Gbps throughput in single direction with hardware I mention above.

 

[Joshua Parker Ruehlig]

Hi Joshua,
You wrote you have achieved 2 Gbps speed when sending data from Failover ports.
It is a very interesting for me, which setup did you use during this test.
What was Switch you have used during this test?
Also, you've wrote about LACP allows 2Gbps in and out from another LACP client. Is this setup have any hardware dependences, or it can be realized using ordinary hardware like HP 1810 managed switch and two FreeNAS boxes with a pair Realtek 8111 NICs?
I'm unable to implement such setup which would allow 2 Gbps throughput in single direction with hardware I mention.

not sure what you mean when you say "failover ports"?

what do you mean what setup did I use? I tested multiple IPs, loadbalance, and roundrobin. I tested using FreeNAS and multiple other machines either sending or receiving using iperf.

All my testing was on a TEG-S80g.

I can't really answer much about LACP. I know it's supposed to allow NIC bonding to allow theoretically higher then a single link speeds even with just 2 machines, but I've never used it myself, and don't have a switch that supports it. Possibly the Realtek NIC is your issue, but I can't comment cause I don't have any managed switches, or Realtek NICs.

 

[ABR]

Failover ports = ports with Failover configuration. I meant your previous reply.
Thank you for description of your testing setup. As I understood, you have not used any managed switch. It was enough.
But there are some misunderstandings.
You have not performed any tests with LACP, right?
Topic starter have reported about his LACP usage and he have low network performance with his setup.
I've made some tests with LACP and I have reported about low network performance with this setup too.
You have not any experience with LACP.
As you said he should use LACP. I think it is a wrong idea, until we will know about correct LACP setup which will allow to achieve 2 Gbps throughput in both directions.

 

[Joshua Parker Ruehlig]

Failover ports = ports with Failover configuration. I meant your previous reply.
Thank you for description of your testing setup. As I understood, you have not used any managed switch. It was enough.
But there are some misunderstandings.
Why did you say that he should use LACP, even if you have not performed any tests with this setup (and you have not any managed switch)?

I'm just suggesting this because of my (limited) understanding of how LACP works. If his switch supports LACP then it's smart enough to handle multiple in/out and failover. But, feel free to test everything, cause I really only know about unmanaged switches.

 

[zambanini]

please do not post this lacp nonsense. use the forum search function. only failover works.

 

[Joshua Parker Ruehlig]

please do not post this lacp nonsense. use the forum search function. only failover works.

By "failover" are you referencing failover only or the different failover modes? (loadbalance / roundrobin / failover)

And LACP doesn't work if you have a managed switch that supports LACP?

 

[cyberjock]

Ok, I'll keep this somewhat simple because I'm not going to write yet another book about LACP.

1. If you don't have 10 clients, you're going to be woefully disappointed with LACP. Having two 1GB LAN ports on LACP does NOT give you 2Gb of throughput to a client, no matter how much you want it to.
2. For the network shown above, there is absolutely no reason whatsoever to even consider LACP. I have far more machines in my house and I have not nor would I ever recommend LACP.
3. If you've never setup LACP before, you shouldn't be trying to set it up now. LACP is really designed for businesses and lots of hardware claims to support LACP but doesn't. So if you don't have experience in setting up and troubleshooting LACP, put the LACP down.
4. If you know what you are doing, and if your hardware supports it (properly I might add), and if its appropriate for your network, then you can use LACP on FreeNAS (and it does work properly). But if you don't have all those "and"s lined up, you are wasting your time trying to make something work that will only add complexity and literally provide zero benefit and you are wasting the forums time trying to make something work that you probably shouldn't be messing with to begin with. ;)
5. If you know what the spanning tree feature is, you should know better than try to enable it for your network. To me this is an indicator that you've never done LACP (or at least advanced networking before), so I tend to think you shouldn't even be trying to do LACP...

If you are looking to break that 100MB/sec barrier, I'd recommend you go with a 4 disk RAIDZ1 (yes, this will require you to destroy your pool and create it with the additional disk).

Keep in mind that your limiting factor may be your workstation and not your server. Unless you are running your Win8 workstation from an SSD, you're probably limited by the workstation's hard drive.

 

[zambanini]

with a single ip session and freebsd you can not get more bandwith then your single nic. it is not a trunk.

edit:
ok big cyberJ replied, so mine is a shadow.

 

[Joshua Parker Ruehlig]

@cyberjock thanks for the explanation, so..
LACP - managed switch - LACP
doesn't necessarily give you the 2gbps I thought.

@zambanini
I was getting more than 1 NIC's worth of traffic in my iperf testing, using 'loadbalance' mode, with multiple clients downloading from FreeNAS.
This was only with FreeNAS as sending side. It could only receive nearly 1gbps though.
I'm not sure not sure if this conflicts with you're statement though, lol.

 

[cyberjock]

@cyberjock thanks for the explanation, so..
LACP - managed switch - LACP
doesn't necessarily give you the 2gbps I thought.

@zambanini
I was getting more than 1 NIC's worth of traffic in my iperf testing, using 'loadbalance' mode, with multiple clients downloading from FreeNAS.
This was only with FreeNAS as sending side. It could only receive nearly 1gbps though.
I'm not sure not sure if this conflicts with you're statement though, lol.

It may, but probably not. You didn't give enough information.

In short, a single session cannot exceed the slowest single connection (in this case it should be 1Gb/sec). Using iperf you usually do more than 1 connection, so you could, in theory, see more than 1Gb/sec. But multiple connections are NOT how every file sharing protocol works, so you're basically testing an environment that doesn't reflect real world at all.

Also depending on how the LACP is broken down (this depends on your networking hardware and is not user-configurable) it is possible to do 100 connections from a single machine to a single server and still be limited to 1Gb/sec.

In short, LACP is complicated and is "not the droid you are looking for" if you are a home user. If you want more than 1Gb/sec throughput at home, you need to buy 10Gb. I posted a link yesterday of what 10Gb hardware I bought. Connecting a 10Gb LAN between my server and my main desktop would cost you around $175 total for the project. So anyone that wants to argue that they gotta do LACP for more speed at home not only fails to recognize the technology doesn't do what they think it does, they are wasting quite a few people's time with their thread, and to get the boost they want they don't need to spend a lot of money.

I'd bet for 99% of home users, a single 10Gb link directly between their server and their main workstation is enough to make anyone jizz in their pants and be very happy. Few people will want more than that for their entire home LAN (but they can go with 10Gb switches and such if they want that feature).

 

[NickB]

escalibur, I think the overall setup looks good. I agree that if you don't really have the requirement (either for general testing / learning or for the actual use of), then going with LACP probably isn't necessary. With the setup, you're not saturating the 1 GB connection yet.

I just did a test myself within a Windows 7 virtual machine copying a 30 GB bluray from a drive hosted on a local ESXi datastore to a CIFS share on FreeNAS (separate physical machine with 1 GB connections). It started just over 100mb/sec then settled in at around 92mb/sec (but I do have an NFS mount hosting about 19 vms that are mostly idle). The only differences between the setups is that I'm using RAID 1+0 across 4 WD Red 3 TB drives (not sure specifically of the performance differences between this and what you have) and no compression.

Questions:
1) As cyberjock said, it could be your Windows HDD's write performance. Are your transfer speeds any different when going from the Windows 7 to FreeNAS?

2) Your CPU is more powerful than mine (3.5 compared to 3.1), so I wouldn't expect issues, but what does your CPU usage look like for the smbd process when copying files? From what I understand, Samba is single threaded, so it'll only use one core on your CPU. You can get to it by opening the shell from the GUI and typing "top".

3) In conjunction with #2, you have compression turned on. I have mine disabled right now (was going to do more testing at some point to see the performance hit, which I doubt will be subtle at best). I'd create another dataset without compression and perform the same tests.

4) You didn't mention using encryption for your vdev, so I'm guessing no. But, your CPU supports it anyway, so I wouldn't expect any issues.

5) If you're still not sure what might be causing the problem and want to get that (at least) 10 mb/sec, you could get an Ethernet cross over cable and bypass the switch, going directly from the FreeNAS to the Windows 8 machine. This will eliminate any issues with the switch.

If you can't get any more speed out of it, it might just be that you need more HDDs.

But on a lighter note, if you're looking for improvements, you should get a second switch and integrate. If you lose that switch, your server would be down :)

 

[escalibur]

I forgot to say that I'm using Samsung EVO 840 512GB SSD drive (Asus RANGER VII Z97 motherboard, SATA III + AHCI) so my PC's drive is pretty fast for home usage (I've updated it with trash data issue fix firmware).

LACP was used mostly out of my interest. I'm aware that I cannot get more than 1GB out of a single 1GB port no matter which configurations I try. Yeah it might be useless in home usage but as for now it looks to be working fine by balancing connection between two NICs (not that I would desperately need that neither).

10GB is tempting but 10GB network cards cost a lot + in that case I could connect only a single host to my FreeNAS before going bankrupt by buying 10GB switch etc. :)

Spanning tree is enabled because I have two switches in the same network so is there something I'm missing here why shouldn't I've been using it as it is now?

4th disk could definitely 'do the trick' but there is tons of work to do before I can install it because I'm not backing up everything what's on my FreeNAS' disks.

 

[shahzaib]

Please check if the following kernel value is set to 0, if not, make it and LACP load will be distributed between both interfaces with 2Gbps inward/outward, no matter what number of sessions you use :

sysctl -a |grep net.link.lagg.0.use_flowid

 

[jgreco]

No, that just makes the system pick a binding per flow, instead of hashing based on the IP header. Since NAS protocols tend to establish long-lived TCP sessions to transfer data, this doesn't actually do what you suggest. You can cause LACP to use round-robin delivery to accomplish that, but then packets may arrive out of order, which typically hurts performance for each client.

This is all discussed in the link aggregation sticky. https://forums.freenas.org/index.php?threads/lacp-friend-or-foe.30541/

Please don't revive year old threads. Thread closed.