Performance overview - current setup

[PlowHouse]

Hey all,

Just looking to get some insight from others on the performance I'm currently getting. Created some CIFS and NFS shares and I'm hovering around 80 MB/s when transferring from my Windows system to my newly created FreeNAS box configured for RAID 10 over a gigabit network. The specs below are my current setup:

FreeNAS 9.3.1
CPU: Intel i3-4170 3.7 GHz Haswell Dual-Core (Hyper-Threading off)
RAM: 32 GB DDR3 1600 (PC3 12800) ECC
MB: ASRock Rack C226M WS
PSU: Cooler Master V550 (550W)
HDD: 6 x 2TB WD2002FAEX 7200 RPM 64 MB Cache (RAID 10)
Case: Fractal Design Define Mini - Micro ATX Silent PC

Other notes:
- I'm currently utilizing the 6 SATA III ports on the motherboard rather than a separate HBA
- Compression is set to lz4
- Deduplication is OFF
- Drives are set at 4K sectors (ashift=12)
- As mentioned above, HT on the processor is turned OFF
- I don't have a dedicated SSD connected for the L2ARC or as a SLOG for ZIL as I don't have the recommended RAM (64GB) for L2ARC and pointing the ZIL to an SSD would have no benefit to me as I primarily will be using CIFS for this build
- Autotune is OFF.

For home use 80 MB/s is plenty, but if there's some tweaks I can make to hover around 100 MB/s or more then I'd obviously like to make those changes. I've seen plenty of mention towards buying the IBM M1015 card off eBay which may increase my throughput but I'd like to test with the current setup I have first and see what I can squeeze out of it. Also, if anyone could provide me with instructions using iostat or bonnie++ on FreeBSD I'd greatly appreciate it. I tried installing Bonnie++ within an SSH session on my FreeBSD instance while logged in as root but I couldn't seem to find a wget address or pkg to install from. Most of my Linux/UNIX experience comes from Debian based operating systems such as Ubuntu. So things within FreeBSD look to be a bit different than what I'm accustomed to. Appreciate any insight on the build as well as any benchmark help.

Thanks

 

[BigDave]

As far as performance goes, your entire network setup will have an effect on your
transfer speeds. The quality of your twisted pair cables, switches, NICs, pool config
and your drives (hard or otherwise) will have an effect on performance.
FreeNAS has the iperf utility built in and is used by most to measure performance
between WS/Desktop and Server. The manual has an overview of the utilities they
included with FreeNAS. I just tested my stuff yesterday with iperf. It took me awhile
to get the hang of it, but I'm not an IT professional.;)

 

[solarisguy]

[...] hovering around 80 MB/s when transferring from my Windows system to my newly created FreeNAS box configured for RAID 10 over a gigabit network. [...]

How did you create your pool? In the GUI?
Can 80 MB/s be your Windows system limitation? At what type of the load?
What is the speed when reading (copying) from your FreeNAS?
BigDave has suggested iperf. What are the iperf results? Iperf results would be just for the network. To get a speed from/to FreeNAS disks over the network, but exclude samba, you can use FTP.

 

[DrKK]

nononoo.

If, like 92% of the universe, his client machine/Windows box has a Realtek NIC chipset, then 600-700Mbps (otherwise known as 80MB/s) is perfectly normal/maximal performance, on account of the well-known shittiness of that chipset.

 

[BigDave]

nononoo.

If, like 92% of the universe, his client machine/Windows box has a Realtek NIC chipset, then 600-700Mbps (otherwise known as 80MB/s) is perfectly normal/maximal performance, on account of the well-known shittiness of that chipset.

Woohoo! I'm an Eight Percent (er) :p

 

[Bidule0hm]

Woohoo! I'm an Eight Percent (er) :p

Me too, but mine is even worse than a Realtek (if that's possible...)

 

[joeschmuck]

@PlowHouse
You don't specify how you connected your system nor the files sizes being transferred which both have a serious effect on the throughput, in addition to the RealTek NIC if you have one of those, however I have a RealTek NIC in my Windoze computer and can still transfer to my NAS and saturate at 100MB/sec (not as high as some folks have reported in the past such as 110MB/sec with Intel NICs) the connection, but different hardware will perform differently.

SO, make a direct Ethernet connection from your computer to your NAS. Transfer a single large file and see what the speed is. Files which are just barely larger than your NAS RAM should transfer very fast and then once that RAM fills up the next bottleneck is the hard drive. I guess you have a mirrored vdev since you said RAID10 but I really am not certain what you have established.

Also, turn on Autotune, it may help maximize use of your RAM.

Benchmark testing, well there are different tests for different areas of your system. For example "dd" can be used to find out how fast your internal pool is and well worth conducting (search these forums on how to do it and what the results mean). I like the Intel NAS Performance Toolkit to test my Ethernet throughput because it tests many factors for transferring data but if you don't have a very good Windows computer to run it on then you are doomed to use it as a comparative tool, unless you only compare it to your own tests like seeing if you made any changes for the better, including network hardware changes like maybe a network switch change or Ethernet cable.

As you said, 80MB/sec isn't bad for a home system.

 

[PlowHouse]

Thanks for the replies everyone.

@joeschmuck - The system I'm using to test throughput to the NAS is my gaming rig which is using a Realtek 8110SC NIC (ASUS Sabortooth x58 Motherboard). The files being transferred were various .ISO files along with a separate transfer including music, pictures, and videos equating past 80 GB's in data. The RAID on the NAS is indeed RAID10 (Mirrored vdev) and it is setup correctly. I did turn "Autotune" on breifly and noticed it created a turnable for an L2ARC which I didn't want. However, I left the autotune on, rebooted, and tried transferring the same data to a test folder within the CIFS share. There was no difference in throughput at 80MB/s was still the capping number. Once this completed I turned Autotune back off. Also, I did read up on "dd" as a benchmark test but couldn't actually get dd to run correctly within FreeBSD. I think it complained about missing packages within the dd command but I'll verify this soon enough and figure out where I went wrong.

I think I'll try another system in my household that has Intel NIC's or I can install an Intel PCI NIC card in my gaming rig as I have a few server grade cards lying around. Seems like Realtek is a common cause of performance issues within these threads so it's worth a shot trying.

@solarisguy - The pool was created within the GUI of FreeNAS rather than through the CLI. I'm comfortable with either setup but the GUI was just what I went with at that time. I will test the other options you mention this week.

@DrKK - Since the rig I'm testing with consists of a Realtek 8110SC chipset, I'm guessing I fall in that 92% category =D. I will try adding an Intel NIC card that I have lying around and see if my speeds change.

I'll try reporting back my results during the week, Really appreciate all the insight and help on this. If anything else comes to mind that I should try feel free to recommend it and I'll see what I can get done.

 

[DrKK]

worse than a Realtek

what's worse than a RealTek? A telephone made out of string and coconut shells?

 

[Bidule0hm]

Broadcom...

 

[DrKK]

Realtek 8110 and 8111 are both pieces of Doo Doo.

I don't know why you're upset about this performance. Should be more or less adequate for a Gbps LAN.

 

[anodos]

what's worse than a RealTek? A telephone made out of string and coconut shells?

Honestly, it depends on the coconuts.

 

[PlowHouse]

Tossed in an Intel PRO 1000 PR dual NIC card, now getting 112mb/s. If I had a managed switch I'd look into LACP and see if there'd be a gain from that but for home use, I'm happy with 112mb/s. Thanks all for the help on this.

 

[DrKK]

Tossed in an Intel PRO 1000 PR dual NIC card, now getting 112mb/s. If I had a managed switch I'd look into LACP and see if there'd be a gain from that but for home use, I'm happy with 112mb/s. Thanks all for the help on this.

There you have it folks. This man nearly doubled his throughput by getting away from RealTek ;)

Also, LACP, for 99% of home FreeNAS users, won't actually do anything at all. People think it magically doubles their bandwidth, and generally speaking, nothing like that happens at all.

 

[PlowHouse]

There you have it folks. This man nearly doubled his throughput by getting away from RealTek ;)

Also, LACP, for 99% of home FreeNAS users, won't actually do anything at all. People think it magically doubles their bandwidth, and generally speaking, nothing like that happens at all.

Yea it would be complete overkill to setup LACP, I wouldn't notice the difference. But on the bright side I've seen it first hand how a crappy realtek NIC can hamper performance...

 

[DrKK]

Out of curiosity, how much is a Realtek 8111E chipset, for example, compared to a similar Intel NIC chipset for a motherboard? I can't imagine it's more than a $5 difference. I guess margins are razor thin.

 

[joeschmuck]

That is great news that switching to the Intel NIC helped considerably.

 

[JDCynical]

Out of curiosity, how much is a Realtek 8111E chipset, for example, compared to a similar Intel NIC chipset for a motherboard? I can't imagine it's more than a $5 difference. I guess margins are razor thin.

Granted, AliExpress isn't the best place to get the components (may or may not be counterfeit) and there may be more/less expensive Intel chips out there...

The Intel 8245PI is about 1.15 USD each
The Realtek RTL8111E goes for about a dollar USD on the same place

When I'm making a list for a system, if it doesn't have an Intel network chip, I generally don't consider it as an option.

 

[DrKK]

Granted, AliExpress isn't the best place to get the components (may or may not be counterfeit) and there may be more/less expensive Intel chips out there...

The Intel 8245PI is about 1.15 USD each
The Realtek RTL8111E goes for about a dollar USD on the same place

When I'm making a list for a system, if it doesn't have an Intel network chip, I generally don't consider it as an option.

Fascinating. So we can conclude that motherboard manufacturers are going with a KNOWN INFERIOR NIC chipset to save, potentially, less than $1 USD per shipped unit? Are the margins really that razor thin? Maybe they are.

 

[Ericloewe]

Fascinating. So we can conclude that motherboard manufacturers are going with a KNOWN INFERIOR NIC chipset to save, potentially, less than $1 USD per shipped unit? Are the margins really that razor thin? Maybe they are.

Yeah. Intel's list price for i210-ATs is an unacceptably high $3.20. i217/i218/i219-LMs are $1.92 (they rely on the PCH, so only one per system, maximum).

I was seriously surprised when I first saw this a couple of days ago. I never imagined these things were so damned cheap.

Maybe the Realteks are specced to "work" with crummier isolation transformers, which would increase the price difference...