10 Gig Networking Primer

10 Gig Networking Primer

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
not having anything mission critical .. I can "afford" to play with free/super cheap stuff to live and learn with. I'll give these parts I'm getting a go and if they don't work out .. I'll wait for other free/super cheap stuff to grab and try that out. Billions of problem-free aggregate run-hours have to start somewhere I suppose.

I do have access to a few magic fix-it wands though if need-be :cool:

I appreciate your guys' knowledge and experience and time here on the forums... warnings/advice duly noted

..as a kid I had smaller Honda's, Suzuki's, Kawasaki dirt bikes that I mixed and matched parts with to get 1 or 2 functioning dirt bikes .. they weren't ideal, but they worked good and I had fun doing it. I didn't cry foul and blame Honda when a Suzuki part wouldn't mesh with it.. If I can get better than 110mb/s transfer speeds with un-recommended components, then I'll be happy and will have had fun setting it up and won't expect any help setting up stuff I shouldn't be setting up to begin with if it doesn't work out.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
not having anything mission critical .. I can "afford" to play with free/super cheap stuff to live and learn with. I'll give these parts I'm getting a go and if they don't work out .. I'll wait for other free/super cheap stuff to grab and try that out. Billions of problem-free aggregate run-hours have to start somewhere I suppose.

I do have access to a few magic fix-it wands though if need-be :cool:

I appreciate your guys' knowledge and experience and time here on the forums... warnings/advice duly noted

..as a kid I had smaller Honda's, Suzuki's, Kawasaki dirt bikes that I mixed and matched parts with to get 1 or 2 functioning dirt bikes .. they weren't ideal, but they worked good and I had fun doing it. I didn't cry foul and blame Honda when a Suzuki part wouldn't mesh with it.. If I can get better than 110mb/s transfer speeds with un-recommended components, then I'll be happy and will have had fun setting it up and won't expect any help setting up stuff I shouldn't be setting up to begin with if it doesn't work out.

So you're good to go then. Mainly it's very frustrating having people come to the forum with sour complaints when their system doesn't meet up to their unrealistic expectations after they spent good money following some random poorly informed Interwebz blog post. If you are happy to crank out your own "I own this mess" certificate, I heartily endorse that. Choice is good. Crazy choice might not be good. But well-informed choice is often the beginning of learning something new.
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
oh... I have a slew of those particular certificates
 

Brezlord

Contributor
Joined
Jan 7, 2017
Messages
189
I would like to confirm that the Solarflare SFN5122F works in FreeNAS 11.3-MASTER-201907310626 . Just plug and play no need to manually load drivers. All of the usual stuff works LAG, vLAN, etc.

SFN5122F.JPG
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I would like to confirm that the Solarflare SFN5122F works in FreeNAS 11.2. Just plug and play no need to manually load drivers. All of the usual stuff works LAG, vLAN, etc.
Are you saying that you have used it successfully?
 

Brezlord

Contributor
Joined
Jan 7, 2017
Messages
189
I had a typo that I've fixed in my previous post. I'm testing on 11.3-MASTER-201907310626 not 11.2.

I have so far tested a 56GB SMB transfer back and forth with no issues. I'm testing on an IBM x3650 M3 2 x Intel(R) Xeon(R) CPU X5670 @ 2.93GHz 32GB ram 4 x 72GB 10K SAS raid 0 on IBM M5015 raid controller (Machine only used for testing so no HBA) LAG with vLAN.

I have not had any issues with the card. If there are specific test you would like run just let me know.

tests.JPG
 

virusbcn

Explorer
Joined
Apr 22, 2016
Messages
92
I am thinking try to test my first 10 Gig LAN, i think buy some X520 Intel nics but what switch can i buy, a SOHO switch for home, i need a cheap switch, what switch recommends me ???
 

c77dk

Patron
Joined
Nov 27, 2019
Messages
467
I am thinking try to test my first 10 Gig LAN, i think buy some X520 Intel nics but what switch can i buy, a SOHO switch for home, i need a cheap switch, what switch recommends me ???

I'm considering a MikroTik CRS305-1G-4S+IN myself - nice size for a home switch - and fanless :smile:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'm considering a MikroTik CRS305-1G-4S+IN myself - nice size for a home switch - and fanless :)

I asked MikroTik to send me its 8 port big brother and I'm incredibly impressed at the value. It's absolutely not the enterprise grade five-figure switchgear I normally use, but it's definitely a big step up compared to the normal Netgear etc. stuff, and, holy mackerel, the price. Can you go wrong? I think you can't.
 

marcus8699

Dabbler
Joined
May 4, 2020
Messages
17
Hi All,

After continued tries to get 10Gb speeds or close to that I have come to the point that I think that I am getting what is expected of my hardware. I wanted to make a post within here to see if there are any suggestions to get better results. Below is a list of hardware that I have:

MB: Asus Rampage IV Formula
CPU: Intel Xeon 24.GHz 12C/24T
ESXi FreeNAS VM: 10vCPUs
Memory: 32GB total system Mem
FreeNAS VM: 17GB FreeNAS VM
4xWD 3TB RAIDz array
Cisco Catalyst 9200L 24T-4X
Cisco SFP-10G-SR-S
Twinex 10Gb cable (not sure on the make. Looking to replace with another Cisco SR)
Dell Chelsio T520-CR (vmxnet3)
SAS9211-8i (Passthrough to FreeNAS)

Test Machine:
ESXi Windows Server 2019 VM
CPU 8vCPU
NIC Chelsio (vmxnet3)
Source File: Fusion IO Drive in RAID0 2000MB/s read/write seq
5GB Memory

+ Jumbo Frames have been enabled.
+ No Tuneables have been added
+ FreeNAS is in VLAN 20 and Windows Server is in VLAN 30. Default gateway is the switch so the traffic would need to go up and come back down on the secondary port on the Chelsio NIC
+ Crystal Disk Benchmark gives me a 982MB/s Seq Read with a 573MB/s write (write is my concern)
+ Screenshots are attached of both iPerf testing and Crystaldisk benchmark

My first thoughts are I think there may be some tuning I can achieve by changing either some settings on the FreeNAS with tuneables or via Windows machine. However, googling around and looking at multiple threads it seems these turntables are more recommended on a machine by machine basis so I decided to post before making any adjustments. Question is, with the current setup are these the best case scenario speeds that I can expect to achieve with the current hardware or is their potential to increase the write speed that is suffering so much behind the read?

Thanks in advance for the help and please let me know if I need to upload anything else.
 

Attachments

  • Current.PNG
    Current.PNG
    52.7 KB · Views: 715
  • FreeNAS_client_iperf.PNG
    FreeNAS_client_iperf.PNG
    24.1 KB · Views: 630
  • FreeNAS_server_iperf.PNG
    FreeNAS_server_iperf.PNG
    21.8 KB · Views: 616

marcus8699

Dabbler
Joined
May 4, 2020
Messages
17
Also, forgot to mention my use case for FreeNAS. Its a streaming server with large video files. I would like to be able to transfer these files to the server as quickly as possible.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
4xWD 3TB RAIDz array

You are hitting the limit of your drives. You are talking sustained writes here, large video files. If you want faster writes, given that you started with one raidz1 vdev, you are using SMB and you are writing large files, add a second raidz1 vdev to the pool, test performance again. Beware of DM-SMR in the current crop of small (2-6TB) WD Red drives.

You are getting almost 600MB/s write, I am assuming the pool is currently entirely empty. That, too, will slow down as the pool fills.
 

marcus8699

Dabbler
Joined
May 4, 2020
Messages
17
You are hitting the limit of your drives. You are talking sustained writes here, large video files. If you want faster writes, given that you started with one raidz1 vdev, you are using SMB and you are writing large files, add a second raidz1 vdev to the pool, test performance again. Beware of DM-SMR in the current crop of small (2-6TB) WD Red drives.

You are getting almost 600MB/s write, I am assuming the pool is currently entirely empty. That, too, will slow down as the pool fills.

Hi thanks for the response. I figured I was hitting some type of bottleneck and the drives were the first thing I thought of but wanted to make sure there wasn't some tweaking that needed to be done.

I am definitely a noob when it comes to FreeNAS and storage in general. Being a TAC engineer for Cisco, networking is my bread and butter. So forgive me for any silly questions. My first question is about the separate vDevs. I know that when you create another vDev in the pool it's two separate physical arrays but I assume they are one logical array? With this being said, I would then need to match the 4x3TB and could not add say 4x6TB instead to increase capacity?

Second question, what exactly is DM-SMR (Asking this question pre-google/forum search) ? Do you have any documentation that I can read up on it?

Many thanks for the suggestions and answers.

Marcus
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Oh hi fellow Cisco person. I work as an SA for GES NE :).
There is an intro to ZFS in resources that explains how vdevs work. ZFS will distribute writes through the pool, redundancy is on a per vdev level.
You can absolutely do 4x6TB, nothing wrong with that.
For DM-SMR, check resources again. In a nutshell, it kills sustained write speeds.
 

marcus8699

Dabbler
Joined
May 4, 2020
Messages
17
Oh hi fellow Cisco person. I work as an SA for GES NE :).
There is an intro to ZFS in resources that explains how vdevs work. ZFS will distribute writes through the pool, redundancy is on a per vdev level.
You can absolutely do 4x6TB, nothing wrong with that.
For DM-SMR, check resources again. In a nutshell, it kills sustained write speeds.

Small world haha. Maybe I'll ping ya on Teams to help me with this FreeNAS build haha.

I'll give the resources tab a look and post back if I have any questions. Thanks alot!
 

marcus8699

Dabbler
Joined
May 4, 2020
Messages
17
Oh hi fellow Cisco person. I work as an SA for GES NE :).
There is an intro to ZFS in resources that explains how vdevs work. ZFS will distribute writes through the pool, redundancy is on a per vdev level.
You can absolutely do 4x6TB, nothing wrong with that.
For DM-SMR, check resources again. In a nutshell, it kills sustained write speeds.

So I read the following guide: ZFS Pool Performance and I see now the differences in the pools and the limitations you get with each one. My question goes back to what I asked about creating a separate vDev with 4x 6TB drives even though my first vDev is 4x 3TB. In the guide I see the following,

"Using identical vdevs is strongly recommended, but it is possible to mismatch vdevs in a pool. For example, you could configure an 11-wide Z3 vdev and add a single striped vdev as the 12th drive in the pool. This would not be smart. Your extremely fault-tolerant Z3 vdev now depends on that single 12th drive to maintain your data. If that drive goes, your whole pool is gone."

I know this is simply saying that if the drive fails in the 2nd vDev the whole pool fails but are those same recommendations of them being Identical, do they apply to the storage size of the vDev as well?

This is probably now a topic for another forum and I don't want to start clogging this one up with unrelated topics. Feel free to PM me if that would work much better.

Thanks,

Marcus
 

warllo

Contributor
Joined
Nov 22, 2012
Messages
117
I just thought I would drop by and say I was successful in getting a Solarflare SFN5122F working well with FreeNAS-11.3-U3.2. I tested with a DAC+ connection and the Finisar FTLX8571D3BCL. Everything within FreeNAS was plug and play. I did have to change a PCIE OPROM setting in my BIOS to get the machine to boot. I hope this info may help someone looking for 10GB networking for possible home use. If this were business use I would probably get a Chelsio card since they are officially supported but for my purposes, the Solarflare works great. The total cost was $40 for the SFN5122F and two of the Finisar FTLX8571D3BCL.
 

msbxa

Contributor
Joined
Sep 21, 2014
Messages
151
Can you specify the motherboard type and what have you change from PCIE OPROM setting in BIOS to get the machine to boot?
 

warllo

Contributor
Joined
Nov 22, 2012
Messages
117
The motherboard was a rather old Supermicro X8SIl-F and I had to set the OPROM disabled for the PCIE slot that Solar Flare Network Card was installed on. I found the setting in the BIOS under PCIE/PnP configuration. Hope this helps.
 

msbxa

Contributor
Joined
Sep 21, 2014
Messages
151
Just received 2x of my Solarflare SFN5122F to replace my old 10G card Chelsio cc2-s320e-sron SFP+ on Supermicro X10SL7-F ( TrueNAS Core 12 Beta) and X9SRL-F ( FreeNAS 11.3-U4). I can confirm they all work well without any loading drivers or anything just plug and play. Solarflare gives you almost full speed while Chelsio gets you half the speed.
 
Top