Upgrade recommendations

[Chris Moore]

Quick question: is there any specific reason you went with VyOS? Does it give any advantages over let's say pfSense?
I run a pfSense box as my router atm and were thinking of adding more NICs (10Gb), but would it be a better choice to go for VyOS as a separate switch?

Quite curious to know your take on this.

I used VyOS because it is so very flexible. Take a look at this series of videos about configuring it:
https://www.youtube.com/channel/UCc4gksszq8x6_4pJ7CItMqg/videos

I used a pfSense system to be my firewall / router for a while and I used the VyOS system to be a network switch. The original video series that got me interested in VyOS talked about using pfSense, but they tried it and found VyOS provided more bandwidth. I didn't do that much testing.

I enjoyed working with pfSense and VyOS, but I have moved away from both of those solutions for this Aruba switch and a Netgear Nighthawk X4S R7800 for my router, because they draw much less power and produces much less heat.

 

[Elliot Dierksen]

Quick question: is there any specific reason you went with VyOS? Does it give any advantages over let's say pfSense?

I know Chris replied as well, but I wanted to throw in something. This is another one of my soapbox rants, so apologies if it goes off the rails...

[ RANT ]
Firewalls DO NOT make good inter-vlan routers unless you really have some security policy restrictions you wish to apply. The VyOS device that @Chris Moore is using is a router. It all comes down to a simple difference in philosophy for routers versus firewalls. A router WILL forward your packet unless you stop it. A firewall WON'T forward your packet unless you make it.
[ /RANT ]

 

[melloa]

This is another one of my soapbox rants

Love it :)

@Chris Moore - N00b way to get that S2500 working ... It has a builtin function for Quick Setup that enables a DHCP server on 172.16.0.0/24. The switch IP will be 172.16.0.254. Just connected a laptop to it with DHCP enabled and used a browser to access the switch, changed the IP to match my network, changed default passwords, upgraded firrmware using the GUI, etc. Had to ssh to it to remove the ports from the trunk group. Didn't find it how to do from the GUI (didn't want to waste time also).

 

[Chris Moore]

Love it :)

@Chris Moore - N00b way to get that S2500 working ... It has a builtin function for Quick Setup that enables a DHCP server on 172.16.0.0/24. The switch IP will be 172.16.0.254. Just connected a laptop to it with DHCP enabled and used a browser to access the switch, changed the IP to match my network, changed default passwords, upgraded firrmware using the GUI, etc. Had to ssh to it to remove the ports from the trunk group. Didn't find it how to do from the GUI (didn't want to waste time also).

Not a Noob way, that is how it should work, but I couldn't get the Web GUI to load from my switch. Did you do anything special to make that work? What operating system was the computer running? My Windows 10 computer was saying that the certificates were not valid / expired and it wouldn't let me load the page using any of the four browsers I have installed.

 

[melloa]

Did you do anything special to make that work?

Besides selecting the Quick Setup from the menu, it only worked from windows/IE with the firmware it was shipped (can't remember which version) and to get the page to display I had to add the switch IP to the Compatibility View Settings list.

After updated to the latest firmware, I'm not able to get to the GUI using FireFox and Chrome from my ubuntu and windows.

My Windows 10 computer was saying that the certificates were not valid / expired

It continues displaying that warning on my win10/IE, but I'm able to continue to the GUI.

Using ubuntu without any issues:

Note the three ports on the bottom/right connected to my two NASes and my workstation with the Chelsio T310/Finisar transceivers/LC-LC Duplex 50/125 Multimode 10Gb Fiber Patches (My workstation with a 20m one).

I wasn't able to get the port UP with the cisco DAC as well, so to your point they should have a manufacture block. They worked fine connecting two servers using chelsios T310.

Now is playing with ESXi as I can see the T310 in the hardware list, passthrough, assign and use on the NAS VM, but it doesn't appear on the ESXi network adapters list ?! Go figure... VMWare lists, for ESXi 6.5, T520 up.

 

[melloa]

For the guys running ESXi 6.5, that card worked fine:

Code:

root@esxi:~] lspci -v | grep -A1 -i ethernet
0000:05:00.0 Ethernet controller Network controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE , PCIe 2.0 5GT/s] [vmnic4]
	 Class 0200: 15b3:6750

 

[Chris Moore]

Using ubuntu without any issues:

The GUI working on yours, I took another try at it and got mine working also. I was sure that I had tried that before, but I don't recall if I tried it after updating the firmware.

Love it :)

I am enjoying it too, but I already want more 10Gb ports.

 

[melloa]

I already want more 10Gb ports.

I'm getting there ... Will be hard to add another switch for a net increase of three ports :D

 

[digity]

I'm interested in this switch now too for heat and power benefits (I currently have a vyOS + Supermicro X9SCM powered DIY 10 Gbe 6 port switch). So the Arista DACs and FS transceivers work and the Cisco DACs don't work - are there any other known compatible and/or incompatible DACs, transceivers or NICs?

 

[Chris Moore]

I'm interested in this switch now too for heat and power benefits (I currently have a vyOS + Supermicro X9SCM powered DIY 10 Gbe 6 port switch). So the Arista DACs and FS transceivers work and the Cisco DACs don't work - are there any other known compatible and/or incompatible DACs, transceivers or NICs?

I have a set of Intel transceivers that I hope to have time to test this weekend, but for now, what I said above is all that I have tested with. I have used both Mellanox and Chelsio NICs successfully.

 

[digity]

I have a set of Intel transceivers that I hope to have time to test this weekend, but for now, what I said above is all that I have tested with. I have used both Mellanox and Chelsio NICs successfully.

Cool, keep us posted

Sent from my SM-G955U using Tapatalk

 

[digity]

I have a set of Intel transceivers that I hope to have time to test this weekend, but for now, what I said above is all that I have tested with. I have used both Mellanox and Chelsio NICs successfully.

Any luck with the Intel transceivers and this switch?

 

[Chris Moore]

Any luck with the Intel transceivers and this switch?

No luck with the Intel Transceivers on the switch end of the link, but I have it working with an Intel transceiver in the Chelsio NIC in the NAS and a Fiberstore (fs.com) branded transceiver in the switch. Those Intel transceivers work fine with Cisco gear at work, but the Cisco DAC and the Intel transceivers are not working in the Aruba switch. So I know they work in general, which means to me that it is some kind of brand lockout.
One of the ones I have that is working is this one: https://www.fs.com/products/36982.html

 

[melloa]

I am enjoying it too, but I already want more 10Gb ports.

Just found a scratched S2500 on eBay for $50/free s&h, so why not? I do have a second Cisco 3750 running that can be replaced with the Aruba and add three 10Gb ports :D

Now, one thing I can't figure why I'm getting so low speeds:

Testing from FreeNAS ESXi 6.5-U2 VM on a 10GbvSwitch (using Melanox), I get:

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.200 port 45906
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  1.73 GBytes  1.49 Gbits/sec

Testing from a FreeNAS on baremetal with Chelsio T320, I get:

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.220 port 47580
[  4]  0.0-10.0 sec  1.72 GBytes  1.48 Gbits/sec

From my Plex ESXi 6.5-U2 VM on a 10GbvSwitch (using Melanox), I get:

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.203 port 40730
[  4]  0.0-10.0 sec  1.73 GBytes  1.48 Gbits/sec

I'm using autotune, so assume most of the required setting are loaded; tests with mtu set to 9000 didn't improve a lot:

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.200 port 50862
[  4]  0.0-10.0 sec  1.74 GBytes  1.50 Gbits/sec

Are are your iperf tests reporting?

 

[Chris Moore]

I'm using autotune

I didn't find autotune helpful and I ultimately removed all the settings that it created.

 

[melloa]

I didn't find autotune helpful and I ultimately removed all the settings that it created.

No changes:

- Fresh FreeNAS 11.1 U6
- No turnables
- mtu 9000
- T320
- Aruba S2500

Code:

iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.60 port 5001 connected with 10.10.10.152 port 25693
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  1.72 GBytes  1.48 Gbits/sec

Code:

root@freenas:~ # ifconfig
cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:07:43:07:cb:17
	hwaddr 00:07:43:07:cb:17
	inet 10.10.10.152 netmask 0xffffff00 broadcast 10.10.10.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet 10Gbase-SR <full-duplex>
	status: active

Not sure what I'm doing wrong.

 

[Johnnie Black]

In my experience default tcp settings don't really take advantage of 10GbE, auto tune didn't work for me, it made things worse, these tunables worked, they are not mine, found them on this forum somewhere.

 

[melloa]

In my experience default tcp settings don't really take advantage of 10GbE, auto tune didn't work for me, it made things worse, these tunables worked, they are not mine, found them on this forum somewhere.

Will check against what was already added by autotune. I do see several settings already there:

 

[Elliot Dierksen]

I just ran iperf between my two FreeNAS units, and got pretty darn good results.

Code:

root@freenas2:/tmp # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  512 KByte (default)
------------------------------------------------------------
[  4] local 192.168.252.27 port 5001 connected with 192.168.252.23 port 18841
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  11.0 GBytes  9.41 Gbits/sec

Code:

root@freenas:/nonexistent # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.252.23 port 5001 connected with 192.168.252.27 port 22038
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  11.0 GBytes  9.41 Gbits/sec

That is using just what was done via autotune.

Edit: forgot to include what autotune has set.
Main system (freenas2):

Code:

net.inet.tcp.recvbuf_inc: 16384 -> 524288
net.inet.tcp.recvspace: 65536 -> 524288
net.inet.tcp.mssdflt: 536 -> 1448
vfs.zfs.arc_max: 132817707008 -> 123624000000
vfs.zfs.l2arc_norw: 1 -> 0
vfs.zfs.l2arc_write_max: 8388608 -> 10000000
kern.ipc.nmbclusters: 8383856
sysctl: kern.ipc.nmbclusters=8383850: Invalid argument
net.inet.tcp.recvbuf_max: 2097152 -> 16777216
net.inet.tcp.sendbuf_max: 2097152 -> 16777216
net.inet.tcp.delayed_ack: 1 -> 0
vfs.zfs.metaslab.lba_weighting_enabled: 1 -> 1
vfs.zfs.l2arc_noprefetch: 1 -> 0
net.inet.tcp.sendspace: 32768 -> 524288
vfs.zfs.l2arc_headroom: 2 -> 2
vfs.zfs.zfetch.max_distance: 8388608 -> 33554432
net.inet.tcp.sendbuf_inc: 8192 -> 16384
kern.ipc.maxsockbuf: 2097152 -> 8388608
vfs.zfs.l2arc_write_boost: 8388608 -> 40000000

Backup system (freenas):

Code:

net.inet.tcp.delayed_ack: 1 -> 0
net.inet.tcp.recvspace: 65536 -> 262144
vfs.zfs.l2arc_write_boost: 8388608 -> 40000000
net.inet.tcp.recvbuf_max: 2097152 -> 16777216
net.inet.tcp.recvbuf_inc: 16384 -> 524288
kern.ipc.maxsockbuf: 2097152 -> 4194304
vfs.zfs.zfetch.max_distance: 8388608 -> 33554432
net.inet.tcp.sendbuf_max: 2097152 -> 16777216
vfs.zfs.arc_max: 65618808832 -> 61605000000
net.inet.tcp.mssdflt: 536 -> 1448
kern.ipc.nmbclusters: 4178128 -> 4178128
net.inet.tcp.sendbuf_inc: 8192 -> 16384
vfs.zfs.l2arc_write_max: 8388608 -> 10000000
vfs.zfs.l2arc_headroom: 2 -> 2
vfs.zfs.l2arc_norw: 1 -> 0
vfs.zfs.l2arc_noprefetch: 1 -> 0
net.inet.tcp.sendspace: 32768 -> 262144
vfs.zfs.metaslab.lba_weighting_enabled: 1 -> 1

I know, really creative system names....

 

[melloa]

I just ran iperf between my two FreeNAS units, and got pretty darn good results.

I'd expect to see something on that neighborhood, for sure.

The two servers I'm testing are:

1 - (BackUpNAS * yes we do have creative system names *) X9DRi-LN4F+ with E5-2630 V2/192Gb ECC, using a T320 with Chelsio transceiver.
2 - (TestNAS) X8STE/i7/32Gb, also using T320 with Chelsio transceiver.

They are connected to a Aruba S2500.

I'm fresh installing the test box to remove any setting changes and will compare with your autotune above, repeat the test via the switch, and direct connected.

Will report back.