Upgrade recommendations

[melloa]

I bought a couple of these cards to use in the switch build because I wanted the port count:
https://www.ebay.com/itm/Chelsio-10...8-30-2x-10G-SFP-Tranceivers-HIGH/192647386240
On the Windows desktop and on the FreeNAS, I used these:
https://www.ebay.com/itm/2x-Mellano...Card-1x-SFP-10G-gigabit-cable-3M/153053070898

Good deal for the test switch. I also have a quad intel laying around, thinking about getting the chelsion and the quad intel on the "switch" and use the mellanox on the NAS and the desktop. Only issue is how I'm going to connect the desktop to it been 50ft away?

Will check on MBs options that I might have around ... for sure don't want to fire up a 2950 or use my X9SRW-F for that 24x7 :D

 

[Elliot Dierksen]

I would say short range fiber transceivers (aka SX). There are 10G twisted pair options, but you would have to use some kind of transceiver to do that. That would mean you would have to find one that both sides (NIC's on both ends) like, and I think there are some fairly tight distance restrictions on 10G TP. I have lots of customers doing 10G, and 0 (aka zlich point squat) of them are doing TP. Most likely the connectors on the fiber will be LC. That means you would need 20m LC to LC OM3 fiber optic cable. Assuming the NIC's on both ends are Chelsio, you should be able to use these transceivers which are ~=$15 each. You would need two of these.
https://www.ebay.com/itm/Chelsio-26...SFP-Transceiver-FTLX8571D3BCL-CL/142735646110
The 20M (giving you a little bit of wiggle room for 50') fiber patch cord would be ~= $22. You only need one of these.
https://www.ebay.com/itm/20m-LC-LC-...10-Gb-Fiber-Patch-Cable-Aqua-om3/131824995659
For the twinax cables (SFP+ hard wired onto each end), you would need a total of two since each side has an end. These should work and are ~= $10. I am assuming 1M would be enough distance between the servers and the switch. Other lengths are available.
https://www.ebay.com/itm/Cisco-Genu...winax-10GB-Cable-1-year-warranty/222586985134

< Disclaimer > I have no connection with any of these sellers. Just trying to give you a point of reference. < /Disclaimer >

 

[c32767a]

It concerns me that they only list the SFP+ ports as being uplink. Have you tried this? The Aruba switch I bought has 4 SFP+ ports that were intended to be for stacking and I can't (so far) get them to work to connect a computer as an end point but they do link when connected to another switch.

I don't mean to hijack the thread but I have 2 comments.

Many switches of the era of the Aruba 2500 were designed to be aggregation switches with the 10G uplinks passed back into a network core. Sometimes the chipsets omitted features relevant to connecting those ports to end devices (like buffering, flow control, etc). Sometimes that can be a problem.

The other thought is that typically those chipsets also had a default "stacking" mode that changed the interface to a chipset specific ethernet frame type to facilitate stacking. It may be that your switch has those ports set to "Stacking" mode rather than 10G. Typically this configuration was something you had to set in the switch's NVram and reboot to configure.
Probably some variant of "show stacking interface" and "delete stacking interface" would be my guess.

 

[Chris Moore]

It was like that.
I did delete the stacking interface, but I didn't restart the switch. I will give that a shot.
Thanks.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[melloa]

I followed the instructions and built my own switch with less than $100 of parts from eBay.

@Chris Moore , got mine running with two intel quad I had laying around on an i5 for testing. The Chelsios will be here next week. Can't wait! Do you have any ideas for a MB/CPU to support two 10 gb chesios and two intel 1gb quad? I have a supermicro 2U empty that is asking to have a switch inside ;)

< Disclaimer > I have no connection with any of these sellers. Just trying to give you a point of reference. < /Disclaimer >

@Elliot Dierksen , nice disclaimer, but better yet were your suggestions.

Thanks again.

 

[Dent]

Ubiquiti Networks ES-16-XG EdgeSwitch
I keep trying to find reasons not to buy this switch.

I have the unifi version of that switch as well as the us-48 that has 2 sfp+ ports and they work great now. When I first got them there was a bit of firmware issues but seems solid now. Also using a mix of optics all bought off eBay. Not saying it would be good for you, but great for me.

Sent from my SM-T813 using Tapatalk

 

[Chris Moore]

@Chris Moore , got mine running with two intel quad I had laying around on an i5 for testing. The Chelsios will be here next week. Can't wait! Do you have any ideas for a MB/CPU to support two 10 gb chesios and two intel 1gb quad? I have a supermicro 2U empty that is asking to have a switch inside ;)

Interestingly, this is the exact model board I am using for my switch:
https://www.ebay.com/itm/SuperMicro...-4gb-ram-X8STE-LGA-1366-Socket-B/173539822035
The seller has four of them available already setup with processor and RAM. It is tempting because they make very capable systems for running Linux, which is what I was using mine for before I made it into a switch.

 

[melloa]

Interestingly, this is the exact model board I am using for my switch:
https://www.ebay.com/itm/SuperMicro-motherboard-with-i7-960-4gb-ram-X8STE-LGA-1366-Socket-B/173539822035

https://www.ebay.com/itm/SuperMicro...-4gb-ram-X8STE-LGA-1366-Socket-B/173539822035

Awesome!

The seller has four of them available already setup with processor and RAM. It is tempting because they make very capable systems for running Linux, which is what I was using mine for before I made it into a switch.

Now one three ;)

Thanks again!

 

[melloa]

Folks,

I'm playing with my Chelsio NICs using VyOS (Thanks to @Chris Moore for the idea) on a desktop motherboard while I don't get a server one to finish the switch, using the twinax cables suggested by @Elliot Dierksen - Great help guys.

Transfer speed has doubled and iperf test shows 59% increase, but I assume that's due to the hardware been used. With one 10Gb box and two 1Gb boxes connected to the switch, CPU utilization is going to 27% with rsync running between the FreeNAS test server and the VyOS test switch.

Code:

vyos@melloswitch:/tmp$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.245 port 5001 connected with 10.10.10.4 port 31197
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  1.74 GBytes  1.49 Gbits/sec (*)
[  5] local 10.10.10.245 port 5001 connected with 10.10.10.60 port 50120
[  5]  0.0-10.0 sec  1.09 GBytes	934 Mbits/sec (**)

(*) FreeNAS 11.2-BETA2 @ 10Gb
(**) ubuntu desktop @ 1Gb

I've set mtu for the 10Gb NICs to 9000 (I think that's the recommended) on the shell, as the FreeNAS interface doesn't show on network -> interfaces to be configured.

FreeNAS documentation has:

Code:

Typically, the interface used to access the FreeNAS® administrative GUI is configured by DHCP. This interface does not appear in this screen, even though it is already dynamically configured and in

So I've set the mtu from the shell and I'm 99.99% sure won't survive a reboot.

Question: How can I permanently change the mtu on that interface?

 

[melloa]

Question: How can I permanently change the mtu on that interface?

Never mind... You can add the interface to the entwork -> interfaces and set mtu there.

 

[Chris Moore]

In another recent "upgrade to 10G" thread, @Chris Moore posted this link to a 52-port (48 x GbE, 4 x SFP+) switch on eBay for ~$150. A couple of direct-attach twinax cables, or optics and fiber patch cables, and you should be set. For NICs, I think Chelsio is still the go-to brand.

I thought I would post my results here so anyone that was involved in the previous conversation would get the news.
I purchased one of these switches:
https://www.ebay.com/itm/Aruba-S250...ASE-T-4x1000BASE-X-10GBASE-X-KMJ/232504836333
I could not get it to connect to computers, only to other switches, using the copper DAC cables I had. I updated the firmware to the latest version that is available, and that appears to have corrected he problem with the switch being unwilling to connect. I needed to remove the 10Gb ports from the stacking group, update the firmware and I couldn't use the Cisco DAC cables I have. It may work with other brands, but it will still not work with the Cisco cables. It does work with the Arista Netwoks branded cables I have and I will be trying some other configurations soon with fiber transceivers. I will let you know how it goes. Time invested was the biggest cost here. The switch was only $125 and free shipping but I had to dig through a lot of documentation and even join their forum and ask some questions before I got what I needed.
I will be doing some speed tests later to see if there is an improvement over the home-brew switch I have been using. One thing I noticed immediately, it has a lot less heat output.

 

[melloa]

I will be doing some speed tests later to see if there is an improvement over the home-brew switch I have been using.

I've been using that solution on my Gigabyte GA-Z97-HD3 with an i5, to test the concept and works fine. Waiting on the supermicro MB to build and start testing. One thing that I thought was to try to install FreeBSD and configure it to do the bridge to test the difference.

One thing I noticed immediately, it has a lot less heat output.

That's no good... I already have lots of heat here.

 

[Chris Moore]

That's no good... I already have lots of heat here.

Maybe this weekend I can get my "Kill-o-Watt" tester and see the electrical load of the devices, but I will bet the PC was drawing more electricity than the switch does.

 

[melloa]

So, just received the X8 MoBo and it is DOA.

Good project, but it is time to move on... any good switch with 4 spf+ known to work for the purpose of connecting servers/workstations to the sfp+ ports without too much headaches? Any suggestion will be appreciated :)

 

[Chris Moore]

So, just received the X8 MoBo and it is DOA.

Good project, but it is time to move on... any good switch with 4 spf+ known to work for the purpose of connecting servers/workstations to the sfp+ ports without too much headaches? Any suggestion will be appreciated :)

Sorry about the DOA. The return policy with eBay should allow you to get your money back though.
I did get the four 10Gb ports on this switch working with a firmware update and a setting change to remove them from the stacking group:
https://www.ebay.com/itm/Aruba-S250...ASE-T-4x1000BASE-X-10GBASE-X-KMJ/232504836333
I have that running in my network now and I am pretty happy with it.

 

[melloa]

Sorry about the DOA. The return policy with eBay should allow you to get your money back though.

Please don't take this as your fault, not on you at all!

I did get the four 10Gb ports on this switch working with a firmware update

Cool. Will try it. When you have a chance, can you tell me what firmware you used (latest?) and if you have a link to remove it from the stacking group?

Again, thank you for your help!

 

[melloa]

The return policy with eBay should allow you to get your money back though.

Seller just refunded, by the way. I think was a honest mistake from his part.

 

[Chris Moore]

Cool. Will try it. When you have a chance, can you tell me what firmware you used (latest?) and if you have a link to remove it from the stacking group?

Again, thank you for your help!

I needed the USB to UART driver here to be able to connect to the USB port with PuTTY:
https://support.arubanetworks.com/Default.aspx?TabId=76&DMXModule=514&EntryId=7512

This is how to convert the stacking port. It isn't detailed, but it was enough to get me there:
https://www.arubanetworks.com/techd...#mas_guides/1command_List/delete_stacking.htm

This is the firmware I loaded, it is supposed to be the latest for that switch:
http://support.arubanetworks.com/Li.../121/DMXModule/661/EntryId/22078/Default.aspx

The instructions for installing the firmware are in the release notes. I tried using a USB drive, but that didn't work for some reason, so I used my NAS to create a tftp server and was able to download the firmware from there.

 

[Chris Moore]

PS. It does not like the Cisco DAC cables. It is probably some sort of vendor lockout. I was able to use a pair of these Arista cables with it:

https://www.ebay.com/itm/CBL-00005-...ER-3M-CABLE-/253017086622?hash=item3ae8fe5a9e

I also was able to use a pair of fiber transceivers from Fiberstore: https://www.fs.com/products/11589.html

I have a pair of Intel transceivers I will be testing, but I have not gotten that done yet.

 

[IQless]

Quick question: is there any specific reason you went with VyOS? Does it give any advantages over let's say pfSense?
I run a pfSense box as my router atm and were thinking of adding more NICs (10Gb), but would it be a better choice to go for VyOS as a separate switch?

Quite curious to know your take on this.