Upgrade recommendations

[melloa]

Folks,

I'm planning to upgrade my switch and get one with four 10Gb ports. I only have two servers and one workstation that I'd need the 10Gb speed.

Below some info/brainstorming:

• The servers are on the same rack and the switch 8 ft away.
• The servers have X9DRi-LN4F+ motherboard with four 1Gb interfaces
• All on ESXi VMs, included FreeNAS
• The 1Gb ports would continue been used as 1Gb for all VMs but FreeNAS
• Install one 10Gb NIC on each server and my workstation
• Pass through those 10Gb cards to FreeNAS VM
• My workstation requires a 30+ ft cable

I don't understand anything about SFP+, completely n00b here.

Question:

1. What switch (used/eBay) would meet above need?
2. What cards would be the best one, compatible with FreeNAS?
3. Cables ... recommendation please.

For instance, I saw this on eBay: https://www.ebay.com/itm/Cisco-WS-C...5160068?_trksid=p2349526.m4383.l4275.c10#rwid

Would something like that do what I'm planning? If so, cables ... ;)

Appreciate any help.

 

[Elliot Dierksen]

In shorter range deployments, some people like using the twisted pair 10G but I have never used it. Using twinax (also know as direct connect) cables is usually the cheapest option. It is a think cable with a fixed SFP+ on either end. The disadvantage to those is that they can be difficult to run through cable management. It also depends on the switch and NIC cards you use. The twinax has to be supported by both ends. Who makes the twinax cable can make a big difference. I just looked at the switch you referenced, and twinax wouldn't be an option there. The switch has two X2 ports in it. I have some similar switches that I use, but you would have to use SFP+ optical transceivers and fiber patch cords. That isn't a bad thing IMHO. You can give each side of the link optics that it likes. Fiber patch cords are smaller and easier to run, as well as cheap to replace if you need different length cables later. As far as NIC cards go, Chelsio cards are the NIC of choice for FreeNAS. I am a bare metal FreeNAS guy, so I can't comment on the pass through stuff.

 

[Elliot Dierksen]

I thought of some other things. I am inferring that you sit in the same room with this equipment. Noise can be a HUGE consideration with some enterprise networking gear. I mostly work with Cisco stuff, and an older Nexus 5010 was in the same price range as the HP 5820X (really a re-badged H3C) switch I purchased. The Nexus stuff has really loud and high pitch fans that are AWFUL to sit near. Some of that comes with the territory of 1U equipment (I can see @ChrisMoore twitching at the mention of 1U fans. :) ). There are versions of the 4948 switch that have 4 SFP+ ports on Ebay as well. I do wonder if your workstation can really make use of that 10G connection, but I don't want to diminish the nerd coolness of 10G to the desktop! :)

 

[Elliot Dierksen]

Something like this perhaps? https://www.ebay.com/itm/Cisco-WS-C...688038342?epid=2202127575&hash=item5b326d27c6

I haven't used these, so I can't comment on the noise level.

 

[Chris Moore]

Something like this perhaps? https://www.ebay.com/itm/Cisco-WS-C...688038342?epid=2202127575&hash=item5b326d27c6

I haven't used these, so I can't comment on the noise level.

I have one like that at work. It is a little on the loud side for in the home office. I am also pretty sure those are just SFP ports, not SFP+, so 1Gb instead of 10Gb.

I can see @ChrisMoore twitching at the mention of 1U fans. :)

True. Those buzzing little monsters are the bane of my existence at work. I hate having to work in the hot isle and it isn't the heat that bothers me. I am supposed to wear hearing protection, but it makes hearing other people talking to me difficult.

You might want to watch this video series on YouTube. I followed the instructions and built my own switch with less than $100 of parts from eBay. It works great.

10Gb Home Network (P1) - Introduction - https://www.youtube.com/watch?v=MgNpI6VAAhI

10Gb Home Network (P2) - Peer to Peer - https://www.youtube.com/watch?v=KHiucyRYX_w

10Gb Home Network (P3) - Build A Switch - https://www.youtube.com/watch?v=p39mFz7ORco

PS. Plus some parts I already had laying around the house.

 

[Elliot Dierksen]

I have one like that at work. It is a little on the loud side for in the home office. I am also pretty sure those are just SFP ports, not SFP+, so 1Gb instead of 10Gb.

Crap, you are right. Here is what I meant to reference. It is more expensive because of the 10G. https://www.ebay.com/itm/USED-Cisco...x-10-100-1000-4-SFP-AC-PS-Fr-Ext/192226238698

 

[melloa]

@Elliot Dierksen and @Chris Moore

Thank you guys for your replies.

I thought of some other things. I am inferring that you sit in the same room with this equipment.

Not really, it is on a closet inside the wife's deposit room, but I'd prefer to keep the noise down as I open the doors when it's too hot and have a little fan exhausting the hit (yes ... another unfinished project: install the A/C duct for that closet and a return).

In shorter range deployments, some people like using the twisted pair 10G

Will consider after the final decision on switch.

I do wonder if your workstation can really make use of that 10G connection, but I don't want to diminish the nerd coolness of 10G to the desktop! :)

I do all my work, including saving my DVDs and b-rays on my desktop. Copying at a faster speed would be nice, but I got your point. If connecting only the two servers, I could go the two NICs approach and not have the switch cost. Decisions, decisions ...

Crap, you are right. Here is what I meant to reference. It is more expensive because of the 10G.

The funny thing is that the Netgear Prosafe XS708E is going down in price. When @jgreco wrote the 10G network primer, it costed $800+. I just did a search and found one $65 more than the WS-C4948E: https://www.ebay.com/p/NetGear-XS708E100NES-Network-Device/168511862?iid=202427728868&chn=ps

Didn't have time to run a tab on three nics, switch, and cables, so not sure that $65 more would hurt my packet when all considered.

I have one like that at work. It is a little on the loud side for in the home office.

Thanks man! I'll scratch it from the list :)

. I followed the instructions and built my own switch with less than $100 of parts from eBay.

You know ... I was thinking about that last night. Will watch and I'm 99% sure I'll go the BYOS (build your own switch) route as I only learn doing, so more experience for this old guy.

Time for a new pot of coffee and watch those videos.

Thanks again, guys, for your replies.

 

[Stux]

FWIW, I quite like my netgear pro safe XS716T.

Even works well for gigabit clients as it ensures them a full gigabit to the 10gbit servers.

 

[Chris Moore]

I used an old system board, processor and RAM that I already had and was getting 5x better performance than the 1Gb switch I replaced. I still have a bottleneck somewhere, but have not done enough testing to find it. It was a fun project to build and I got four SFP+ ports and ten 1Gb copper ports. That's plenty of connectivity for my home.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[melloa]

I used an old system board, processor and RAM that I already had and was getting 5x better performance than the 1Gb switch I replaced. I still have a bottleneck somewhere, but have not done enough testing to find it. It was a fun project to build and I got four SFP+ ports and ten 1Gb copper ports. That's plenty of connectivity for my home.

@ home I'm using a WS-C3750G48PS, that I got to power my PoE devices, but as the runs are ~100ft, I have noticed that the deivices were not powering on correctly, so PoE injectors were added. With that said, I can stop using that and replace with one box built using VyOS. But I'll need to find good prices on parts, or will cost more than buy a pro safe. Today's cost is ~650 for the switch and ~200 for the cards, so for three boxes we are talking about $1250 ... so need to wait for Santa.

 

[danb35]

In another recent "upgrade to 10G" thread, @Chris Moore posted this link to a 52-port (48 x GbE, 4 x SFP+) switch on eBay for ~$150. A couple of direct-attach twinax cables, or optics and fiber patch cables, and you should be set. For NICs, I think Chelsio is still the go-to brand.

 

[Chris Moore]

In another recent "upgrade to 10G" thread, @Chris Moore posted this link to a 52-port (48 x GbE, 4 x SFP+) switch on eBay for ~$150. A couple of direct-attach twinax cables, or optics and fiber patch cables, and you should be set. For NICs, I think Chelsio is still the go-to brand.

I actually bought one of those. I am having trouble getting the 10Gb ports to work with a computer. They are intended to be for connecting to other switches in a stack and it isn't playing at all so far, but I am still digging into the interface. Right now, I am still using my home brew switch. If I get the Aruba switch working, it will be interesting to see if it improves the transfer speed I am seeing with the VyOS switch.

PS. If I get the Aruba switch working, I will certainly be telling folks it is the way to go and what I had to do to get it working.

 

[Chris Moore]

@ home I'm using a WS-C3750G48PS, that I got to power my PoE devices, but as the runs are ~100ft, I have noticed that the deivices were not powering on correctly, so PoE injectors were added. With that said, I can stop using that and replace with one box built using VyOS. But I'll need to find good prices on parts, or will cost more than buy a pro safe. Today's cost is ~650 for the switch and ~200 for the cards, so for three boxes we are talking about $1250 ... so need to wait for Santa.

I bought a couple of these cards to use in the switch build because I wanted the port count:
https://www.ebay.com/itm/Chelsio-10...8-30-2x-10G-SFP-Tranceivers-HIGH/192647386240
On the Windows desktop and on the FreeNAS, I used these:
https://www.ebay.com/itm/2x-Mellano...Card-1x-SFP-10G-gigabit-cable-3M/153053070898
It may be the Mellanox cards that are the reason my transfers are a bit slower than I would like, but the drivers are supported by FreeNAS as of FreeBSD 10, if I recall, but it does work. It might work better / faster with the Chelsio cards but I can't say. I bought four of these Mellanox cards and they are in all my end points with the two Chelsio cards in the switch.
For 1Gb connectivity, I already had some of these laying around:
https://www.ebay.com/itm/DELL-INTEL...-SERVER-ADAPTER-EXPI9404VT-YT674/201528557612
I had four of them and two of them turned up defective, or didn't work, still I have eight ports there and the two built into the system board, for ten 1Gb ports and four 10Gb ports and since I already had the computer I put the cards in, the only cost for me was the purchase of the 10Gb cards.

 

[Jessep]

I found this recently when looking for 10Gb switches (changed to US store, AU prices not as nice):

https://www.fs.com/products/72944.html

I haven't bought one, but by the spec's its a Broadcom based reference design.

Fan-less, cheap, (x4) SFP+, 21w MAX power, plus a free 24 port switch.

 

[Chris Moore]

I found this recently when looking for 10Gb switches (note this is AU store, change to US for even cheaper prices):

https://www.fs.com/products/72944.html - - $279.00 USD and free shipping.
It is more than that Aruba switch @danb35 linked above, but if it works reliably, it might be a good value option.
I notice it has a MAX power draw of 21watts and that is nice. The VyOS switch I built is running in a desktop computer chassis and that draws about 120watts. Your mileage may vary depending on your components.
Still, if you build the VyOS switch, it might do all you need it to do.

Options are good.

 

[Chris Moore]

Fan-less, cheap, (x4) SFP+, 21w MAX power, plus a free 24 port switch.

It concerns me that they only list the SFP+ ports as being uplink. Have you tried this? The Aruba switch I bought has 4 SFP+ ports that were intended to be for stacking and I can't (so far) get them to work to connect a computer as an end point but they do link when connected to another switch.

 

[Jessep]

It concerns me that they only list the SFP+ ports as being uplink. Have you tried this? The Aruba switch I bought has 4 SFP+ ports that were intended to be for stacking and I can't (so far) get them to work to connect a computer as an end point but they do link when connected to another switch.

The 3900 is new so there isn't much info about it yet. I did find info on the 3700 and 3800 which use the same term "uplink" for the SFP+ ports and people are using them as normal ports.

https://www.reddit.com/r/homelab/comments/7ffzxa/reviews_of_fscom_1g_switches/

Other thing that intrigued me was that it is listed as having dual power supplies standard.

If I needed more than (2) 10Gb ports I would buy one, as it is I have an AIO ESXi/FreeNAS and a workstation in use, my x3550 M5 was just too loud so I've retired it.

 

[kdragon75]

Ubiquiti Networks ES-16-XG EdgeSwitch
I keep trying to find reasons not to buy this switch.

 

[Elliot Dierksen]

Ubiquiti Networks ES-16-XG EdgeSwitch
I keep trying to find reasons not to buy this switch.

A friend of mine has a 1G flavor of the Ubiquiti network switch. It does appear to be kind of fussy about the optics/twinax cables. I have only looked at the GUI in it a bit to help him out. I am crusty CLI kind of guy...

 

[kdragon75]

A friend of mine has a 1G flavor of the Ubiquiti network switch. It does appear to be kind of fussy about the optics/twinax cables. I have only looked at the GUI in it a bit to help him out. I am crusty CLI kind of guy...

Yeah that's why I would go with the edge version and not the unifi version.