Can ESXi act as a switch?

[rvassar]

I've been running a "SAN" network here on my home office desk using an isolated switch, and a VM acting as a router. It was mostly to keep the FreeNAS <-> ESXi traffic off the house network. Now that I've settled in to a more stable FreeNAS configuration, I'm looking at adding 10GbE SFP+ cards to each host and start taking advantage of the performance I have untapped. However...

A couple items crept on to my little 8 port desktop switch... A private WiFi AP to isolate me from the wife & kids streaming, and my primary workstation. So I've looked at the VyOS solution, where I install a DP 10GbE card and a quad 1GbE card in a simple PC and configure the VyOS to bridge the ports. I'd prefer to not expand my chassis count if possible. I'm wondering if I can coerce ESXi to do the same thing using a vSwitch.

Has anyone tried this? (On Edit: I do not have vCenter available to me, so DvSwitches are not an option...)

 

[Elliot Dierksen]

How many ports are looking to add? Do you need multiple VLAN's, or some 10G and some 1G ports? You might be able to do it, but it really depends on the density that you need. There may very well be some smaller switches that would do a better job.

 

[kdragon75]

In short. No, ESXi will not act as a switch.

 

[rvassar]

How many ports are looking to add? Do you need multiple VLAN's, or some 10G and some 1G ports? You might be able to do it, but it really depends on the density that you need. There may very well be some smaller switches that would do a better job.

I only have two devices that can really make use of, or need, that kind of speed, it would be nice to have the option to expand to a third. I have looked at the little Netgear switch that has 2 x 10GbE RJ45's, and ~8 x 1GbE ports. But those are still $200+, and can't use the cheaper Intel SFP+ cards. They do have a 5 port SFP+ switch, but it's $350+... Either of which gets me into WAF territory, and I'm married to an accountant. ;)

 

[rvassar]

On the VLANs... I have some of that capability now, but I try to contain most of my mass damage stuff to the SAN network on my desk. But I have 1gig fiber to the rest of the world, so I've discovered that "sharing the wire" into the house router with a teenage boy can become something of a challenge. I've also formed opinions on the speed of Telco DNS servers, and have lowered my expectations of AC1900 router CPU's. I actually have to run a separate DHCPd because the house router is so busy hauling the mail that it drops address renewal requests.

 

[Elliot Dierksen]

If you have one host, and one FreeNAS then you could just do a twinax between the two. Is the non-freenas host ESXi? If so, you can do this. You would want to create a new Vswitch in the ESXi host. Where you go next depends on whether you connect to the data store via NFS or iSCSI. If NFS, you would create another VMkernel port in the new (10G) Vswitch and assign it an IP on the same network as the 10G port on FreeNAS. It likely similar with iSCSI, but I can't say for sure as I don't do iSCSI.

 

[kdragon75]

new Vswitch in the ESXi host.

No need. you can use portgroup based overrides to push the traffic out he 10gb card. Then you keep the ability to failover to your 1gb network if needed. Just setup iSCSI PSP for failover and back.

 

[rvassar]

If you have one host, and one FreeNAS then you could just do a twinax between the two. Is the non-freenas host ESXi? If so, you can do this. You would want to create a new Vswitch in the ESXi host. Where you go next depends on whether you connect to the data store via NFS or iSCSI. If NFS, you would create another VMkernel port in the new (10G) Vswitch and assign it an IP on the same network as the 10G port on FreeNAS. It likely similar with iSCSI, but I can't say for sure as I don't do iSCSI.

Yes, 1 NAS + 1 ESXi... That's sort of what I was looking at, but I was trying to avoid adding yet another subnet for 10GbE. I do have a router VM that can tie it all together... Just odd complexity and introduces VM state dependance. NFS is the primary access method. I dabbled with iSCSI, and would like to look at it further, but I don't have any running at the moment.

 

[Elliot Dierksen]

No need. you can use portgroup based overrides to push the traffic out he 10gb card. Then you keep the ability to failover to your 1gb network if needed. Just setup iSCSI PSP for failover and back.

I am more of a network infrastructure person than a server person. That said, I thought it was best practice to do the storage in a separate vswitch. It exceeds my ESXi knowledge about how it would handle that kind of failover.

 

[kdragon75]

adding yet another subnet for 10GbE.

is EXACTLY the best way to do it as long as its on its own VLAN. You don't need to do any routing or DHCP on this network either.

If this is point to point, just set the IPs in the same subnet on FreeNAS and ESXi. You don't need anything else. No router.

 

[Elliot Dierksen]

Yes, 1 NAS + 1 ESXi... That's sort of what I was looking at, but I was trying to avoid adding yet another subnet for 10GbE. I do have a router VM that can tie it all together... Just odd complexity and introduces VM state dependance. NFS is the primary access method. I dabbled with iSCSI, and would like to look at it further, but I don't have any running at the moment.

I could be wrong, but I don't think you could get FreeNAS to share an IP across multiple interfaces unless you build a bridged interface (like for VM's in FreeNAS, but I haven't done that). I have my network set that way and use NFS. Nothing can reach the IP network/VLAN that is used for storage but FreeNAS and the storage VMkernel NIC's. As long as you use an RFC1918 private address that doesn't conflict with your main network, it doesn't matter what IP's are on that network.

 

[Chris Moore]

So I've looked at the VyOS solution, where I install a DP 10GbE card and a quad 1GbE card in a simple PC and configure the VyOS to bridge the ports.

VyOS runs in a virtual machine, so you can pass the cards to the VM and have that running inside the ESXi system you already have, if you have enough slots to plug the cards into.

I run a VyOS system for my switch as a separate chunk of hardware and it works for what I need to do at home. I am currently looking at purchasing a more 'normal' switch because I think it will be a little more power and space efficient. Maybe something like this would appeal to you also:

https://www.ebay.com/itm/Aruba-Netw...-100-1000-Mobility-Access-Switch/132135155050

 

[rvassar]

VyOS runs in a virtual machine, so you can pass the cards to the VM and have that running inside the ESXi system you already have, if you have enough slots to plug the cards into.

That would be an interesting configuration... But if I pass thru the cards to VyOS in a VM, how does ESXi itself gain access to the 10GbE network? I'm guessing a internal vSwitch with a vNIC assigned to VyOS as well? That matters because that VM has to be up before any other VM can access storage.

 

[Elliot Dierksen]

That would be an interesting configuration... But if I pass thru the cards to VyOS in a VM, how does ESXi itself gain access to the 10GbE network? I'm guessing a internal vSwitch with a vNIC assigned to VyOS as well? That matters because that VM has to be up before any other VM can access storage.

I can't help but feel that the separate Vswitch and twinax cable between FN and ESXi seems a lot less complex. Not to mention that fact that you don't actually need to route between the two. All IMHO, of course.

 

[Chris Moore]

But if I pass thru the cards to VyOS in a VM, how does ESXi itself gain access to the 10GbE network? I'm guessing a internal vSwitch with a vNIC assigned to VyOS as well?

I would expect that to work, but I have not tried it. VyOS does have support for running in a VM listed as one of their features. It could be interesting to try. As I understand it, VyOS can also be configured to handle some routing and firewall functionality.

 

[rvassar]

I am currently looking at purchasing a more 'normal' switch because I think it will be a little more power and space efficient. Maybe something like this would appeal to you also:

https://www.ebay.com/itm/Aruba-Netw...-100-1000-Mobility-Access-Switch/132135155050

I used to run a used 1U 24 port switch at home maybe 10 years ago, but it was in my garage, where I didn't have to listen to it.

I have looked at the Netgear XS505M and GS110MX. But they're just a wee bit too spendy...

https://www.netgear.com/business/pr...ti-gigabit-unmanaged-switches.aspx#tab-models

 

[Chris Moore]

From what I have read, this Aruba switch is supposed to be pretty quiet.

 

[rvassar]

From what I have read, this Aruba switch is supposed to be pretty quiet.

Too gun shy here... It might be, but I'm still getting used to the new normal here without the howling Poweredge (build #1). I also have to wonder what features are disabled. A lot modern Cisco kit is useless, or has functions restricted without a license, and that listing clearly states some activation key is not available.

I suspect I'm going to do the SFP+ direct lash up, and keep an eye out for a further break in 10GbE switch costs. The recent 802.3bz standard should result in more options. I had my house wired for Cat 5e when I had it built, so I should be able to use the 2.5 & 5 Gig signalling rates eventually. There are SFP+ to RJ45 10GbE GBIC's, but they seem to cost almost as much as the switches. So there is an expansion option coming... I just have to bide my time and wait for the right listing...

 

[Ericloewe]

I had my house wired for Cat 5e when I had it built, so I should be able to use the 2.5 & 5 Gig signalling rates eventually.

Why not just replace it with Cat. 6a, at least for the more interesting runs? Should be pretty easy with previously-installed cables.

There are SFP+ to RJ45 10GbE GBIC's, but they seem to cost almost as much as the switches.

Got a link?

 

[rvassar]

Why not just replace it with Cat. 6a, at least for the more interesting runs? Should be pretty easy with previously-installed cables.

I'm in Texas at roughly the same latitude as Cairo, Egypt, in a U.S. 2016 Energy Star compliant house. The exterior walls and attic have chemically expanded foam insulation. Any cables passing those spaces are effectively "glued" in place. It might be possible to pull some of them on the interior runs, but they're threaded thru drilled holes in the 2x4 framing structure, not in cable runs. Without pulling the sheetrock, it's probably not worth the attempt. Consider that 802.3bz specifies 2.5Gb for up to 100m of Cat 5e, and 5Gb on up to 100m of Cat 5e "on defined use cases and deployment configurations". There's only three devices in the entire rest of the house that could even handle receiving data > 1GbE, and the longest run is likely less than 25m... All the speedy stuff is in my home office.

Got a link?

There are lots of 1GbE listings to wade thru... Have no idea how compatible these are either.

https://www.ebay.com/itm/SFP-10G-T-...394170?hash=item25f3c2687a:g:V04AAOSwWHZa1Y1w

https://www.ebay.com/itm/SFP-10G-T-...261415?hash=item239c7c8467:g:s6MAAOSweM1Z9GL0

https://www.ebay.com/itm/HPE-10GBas...250728?hash=item3f9a373be8:g:~GUAAOSwcwVa2mj9