Dual NIC with Scale apps? Networking Sanity Check

[layer710]

To preface, I try to avoid posting questions on any forum whenever possible. I've spent a few days searching for answers on Google and here on the forum, and while I found a few threads that seemed promising, they were mostly talking about TrueNAS Core/FreeBSD. Ultimately, I feel more confused now than when I started searching. So, if I've somehow missed a thread containing an answer or solution to what I'm about to post, please forgive me and/or smack me with the stupid-stick.

I recently migrated my NAS rig from Proxmox to Scale RC2 with no issues, it's been a pretty good experience so far. I think Scale has a lot of potential, and I'm excited for the full release.

That said, I'm running into some confusion regarding networking.

For reference, my (simplified) topology:
WAN <-- Coax --> Cable Modem <-- 2.5G Eth --> opnSense bare metal on R420 <-- 10G Fiber Transit --> Ruckus ICX6450 <---> LAN

The goal for my Scale rig is as follows (system details in sig):
10G NIC -> Dedicated to local access (GUI, SMB, etc) via my local storage VLAN.
1G NIC -> Dedicated to apps. This interface would sit on my VPN VLAN which routes external traffic through my ProtonVPN gateway on opnSense. Purpose: Allow apps to access the internet via VPN. Not intending to allow any sort of remote/external access, just wanting apps to be able to pull downloads via VPN.

Am I barking up the wrong tree with how I'm envisioning this? It seems logical based on my knowledge, but I've still got a lot to learn - especially when it comes to TrueNAS, Kubernetes, and their respective nuances. If I am indeed on the right track, I could use some guidance regarding best practices for configuring the interfaces on TrueNAS Scale.

Also, a side note... so far I'm not able to maintain a link UP status with my current 10G NIC (Emulex) regardless of how I try to configure it. Best I've been able to achieve was getting the interface to cycle betwen UP and DOWN every minute or so, but now it's just stuck DOWN. Based on what I've been able to find during my searches, I'm assuming this is due to an incompatibility between TrueNAS and Emulex NICs. Regardless, I have a Chelsio 310 en route, and am hoping that will help resolve the issue. I'm using 3M SFP+ Active Fiber DAC's for 10G connectivity to my ICX6450 switch. These cables work great for the transit network between my switch and firewall (running a Mellanox CX3). Since I have a couple spares available, I'm hoping that they will work with the Chelsio card as well.. but I'll pick up separate fiber & transceivers if needed.

Anyway, the NIC-specific issues aren't terribly important yet, as I don't want to waste time chasing ghosts - I'll dig back into that issue once the Chelsio arrives this weekend. Just wanted mention it in case anyone out there has had a similar experience and managed to get it sorted out.

For now, if someone could please provide feedback regarding my goal listed above, I'd really appreciate it. I'm eager to learn the proper way to configure this system.

Thanks so much!

 

[layer710]

Just thought I'd post a follow-up.

Swapping to the Chelsio 310 went nice and smooth, no extra fuss needed to get it working. Looks like the Emulex card was indeed the root of my previous NIC issues. Still yet to confirm the exact cause, but I suspect either an incompatibility with my DAC, Scale itself (doubtful), or it's just a dud card. Will test in another rig with a compatible RJ45 transceiver when time permits.

As for networking side of things, I was able to easily swap the interface configs so that the 10G sits on my storage vlan and serves the main system/GUI, and the onboard 1G serves the Kubernetes cluster and routes all outbound app traffic through my VPN vlan. So far, this seems to work well and accomplishes my goals for the time being. We'll see how well it holds up over time as I continue adding and customizing apps.

That said, I'm still eager to learn more about best networking practices with Scale. Can't wait for the full release on the 22nd!

 

[jgreco]

the Emulex card was indeed the root of my previous NIC issues

Yeah, it's not expected to work well. Lots of hardware out there isn't particularly compatible with FreeBSD or Linux, which is why we try to guide people towards stuff like the Chelsio and Intel cards.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

 

[layer710]

Yeah, it's not expected to work well. Lots of hardware out there isn't particularly compatible with FreeBSD or Linux, which is why we try to guide people towards stuff like the Chelsio and Intel cards.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

When I started running into issues with the Emulex last week, I dug around and wound up reading through the primer you linked. It was a big help in selecting a replacement NIC. I greatly appreciate all the effort you put into that guide.

FWIW, I had picked up the Emulex on a whim for cheap right before I migrated my NAS over to Scale. Since it wasn't purchased with TrueNAS in mind, it was a 'throw it in and cross my fingers' situation. I'm not heartbroken that it didn't play nice, overall it was a pretty painless learning experience. And, with any luck, I may be able to reuse the Emulex to upgrade the wife's workstation to 10G, so.. really can't complain!

 

[jgreco]

I greatly appreciate all the effort you put into that guide.

Definitely my pleasure. It's always my hope to help someone avoid the sometimes rough learning curve of needing to learn several unfamiliar things all at once. Once you get down the right path, it's networking nirvana. Or at least as close as networking gets to it. Heh.