SOLVED Multiple NICs, 2 WAN, Separate Networks?

[jessicabrown8110]

Hey everyone,
Not to sure if this is even possible, but I thought I would ask to see if anyone has ever done anything like this and may be able to give me to pointers.

(Ultimate setup eventually)
I have 2 connections, 2 modems, and 2 different dynamic IPs from my ISP (Spectrum). I have 1 of the connections to my Asus Router, which feeds the house, and IOTs with 10.0.0.0/16 network. The second is into a TP-Link Router which feeds my TP-Link T2600G-52TS which then I have a TrueNAS cluster (TNA TNB, TNC). Eventually I will setup vLans with my managed switch, but for now, I just have the router. Each one of my TN systems have a NIC on the motherboard, and a 4 port 2.5GB NIC PCI. So each has 5 NICs. I have all 25 ports plugged into the switch. The switch: I have a trunk link vlan going to my Asus router called UPLINK, and I have. (This is the end setup once I get it all figured out).

(After failures, current setup)
Ok after a full weekend of failures trying to set that up. I decided to simplify, take it down to 1 TrueNAS, 5 NICs, 2 modems, 2 routers... 4 of my ports on the TrueNAS is plugged into the TP-Link Router, the 5th port is plugged into my Asus Router. (As I am explaining all of this, I am starting to think of things... Wheels spinning here... maybe I should only use 3 ports on the TN, and plug one from the TP-Link to the Asus, then port forward)... Anyways, what I am thinking is one NIC would be a connection to my Asus, and the other to my TP-Link... I don't know at this point, I am trying to be able to access TrueNAS, from my house network... Maybe I am doing this all wrong. Any thoughts?

 

[sretalla]

Your explanation is a bit difficult to follow. Perhaps a diagram would help.

First and possibly most important to read is this:

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

If that's clear (looks like maybe it is from what you said so far, but wanted to be sure), then it would be helpful to understand if the dual internet connection actually has any relationship with the TrueNAS systems in terms of what you want...

Do you want TrueNAS to run apps that serve either IoT or "home" apps, but are somehow separated from each other and use the two internet connections separately?

How prepared are you to work with kubernetes networking (and the extreme complications that come with that)?

How do you see the load balancing/failover scenarios between your internet connections? (if port forwarding will be done from both to one or other or both TrueNAS systems)

 

[jessicabrown8110]

I have drew up a diagram to help explain, I may have this not setup correctly, or maybe it is all in my configurations, I am not sure...

 

[sretalla]

OK, that diagram is an excellent start.

Now just to clarify what it is you want to happen and what's not working. See the questions in my last post (only a couple are answered by the diagram.

 

[jessicabrown8110]

Yeah, sorry about that, I was in the middle of answering that and this place that pays me to do THEIR work... (Pffft...), messaged me and I guess I had to join a stupid meeting... With 5 minutes notice, might I clearly add (HAHA)...

Anyways, I did figure out some of my issue (You are allowed to laugh at me, but I deserve it)... When I was drawing up that diagram I was grabbing IP addresses, and switching between my networks to get them. Anyways, like a 6 month old rookie playing 1st base in 4th grade, I had my ER7206 set with a /16 mask. My VLan 2 & 3, has 10.2.x.x and 10.150.x.x. (Once I get this completely setup I will have 2 Cybertronic 200TB SAN Systems that will also be included with VLAN 2, and clustered (maybe))

Back to your questions:
1. TrueNAS (As shown in the diagram) will run home apps

1.5. TrueNAS will use WAN 2 (74.x.x.x) as it's primary IO External Network connection segregated from VLan 1 and 3

2. Believe it or not I have never used Kubernetes, I am willing to learn, and if that is something that would benefit me I will take any courses I can find or are recommended.

3. Since TrueNAS will only use WAN 2, I only have port 80 forwarding from the ER7206 and forwarding to 10.0.2.x (which is my Portainer nGinX-Proxy-Manager). -- Should I have another Port forward for devices on VLan 1 to connect to VLan 2, or should I use VLan management to handle (Although I am worried about access from VLan 2 to VLan 1 by bad actors)

Hey... Also I really appreciate your insight and taking the time to answer these questions.

 

[jessicabrown8110]

The 2 main issues are:

1. Connecting to TrueNAS Web GUI while on VLAN 1
2. Having the Asus Router use the PiHole DNS to manage the kiddos and their ability to click on ANYTHING... (Yes I did scream that so they can hear my typing)

 

[jessicabrown8110]

The 2 main issues are:

1. Connecting to TrueNAS Web GUI while on VLAN 1
2. Having the Asus Router use the PiHole DNS to manage the kiddos and their ability to click on ANYTHING... (Yes I did scream that so they can hear my typing)

2 Words...
"VLAN Tagging"

mic drops (very dramatically I might add)

 

[sretalla]

Great that you managed to work that out... it would have taken a few questions around "where are your VLANs trunking internally" to get to that answer.

 

[jessicabrown8110]

For anyone who is interested, I have successfully setup everything. The Verizon MiFi Stick, SANs, and Servers are the only thing missing from this diagram because I am too lazy to add it tonight.

For those who would like to critique how I done it, any input would be appreciated, although prepare for stupid questions:

1. Setup the TP-Link Dual WAN router
   1. Created all the VLANs (10, 30, 50, 70, 150)
   2. Assigned IPs, Gateways, DHCP Pools, DNS, etc to each of the VLANs
   3. Assigned Tag and UnTag for each of the VLANs
2. Moved to VLAN 10, and connected to Asus Router
   1. Set Router to AP Mode, and created a DHCP for the 192.168 network
   2. Pointed the DNS to 192.168.0.1
   3. Setup the computers and Laptops
3. Moved to VLAN 150, and connected it to the TP-Link EA245 Wireless Access Point
   1. Created the VLAN 150, 70, and 30 in the EA245
   2. Created each Wireless SSIDs for printernet, iotnet (2.4 Ghz only)
   3. Created the Wireless SSIDs for printernet, worknet (5 Ghz only)
   4. Setup Tagging for each VLAN in the EA245
   5. Setup all home light switches, printers, googles, etc
   6. Setup work laptop
4. Moved to VLAN 50, and connected the TrueNAS Cluster with Bridge for all the NICs
   1. (Still working on this)
5. Setup 2 dedicated Bare-Metals for pfSense firewalls
   1. (Too much to explain, pretty much followed the docs with caveats to my network)
6. Document, Document, Document
   1. Started with the diagram, created a .md file to give each configuration for each endpoint
   2. The diagram shows what is connected to what device.
7. Backed up all Router, Switch, and Network related configurations.

On a side note, I did ditch the TP-Link T2600 because I have been messing with this for 5 days now, and I didn't want to mess with it since the EA7206 does a very good job with Firewall, VPN, and VLANs.

I have been an system administrator/engineer, and script/application developer for over 22 years professionally. I have never really setup a network with this much detail to security and segregation. Overall I am happy with how it is working, I am downloading about 1.2 - 1.5 Gibps and my uploads are close to 60 Mibps. directly from the router. I did test for WAN failure and the second WAN picked up without delay. I did put my generator in test mode, it killed the power to the house, and kicked on the generator just under 2 minutes. My battery backup allowed 22 minutes of operation for all of the devices connected to stay running. Watching the UPSc exporter, everything went very smoothly.

I hope this helps some others in creating a fun little project that caused me to loose most of my hair, and what is left is much greyer now.