Issues with VM's and Dockers assigned IP's

[SHoover80]

Frankly, I'm a little livid at this point. I've been dealing with an ongoing intermittent random issue with one of my TrueNAS servers. I'm running a Windows VM, a couple of media servers and NAS for my media. I'm running this on a Dell R510 server with TrueNAS-SCALE-22.02.3. I originally started experiencing this issue with TrueNAS-SCALE-22.02.2. I started off with just the two LAN ports on the server and started having issues where I couldn't access my services over the reserved LAN IP, I had setup and managed by my PFSense firewall. For example, I had one ethernet port reserved as 10.0.0.10, and the other setup as 10.0.0.7. I had 10.0.0.10 for what I used for my services (VM and media server) and 10.0.0.7 for my NAS. Then it started off where I couldn't access anything through 10.0.0.10. and had to use 10.0.0.7 for everything. While troubleshooting I verified the NIC had the correct IP and Mac in my firewall settings, and it showed as an active connection but something in TrueNAS scale appeared to have been refusing to utilize that local IP or that ethernet port. And while trying to make changes in the config to find a way to fix it I ended up doing something where I couldn't even access the TrueNAS GUI or any of my services. I had to wipe everything and do a fresh install then backup from my config and everything worked as setup. But in case I ran into the issue again and got blocked I added a USB to ethernet adapter I had spare just lying around. and I've upgraded to 22.02.3 after I had that issue a couple weeks ago and now I'm going through the same stuff where the other devices on my network I have setup to use the 10.0.0.7 and 10.0.0.10 are having problems connecting though everything with the hardware is functioning and working, Firewall and all other devices, servers, firewall everything thing else is working as it should but this TrueNAS Scale isn't doing what it's supposed to do. I've made no changes to any of the configs or hardware. The only way I can access anything is through the USB to LAN IP. And to be honest, I work full time dealing with IT, Networking, and nothing worse than being off work with other priorities I should be focused on or simply be able to relax TrueNAS Scale OS I'm running on this machine not doing what it's configured to do again is a bit infuriating.

On top of it, refusing to use the reserved IP's and Ethernet port I have configured. I was dealing with Windows 10 VM remote connection through Remote desktop randomly not wanting to connect, the configured MAC# assigned to VM changing on its own, then when I fix one issue and able to log into Windows 10 VM, it connects or detect network shared drives files hosted on the same machine with SMB shares though I can access from other devices on my network. And the Windows 10 VM can see those other networked devices. So, I restarted the VM again. I then randomly I can't connect through the remote desktop to the VM running on the same machine, but I can connect to other VM running on other servers I have something other than TrueNAS on. Though I can't connect through Remote Desktop at that moment and can choose the display and connect through VNC through the small limiting display. And at this point I can see the shared NAS folder hosted on the same machine I couldn't before the restart but now I can do that I can't connect through Remote Desktop. So, I restart the TrueNAS and I restart my firewall once everything boots up, I check. Still no RDP remote connection, only through VNC's limited display it won't allow to adjust the size and I notice it's showing no internet connection at all through the Virtual machine. It's as if every time I restart the machine it's a luck of the draw what's going to work and what's not going to work as configured. When troubleshooting gets to where something works something else stops working as it's configured.

Though I can access the TrueNAS server GUI on any of the reserved IP's. So, at this moment it's as if everything with the TrueNAS itself works, I can access my NAS running on that machine through the correct reserved IP setup in my firewall. I can access the TrueNAS Scale GUI with no issue but it's as if the VM's and the Docker containers are having inconsistent intermittent issues randomly happening with no justification, no errors, nothing as to why it's not being reliable or running as configured. My other TrueNAS server running on a Dell R620 has had no issues, but I don't run any VM's or dockers on it, only NAS. Also, I have a Template I use for my Windows 10 VM. It's a fresh installation of Windows 10. Then in case I have issues with my windows 10 VM after using it for a while I can just erase and make a copy of the fresh install, I've tried that and am still getting these random issues with the VM. and I've tried the same thing with the dockers. Uninstalling/deleting everything and starting over but the networking issues are currently persisting though everything was working just fine yesterday. I had just wiped the TrueNAS and about to do a fresh install of it which I believe will fix it for now but based on my past experience I'm likely to be forced to deal with this again so I figured I'd reach and see if it's a bug or if by chance someone has a recommendation that may help. I've wasted an entire day with this, and I would prefer to be able to enjoy my days off and step away from the computer. If this is what i can expect to deal with using TrueNAS I'll go back to Unraid.

 

[jgreco]

For example, I had one ethernet port reserved as 10.0.0.10, and the other setup as 10.0.0.7. I had 10.0.0.10 for what I used for my services (VM and media server) and 10.0.0.7 for my NAS. Then it started off where I couldn't access anything through 10.0.0.10. and had to use 10.0.0.7 for everything.

Without going into a lot of detail, this is a broken networking design on your part. Please review

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

and use link aggregations and IP aliases instead. It's really difficult to unroll problems like yours when the fundamentals have been sabotaged so that things don't work correctly.

 

[SHoover80]

Without going into a lot of detail, this is a broken networking design on your part. Please review

Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking practice. Multiple interfaces on a single network...

www.truenas.com

and use link aggregations and IP aliases instead. It's really difficult to unroll problems like yours when the fundamentals have been sabotaged so that things don't Y

You're either not understanding or don't know what you're talking about with your condescending response. There's not an issue with my network design. Evidence by the fact that everything else works flawlessly on my network Not to mention my degrees, certifications, and many years of experience in networking. So how about a little more than a condescending response and the irrelevant link. This is not an issue pertaining to Windows nor is it an issue pertaining to having multiple network interfaces. As TrueNas is not windows and the issue initially started when I was using a single LAN port I had assigned as 10.0.0.10. Granted it could be a networking configurating within the TrueNAS server itself as I am new to TrueNAS and have limited experience with it. But I do have some experience with other VM and NAS software currently running on my network flawlessly nonstop for over two years only down time being due to updates and/or cleaning dust out of hardware. And I've followed all information provided by tutorials regarding how to install and configure dockers and VM's in TrueNAS Scale. Which are where I'm experiencing issues Only since the first update on TruNAS Scale 22.02.2 then on to 22.02.3. No problems accessing the GUI on the configured IP. 10.0.0.10 and by default when configured is the same IP I was using with the port assigned to my media servers to access from my media players: Roku, Firestick, Windows Browser, ETC. For example, I'd enter http://10.0.0.10:32400 Being how it's configured and suppose to work and has been working this way. Till it stops working from any PC on the network or media player. So I log into the TrueNAS Scale server by 10.0.0.10/ui/sessions/signin and log into the GUI and go to Apps and click on Plex where it says Open and suddenly it opens for example 10.0.0.172:32400 instead of the 10.0.0.10:32400 address it has been working on. During the same time this shift happens I experience the same thing from other apps while trying to access them. The Windows 10 VM running I try to access SMB shares running on the same TrueNAS which is done by typing \\10.0.0.10\ into file explorer no longer working from the Windows 10 VM but does from any other computer on the same network including VM's from another machine hosted on a different OS. After dealing with that is when I tried utilizing the second LAN port. So yes, it is definitely a networking issue, but something within the TrueNAS system itself. possibly a configuration I'm not aware of. But I left all the networking configurations default when installing apps and in TrueNAS Scale itself. And in my PFSense Firewall I have the MAC # for that TrueNAS Scale Lan port reserved (Static) as 10.0.0.10. so, on my local network all traffic going to anything on that server should be going to 10.0.0.10. There is no difference in configuring a static IP on the TrueNAS server itself or allowing DHCP on server and configuring said DHCP to assign only desired static Ip to that same MAC address.

 

[Samuel Tai]

You're confusing terseness for arrogance. With multiple interfaces on the same subnet, they're essentially the same routing preference, so output packets randomly select 1 of the 2. This can update ARPs on other devices on the subnet, so pods can randomly change IPs they answer on.

If this isn't what you intend, then you can either:

• Set routing metrics on interfaces
• Create bonded interfaces
• Move one interface to a different subnet

What you have now is a recipe for chaos, which is what you're currently experiencing.

 

[SHoover80]

When someone knows what they're talking about and how to trouble shoot don't start off by making assumptions. As I said in my second reply after the 22.02.02 update is when I first started experiencing these issues. When I was only using a single interface assigned the IP: 10.0.0.10. Then I started experiencing these issues. Then I added the second then the third trying to set it up where my dockers or apps could use one IP, and the GUI for TrueNAS would have a dedicated IP. I even tried setting that on a separate network 10.0.10.* But regardless of which method I've used It works great for a while on the way I have it configured then randomly not long after a update of the apps or a restart and suddenly I'm not able to access my services by the IP they're configured and have been working. So, how about fewer trolls with nothing to offer. I'm only interested in valid insight or suggestions.

 

[Samuel Tai]

I meant to say you're confusing @jgreco's terseness for his arrogance. Your anger and long-windedness is making us helping you very difficult. Your stream-of-consciousness report is very difficult to parse to pick out details of your problem, as it lacks paragraph breaks, and sometimes punctuation in the middle of relevant details. Since you mentioned multiple interfaces on the same subnet near the end of the post, that's what we naturally homed in on.

If you think we're idiots, fine. Explain it to us in small chunks. However, we both have extensive IT experience as well, and aren't idiots. Besides, why are you insulting 2 moderators who can ban you if you get too creative with your insults? A little courtesy goes a long way.

OK, so after the 22.02.02 update, you started to see problems with apps experiencing what, exactly? Apps changing their DHCP addresses? Did you try:

• Reverting to the earlier release, and submitting a bug report?
• Checking the container network settings: bridge vs host?
• If bridged, did you try hardcoding the container MAC to your desired IP address?

Please, calm down, and step away from the keyboard for a while until you can re-engage without fight-or-flight instinct prompting you to attack us.

 

[jgreco]

don't know what you're talking about

Wow, that's ... unlikely. I've been doing advanced networking with FreeBSD since before it was FreeBSD.

This is not an issue pertaining to Windows

I never said it was. Clearly you're not comprehending what was provided to you.

Not to mention my degrees, certifications, and many years of experience in networking.

Well, then you already know everything you need to know. I'm happy that everything is working well for you and you're not having any problems with it. There aren't that many people here with deep IP networking experience, and the attitude you're displaying is likely to cause you to burn through the remainder of them quickly, if your issue even comes to their attention. I've written extensive articles trying to make complex TrueNAS issues accessible to newcomers, and generally I'm happy to help straighten out anyone who is willing to soak up clue. I don't get that impression from you, so I'm likely going to bow out at this point, absent some significant attitude readjustment on your part. This is not how you benefit from participation in a community forum with people who are looking to help you.