Direct connection through a managed switch on a network I don't control

[mkaic]

Hello all! Above is a diagram of my network topology. I recently built a big TrueNAS server, and am planning to use it to serve research data to a number of Linux workstations. Initially, I was excited when I managed to get DHCP working properly and was able to mount my pool as simply as `mount -t nfs alice:/mnt/share /mnt/mountpoint`.

The wiring running through our building is only 1Gb, but I wanted something faster, so I bought a beefy switch and some AOC fiber SFP+ and QFSP cables, and ran a direct connection between `alice` and my workstation, `bob`. Unfortunately, this is where my networking knowledge ends. I've read through a number of other forum posts, but am feeling lost and not sure where to even start. In a perfect world, I would *really* love it if all traffic between `alice` and `bob` was routed through the fast cables and switch, *even if the traffic was sent using hostnames instead of explicit IP addresses*.

E.g., if I run `ping bob` from `alice`, that ping should go through the *black arrows*, not the *green ones*. And the same if I run `ping alice` from `bob`. All *other* traffic should still go through the company network as usual. Is this doable? I feel like there ought to be some way I can set up a rule to make this happen, but I haven't found anything close to what I'm thinking of yet in my googling spree. Any and all help/advice is greatly appreciated.

Thanks!
Kai

 

[Patrick M. Hausen]

Since there is no central management on the "black" network and no DHCP server running, you need to manually assign IP addresses to both the TrueNAS system and all the Linux workstations that use a faster connection to the "black" network.

What are the IP addresses of alice and bob on the company managed network? We can suggest how to configure your new interfaces once we know that - the adresses must not conflict.

 

[mkaic]

Since there is no central management on the "black" network and no DHCP server running, you need to manually assign IP addresses to both the TrueNAS system and all the Linux workstations that use a faster connection to the "black" network.

What are the IP addresses of alice and bob on the company managed network? We can suggest how to configure your new interfaces once we know that - the adresses must not conflict.

Unfortunately, the IP addresses of alice and bob on the company-managed network are not guaranteed to be static, which is why we always access them using their hostnames instead of IPs. They generally look like 10.219.130.###/23 though.

Re: no central management on the black network — could adding my own central management to that network possibly help here? I already tried connecting the switch to the company's network in the hopes that the company router would just "figure it out" and route traffic through the switch, but as far as I can tell that did not work.

 

[mkaic]

I should add that when I tried connecting the switch to the company network, I did so using the switch's Management port, since it was RJ45 and all the rest of the ports were SFP+ or QSFP. The connection light came on on both ends of the cable, but I was still unable to ping alice from bob with this configuration:

In fact, I wasn't able to ping alice from *any* workstation on the company network in the above configuration, which makes me think the management port probably can't be used in this way. If I were to find a way to properly connect the switch to the company network, though, shouldn't the company network's router be smart enough to figure out the shortest/fastest path between alice and bob is the black arrows?

 

[WI_Hedgehog]

Bob trying to reach Alice by name--this is kind of a problem because you have two paths to Alice, one via DHCP and one via "the black lines." Without some sort of reliable way to get to Alice you might establish a green connection but request data via black (when you really want to maintain the green connection), or other strange stuff where sometimes things work and sometimes they don't.

I think what you likely want to do is find the subnet you're on, like 192.168.xxx.yyy. yyy is a leased address and can likely change at some point (though it probably doesn't change often), however xxx likely won't change, and xxx is what you don't want to collide with. Let's say it's "1" as in 192.168.1.27, and 27 has been assigned to Bob.

You then stick an IP on Alice's new network card, like 192.168.ZZZ.1, and Bob's new network card gets 192.168.ZZZ.47, if you're partial to 47. So Alice is something like 192.168.152.1 noting 152 is probably going to remain unused by the company if they're using "1" for xxx. Anyway, Bob and Alice have static IPs, and you can have Bob map drives to Alice (and remember those mappings), so no real issues there.

With that said, your switch is kind of superfulous as only Alice and Bob are talking over the wire. If you wanted to share that out then you'd need the MicroTik switch so Alice, Bob, and "Tammy" could all talk. (Hopefully Bob and "Tammy" don't like each other, otherwise if they start talking lots you might wind up with other issues, and that's why we like closed ports.)

(cool "network person" background music)

 

[Patrick M. Hausen]

Unfortunately, the IP addresses of alice and bob on the company-managed network are not guaranteed to be static, which is why we always access them using their hostnames instead of IPs. They generally look like 10.219.130.###/23 though.

That's perfectly fine. We will pick a network that is completely separate from 10.* and no harm will be done.

1. You probably don't need the management port of the switch. A factory new switch will generally "just work" providing a single network just like an unmanaged one would.

2. You can keep the green connection to your company network for your TrueNAS. This way you can e.g. download updates, use NTP etc.

3. Now set static IP addresses for the black interfaces on both your PC and TrueNAS. E.g. 192.168.0.1/24 for TrueNAS and .2, .3, ... for all the PCs that will be using the black network.

4. Instead of the hostname "alice" access your NAS via its IP address 192.168.0.1.

That should do it.

 

[mkaic]

Bob trying to reach Alice by name--this is kind of a problem because you have two paths to Alice, one via DHCP and one via "the black lines." Without some sort of reliable way to get to Alice you might establish a green connection but request data via black (when you really want to maintain the green connection), or other strange stuff where sometimes things work and sometimes they don't.

I think what you likely want to do is find the subnet you're on, like 192.168.xxx.yyy. yyy is a leased address and can likely change at some point (though it probably doesn't change often), however xxx likely won't change, and xxx is what you don't want to collide with. Let's say it's "1" as in 192.168.1.27, and 27 has been assigned to Bob.

You then stick an IP on Alice's new network card, like 192.168.ZZZ.1, and Bob's new network card gets 192.168.ZZZ.47, if you're partial to 47. So Alice is something like 192.168.152.1 noting 152 is probably going to remain unused by the company if they're using "1" for xxx. Anyway, Bob and Alice have static IPs, and you can have Bob map drives to Alice (and remember those mappings), so no real issues there.

With that said, your switch is kind of superfulous as only Alice and Bob are talking over the wire. If you wanted to share that out then you'd need the MicroTik switch so Alice, Bob, and "Tammy" could all talk. (Hopefully Bob and "Tammy" don't like each other, otherwise if they start talking lots you might wind up with other issues, and that's why we like closed ports.)

(cool "network person" background music)

I simplified my network topology for the purpose of distilling my problem in this post, but ultimately the reason I bought a switch is because Bob is one of *several* workstations that I hope to make talk to Alice exclusively through the switch :)

 

[mkaic]

That's perfectly fine. We will pick a network that is completely separate from 10.* and no harm will be done.

1. You probably don't need the management port of the switch. A factory new switch will generally "just work" providing a single network just like an unmanaged one would.

2. You can keep the green connection to your company network for your TrueNAS. This way you can e.g. download updates, use NTP etc.

3. Now set static IP addresses for the black interfaces on both your PC and TrueNAS. E.g. 192.168.0.1/24 for TrueNAS and .2, .3, ... for all the PCs that will be using the black network.

4. Instead of the hostname "alice" access your NAS via its IP address 192.168.0.1.

That should do it.

Thanks a million for this advice. I had read about setting one's own static IP addresses before, but was worried because I don't know what IP address ranges are "allowed" in my company's network. I've only ever seen IPs that start with 10, but I guess I'm just worried about colliding with other IP ranges that are used for internal corporate purposes I'm not aware of.

I will give these steps a shot and report back. I appreciate you taking the time to explain all this!

 

[Patrick M. Hausen]

I had read about setting one's own static IP addresses before, but was worried because I don't know what IP address ranges are "allowed" in my company's network.

Your company's network uses addresses in the 10.0.0.0/8 network. The point is to find addresses that are outside of that, Because your "high speed" private network is completely separate from the company's.

 

[mkaic]

Thanks a million for this advice. I had read about setting one's own static IP addresses before, but was worried because I don't know what IP address ranges are "allowed" in my company's network. I've only ever seen IPs that start with 10, but I guess I'm just worried about colliding with other IP ranges that are used for internal corporate purposes I'm not aware of.

I will give these steps a shot and report back. I appreciate you taking the time to explain all this!

Following your steps, I set the IP of the relevant NIC on the TrueNAS machine to 200.200.200.1/24, and the IP of the relevant NIC on the PC to 200.200.200.101/24. I was not able to get either machine to ping the other using these IPs, though. I was also not able to mount my TrueNAS NFS share using the .1/24 IP.

I am also noticing a distinct lack of blinkenlights at all points along the black arrow route. No LEDs lighting up next to the sockets on the NIC in the TrueNAS server, the NIC in the PC, or the switch in between them.

 

[sretalla]

Following your steps, I set the IP of the relevant NIC on the TrueNAS machine to 200.200.200.1/24, and the IP of the relevant NIC on the PC to 200.200.200.101/24.

I think you specifically missed the point of @Patrick M. Hausen's advice and didn't use the 192.168.0.x/24 network, which he suggested for a reason (it's internationally accepted as a private address space and won't conflict with routes from your company to any other network in the world).

The 200.200.200.x/24 network is routable on the Internet and belongs to a private company, so may (although probably won't) cause issues if a service you want to use on one of those systems wants an address from that real network.

 

[WI_Hedgehog]

I missed the part where you said your company subnet is 10.219.130.###/23.

10.xxx.yyy.zzz is reserved for private networks as @sretalla mentions, you'll want to read up on that briefly. The rest of my and others advice should make sense then.

 

[mkaic]

Appreciate your patience. That makes sense! I changed everything to the following setup:

TrueNAS NIC: 192.168.0.1/24
PC NIC: 192.168.0.101/24

When I attempt to ping from the PC to TrueNAS, I get "From 192.168.0.101 ... Destination Host Unreachable".
When I attempt to ping from TrueNAS to the PC, I get "ping: sendto: Network is down"

So I'm still doing something wrong, I guess.

 

[Patrick M. Hausen]

OK, so it seems despite of what I wrote about factory new switches your interfaces are not active. So possible the switch does need some configuration to establish the links. Until you get the blinkenlights to light up, there is not much you can do about the IP configuration.

Can you check with the switch documentation or get some assistance from whoever sold it to you?

 

[mkaic]

OK, so it seems despite of what I wrote about factory new switches your interfaces are not active. So possible the switch does need some configuration to establish the links. Until you get the blinkenlights to light up, there is not much you can do about the IP configuration.

Can you check with the switch documentation or get some assistance from whoever sold it to you?

Yep, I should be able to look into it later today. MikroTik has good online manuals as far as I can tell so I'll give manual config a shot.

One thing to note is that I don't think the switch is factory-new, due to internal corporate bureaucracy around allowed vendors, I wasn't allowed to buy it direct from any of the common server part sellers. I don't have any reason to think it's defective, but I guess we'll see after I connect to it and try setting it up.

Thanks for all your help so far, will keep this thread updated when I have results.

 

[mkaic]

Update: By connecting directly to the switch with my laptop, I was able to upgrade its RouterOS and reboot it, after which blinkenlights appeared on the ends of the TrueNAS <-> Switch connection. Still no blinkenlights on either end of the PC <-> Switch connection, though.

Upon further investigation, I am suspicious that I do not have the correct firmware installed for the SFP+ NIC in my PC. It's a Lenovo card which (according to lspci) is based on the Aquantia AQC107. When I run ethtool -i on the interface, though, it tells me that the firmware version is N/A. At least now I have a different problem to look into, that's progress!

Will spend some time researching and see if I can't get the proper firmware for this NIC.

 

[Patrick M. Hausen]

Is this "PC" your client PC or your TrueNAS host? If the latter, please do not pursue installing vendor proprietary drivers but instead get a network card that is supported by FreeBSD and TrueNAS. On your Windows PC - of course, go ahead.

 

[mkaic]

Is this "PC" your client PC or your TrueNAS host? If the latter, please do not pursue installing vendor proprietary drivers but instead get a network card that is supported by FreeBSD and TrueNAS. On your Windows PC - of course, go ahead.

Sorry, should have kept using the names I established in the drawings at the beginning of the post. The PC I'm talking about is a Linux workstation, the one referred to as Bob on the diagram. I am aware that TrueNAS is to be considered an appliance and that one ought to avoid tinkering with any low-level system stuff on it if possible.

 

[WI_Hedgehog]

On a used switch (which seems to be a nice one at that) you can use your laptop to inspect the switch configuration and set the switch's IP and network settings. You should then be able to ping the switch's IP from Server-Alice.

Once you get PC-Bob sorted out you should then be able to ping the switch's IP from PC-Bob. Once that's working you can try pinging Server-Alice from PC-Bob and PC-Bob from Server-Alice.

Once the above works you can set up DHCP on the switch, excluding Alice's IP, then setting PC-Bob to use DHCP instead of having a static IP. The remaining PC units can then use DHCP.

I would think you want to avoid DNS on machines that have more than one path to Alice, that's a bit messy when you do not have full access to the network and probably do not want to impose network setting constraints on those who do.