Unable to set network settings in Console, no LAG being used.

[jgreco]

That's incorrect. The first interface is a static IP, but I'm unable to set the additional 10GbE NIC as a static IP in the GUI due to a bug with TrueNAS. I tried in the Console, but that didn't work either. Got any ideas?

What bug would that be?

 

[superpos]

What bug would that be?

It may just be a UX bug, where underneath the issue is that the NIC is not on a different subnet. But as reported by others on the forum the alias and MTU won't stick for a second NIC's network interface settings (Network -> Interfaces). And so far, I have not been able to fix this within the Console. Hence why I'm looking at creating a subnet via a VLAN so that the second NIC can live on that with the hope that it conforms to what the docs are saying.

To reiterate, the Network -> Interfaces GUI does work for the first 1G connection in the list - I am able to set a static IP via an alias. I am just stuck in bootstrapping mode for the second NIC - that's the only reason I'm using DHCP while I'm in this limbo.

 

[jgreco]

I would think we'd be seeing a large number of people reporting issues if multiple interfaces couldn't be successfully configured. Where does an alias come into play? I haven't seen any problems myself, but I also don't do that much with SCALE. My test box has four interfaces and when playing with it seemed to work as expected.

 

[jgreco]

I just spent some time playing with it. I think you may need to be real careful to either DELETE unused interfaces or be sure to ASSIGN a static IP to each interface listed in the configuration. I did get some weird stuff happening if I didn't do one or the other, but that's really a PEBCAK issue. The SCALE UI seems to have successfully managed a trick I had been suggesting for a long time, which is to define the entire net config and then commit it all at once.

 

[superpos]

I would think we'd be seeing a large number of people reporting issues if multiple interfaces couldn't be successfully configured. Where does an alias come into play? I haven't seen any problems myself, but I also don't do that much with SCALE. My test box has four interfaces and when playing with it seemed to work as expected.

I'm referencing that I searched and was seeing a few people creating threads here reporting the UX issue including with screencaps. Forgive me if I'm using the incorrect terminology, I'm referring to the fact that TrueNAS calls it an "alias" when you need to define a static IP and it's just not sticking when I set it. However, the issue per the docs seems likely that I need to set a separate subnet for the separate NIC.

Or, I can look at getting rid of the current 1G NIC and relying on the 10G one if that means not needing to worry about the subnet clash. I would still have IPMI for maintenance, as long as that does not itself cause a subnet problem.

So you are not using SCALE. Should I switch to CORE? I was interested to use Tailscale and maybe MinIO, but it's not essential. I'm already able to access the device via Tailscale from my Mac Mini. I also felt more comfortable with Debian underneath since I have a little bit of Linux shell familiarity.

 

[jgreco]

I'm referencing that I searched and was seeing a few people creating threads here reporting the UX issue including with screencaps. Forgive me if I'm using the incorrect terminology, I'm referring to the fact that TrueNAS calls it an "alias" when you need to define a static IP and it's just not sticking when I set it. However, the issue per the docs seems likely that I need to set a separate subnet for the separate NIC.

I have no idea why they are calling it an "alias". It's not, that's idiotic, it's the primary address for the interface if it is the first one defined. This is what happens when UI designers are allowed to code for infrastructure. You know the old saying about "beware of programmers carrying screwdrivers", there should be some analog for this for programmers writing systems code or something along those lines.

You do need a separate subnet for every IPv4/IPv6 interface. This applies to the logical interface. It's fine to have a logical interface composed of two or four physical interfaces in LACP, in which case the LACP interface is the one that gets the IP addresses. If you just have physical interfaces and no software-defined logical interfaces, then the IP addresses go there.

Or, I can look at getting rid of the current 1G NIC and relying on the 10G one if that means not needing to worry about the subnet clash. I would still have IPMI for maintenance, as long as that does not itself cause a subnet problem.

You have a lot of potential options. It depends on what you want for your network topology. If you would prefer a single layer 2 network (single LAN), you could hook up your firewall to your CRS328, hook up the CRS328 and CRS305 with a 1G or 10G link (whichever works for you) and then hook up the TrueNAS to one of the other available 10G ports, which could really be on either switch. Bring up your network on the TrueNAS 10G interface and bam, call it a day, done. Honestly I don't understand all this other complexity you seem to be adding. I strongly prefer to recommend flat layer 2 networks for users when they don't seem to have a strong networking background. It's easier on everyone.

So you are not using SCALE. Should I switch to CORE?

Well, the example I showed is definitely SCALE, Linux and its doofy "ens" ethernet interfaces. You can make the networking work under either SCALE or CORE and the rules are basically the same, though the GUI's are different. CORE is a much more mature and stable platform (by about ten years) and then you also don't have to deal with stuff like the ZFS ARC sizing issue and other Linux downsides. But it's basically a personal choice. If you can swing doubling your RAM on SCALE then that's a large equalizer; with that I think it makes sense to go with CORE if you need high performance for stuff like iSCSI or SCALE if you need Kube and scale-out filesystem stuff.

 

[superpos]

I just spent some time playing with it. I think you may need to be real careful to either DELETE unused interfaces or be sure to ASSIGN a static IP to each interface listed in the configuration. I did get some weird stuff happening if I didn't do one or the other, but that's really a PEBCAK issue. The SCALE UI seems to have successfully managed a trick I had been suggesting for a long time, which is to define the entire net config and then commit it all at once.

View attachment 67775

I have tried deleting unused interfaces, but they don't disappear even after confirming and saving. I think it's because I have 4x1GbE on the motherboard. So three are going unused. At one point I might have tested them in 4x link aggregation, very early on, but I deleted all references to that setup.

I did just notice that for some reason the first NIC that I (thought) I had assigned as a static IP was on DHCP. There was an alias defined, and I thought that meant it had a static IP. So I just unchecked DHCP and confirmed/saved that change, and it stuck. I'll monitor it just to make sure it doesn't somehow change. I've double checked the settings so many times, but maybe I just missed something.

There has been some progress in the right direction with the 10GbE NIC: I've been able to set up a VLAN, and although nothing is working right yet - probably because I can't log on to the 10GbE switch - the switch did pass through a Macbook connected to it which showed up as a device.

I just used the same VLAN subnet to define a static IP for the 10GbE NIC in the SCALE network interface and it has actually stuck. I'm not sure yet if that means it's actually using the VLAN or not, until I get the switch working again. I'll take a look tomorrow! It's not currently showing up as a device on the VLAN, as far as the Firewall sees it - only the Macbook appeared so far.

Thanks for your patience with answer suggestions. This has been super frustrating for me to work out. I just used the 1GbE connection in production for the last eight months in the same way as I traditionally have with my older Synology, and cached locally where needed, but have known I need to take another look at this.

 

[superpos]

I have no idea why they are calling it an "alias". It's not, that's idiotic, it's the primary address for the interface if it is the first one defined. This is what happens when UI designers are allowed to code for infrastructure. You know the old saying about "beware of programmers carrying screwdrivers", there should be some analog for this for programmers writing systems code or something along those lines.

You do need a separate subnet for every IPv4/IPv6 interface. This applies to the logical interface. It's fine to have a logical interface composed of two or four physical interfaces in LACP, in which case the LACP interface is the one that gets the IP addresses. If you just have physical interfaces and no software-defined logical interfaces, then the IP addresses go there.



You have a lot of potential options. It depends on what you want for your network topology. If you would prefer a single layer 2 network (single LAN), you could hook up your firewall to your CRS328, hook up the CRS328 and CRS305 with a 1G or 10G link (whichever works for you) and then hook up the TrueNAS to one of the other available 10G ports, which could really be on either switch. Bring up your network on the TrueNAS 10G interface and bam, call it a day, done. Honestly I don't understand all this other complexity you seem to be adding. I strongly prefer to recommend flat layer 2 networks for users when they don't seem to have a strong networking background. It's easier on everyone.



Well, the example I showed is definitely SCALE, Linux and its doofy "ens" ethernet interfaces. You can make the networking work under either SCALE or CORE and the rules are basically the same, though the GUI's are different. CORE is a much more mature and stable platform (by about ten years) and then you also don't have to deal with stuff like the ZFS ARC sizing issue and other Linux downsides. But it's basically a personal choice. If you can swing doubling your RAM on SCALE then that's a large equalizer; with that I think it makes sense to go with CORE if you need high performance for stuff like iSCSI or SCALE if you need Kube and scale-out filesystem stuff.

Thanks, more food for thought. I'll digest this tomorrow. I definitely do not have grandiose network ambitions (or knowledge) and it's more a thing of "really? I have to add a VLAN?". That is needed though though, if I need to create subnets and stay within the firewall.

I can also test a more simple scenario with the 2x10GbE ports directly on the CS326 to try and trouble shoot this. I actually thought it would be easier to have the 10G devices on their own subnet/switch though.

 

[jgreco]

but I deleted all references to that setup.

I'm guessing you didn't. That's not even to say you're to blame. I am quite happy to say that perhaps something stuck around in the middleware and the GUI isn't handling it properly. I might even consider that to be more likely than an error on your part, given the complexity of the GUI and the number of times I've observed that the developers haven't properly trapped certain errors. I give them credit for trying though. But it also means we have to remember the possibility that things in the NAS might be mucked up in some nonobvious way.

Thanks for your patience with answer suggestions. This has been super frustrating for me to work out.

It's fine. We all went through it at some point. For me, it was back in the '80's when you read what few books were available, source code when you could, and posted on Usenet when you were in despair, waiting several days for any responses. The complexity of modern stuff is far worse. There are lots of people here who can be very useful resources if you take your time and have patience. Everyone has certain things that they're better at, and some that they're not. Don't ask me SMB or Windows questions! Heh.

 

[superpos]

I wanted to follow up on this since I made some progress. I was able to delete the 1 gigabit connection in the console, which then allowed me to configure the 10 gigabit NIC and get the kind of performace I would expect, which felt like a big deal after so long. I hate to say this, but in the GUI the fact there was an alias assigned on the NIC made me think I had configured it as static IP (I thought that's what an alias *meant*). But when I clicked through, it was still DHCP. I had tried a few times to set it to static IP, but for whatever reason it wasn't sticking and I needed to nuke it from the console - wish I'd done that months ago! I thought I was diligent about clicking the multiple confirmations when changing settings, but who knows?

I did attempt to implement a VLAN on the firewall and switch for a separate subnet that would allow preserving the 1 gigabit connection, but I did not get it working smoothly at all. So I'll come back to that some other time.

I'm also temporarily just using the 2x 10 gigabit ports directly on the main Mikrotik CRS328 switch, but will add back in the 4x 10gigabit switch now that the main issue is solved.

After some tuning, using NFS I'm getting a ~850/950 megabytes/sec read/write on the Mac, and ~1 gigabyte/sec read on Linux (need to test write with fio, expecting the same-ish). This is with a combo of fio tests on both, and Blackmagic Disk Speed test on the Mac. The fio test file size was larger than my 128GB RAM so I don't think ARC caching would skew results, but I do also have 2x1TB striped NVMe for L2 ARC. Basically, speeds are in line with what I would have hoped from a single 10 gigabit NIC.

I have managed to mess up my permissions though, but that is going to have to be a separate thread if I can't figure it out. I wanted to see if I could use the WiFi bridge connection included with the firewall to connect to the NAS via NFS without needing the wired 10G connection, if I just wanted access without concern for performance. However, connecting from the Mac wirelessly didn't work out the box, but after some ACL shenanigans it was working - only for me to realize I could no longer modify files. So considering I'm the only user, I'm just going to figure out how to reset/create new users, shares, and ACLs. To be honest, I kind of ignored ACLs because I thought they were not an issue for NFS as in my mind every client was root.

Finally, I think I made a mistake when I initially was setting Scale up by sharing the root level of my pool as my NFS share. So when I mount it on a client, the ix-applications folder is also shared. I did not read that this was a no-no for NFS, but that it is for SMB. However I notice that the share permissions at pool root level are all very stripped down and are basically root, so I think I need to shift the NFS share to a child folder where I get the full edit options.

Overall, I only have one project on the NAS right now and I'm the only user, so it will not break too much if I end up needing to change what is currently my only share mount path.