FreeNAS networking, 'interfaces', MTU setting, fixed IP and reliability.

[diskdiddler]

Hi,

I'm utterly baffled, again!
I continue to test my new server in a test environment.

My normal machine, is available on 192.168.0.9. There is NOT an entry in the 'interfaces' section.
On the new machine, I wanted to set MTU to 9000 (Mellanox ConnectX-2) and set a fixed IP also.
I can't find how to do this, without having to add an entry in the interfaces section, so be it..?
First time I did this, the network was entirely dead, period, couldn't ping it. I even reset network settings in the direct console, nope. Had to do a full wipe of ALL settings.

Second time I did it, exactly the same mind you,... it worked and I can change IP multiple times even, it boots up fine with whatever I set in interfaces.
However while the laptop on wifi can interact fine with the server, my desktop (the only other jumbo frame device) does not like talking to the server. It can ping it, but the webUI isn't found and the samba share kind of locks up explorer when opening it.
So, I rebooted, changed ip, now the Desktop can see the WebUI (kinda, it half loads then stops) and the samba share I can now go into but it's inconsistent and I can't copy from it, just kinda locking up.


It's the Mellanox Connect-X2 in both machines. I set the mtu in FreeNAS simply with "mtu 9000" in the options section.
In Windows I changed "jumbo packet" in advanced adapter properties from 1514 (??! not 1500) to "9000"

Final piece of information.
If I disable jumbo frames on EITHER machine the problem goes away (Win10 Desktop, Mellanox OR FreeNAS Mellanox)

Could it be my switch? This is it, I know NOTHING about managing switches and I'm running it as stock config.
https://www.tp-link.com/us/products/details/cat-40_T1700G-28TQ.html


Thanks, very sorry for long post, hope someone else has ideas.
P.S I'm trying jumbo frames as I'm only getting 200MB/s from the server.

 

[Elliot Dierksen]

Could it be my switch? This is it, I know NOTHING about managing switches and I'm running it as stock config.

If you haven't enabled jumbo frames in the switch, then that could very well be it. Every hop along the path has to support jumbo frames, or the large packets drop on the floor. You could try a direct connect between hosts to prove that out, but I haven't seen anything that that comes out of the box with jumbo frame support turned on by default. Disclaimer - Most of my 10G experience is with Cisco & HP.

 

[diskdiddler]

That reply is appreciated - presumably, with fibre, I don't need a 'crossover' cable? I can just fibre to fibre?
It would certainly be a straightforward test to eliminate the switch. Although one would think the thing is in jumbo mode as default for the SFP+ ports.

 

[Elliot Dierksen]

I can just fibre to fibre?

Yes, that would be the way to go. You might have to switch the tx/rx if the link doesn't come up right away. That is standard stuff with fiber.

Although one would think the thing is in jumbo mode as default for the SFP+ ports.

The port is a layer 1 thing. The MTU is a layer 2 thing. None of the 10G gear I have touched comes out of the box with jumbo frames on.

 

[diskdiddler]

How does one switch the TX and rx?

I found some articles, indicating it might be in automatically with TP-Link gear, but I'll test in the morning to eliminate that variable.

There's firmware updates out there for it, so maybe even that.

 

[diskdiddler]

Well I got a chance, yeah it was jumbo frames on the switch, it appears to be fixed. Although still only 200MB/s. Next step is an iSCSI test to see if SMB is holding me back.

 

[Elliot Dierksen]

How does one switch the TX and rx?

By transposing the the connectors on the fiber patch cord. Sounds like that wasn't an issue for you which is good.

Well I got a chance, yeah it was jumbo frames on the switch, it appears to be fixed. Although still only 200MB/s. Next step is an iSCSI test to see if SMB is holding me back.

That is what I expected on both counts. You can get some marginal gains from jumbo frames, but they only seem to be a benefit in some very specific use cases. Now you are more likely into things like pool/vdev construction, how much RAM do you have, disk/controller speed.

 

[kdragon75]

That reply is appreciated - presumably, with fibre, I don't need a 'crossover' cable? I can just fibre to fibre?

In fiber ints a "rolled" cable. Each cable contains two fibers and each fiber is one direction only. Therefore A and B get rolled to B and A on the other side. If one end is wrong, it just plain will not see the connection. If both sides are wrong, it's right! With fiber and most gear, there is no need to differentiate straight through and rollover as all cables should be rolled over. It only matters if you are using a coupler like from a "patch panel" sometimes referred to as an LIU (light interface unit) to a wall plate. Then the cable from the LIU to the wall plate should be a non rolled A to A and B to B cable as the jumper from the wall to the device should be rolled.

How does one switch the TX and rx?

Most gear nowadays uses LC connecters. The two fibers are usually clipped together with a small plastic part, just pop the end out of the clip and back in on the other side.

Although one would think the thing is in jumbo mode as default for the SFP+ ports.

While jumbo frames are standard-ish, they are not the ethernet spec and therefore not enabled by default to maintain compatibility. If not all devices can handle jumbo frames, they need to get broken up into smaller frames or in some cases they will just get dropped by the receiving end as garbage.

Well I got a chance, yeah it was jumbo frames on the switch, it appears to be fixed. Although still only 200MB/s. Next step is an iSCSI test to see if SMB is holding me back.

Take a look at iperf for testing throughput. This will rule out anything storage related.

That is what I expected on both counts. You can get some marginal gains from jumbo frames, but they only seem to be a benefit in some very specific use cases. Now you are more likely into things like pool/vdev construction, how much RAM do you have, disk/controller speed.

Yeah, I have a set of intel X520 cards with a NetApp 10gbe switch, one host is DAC, the other is fiber, and I can read/write well over 2GB/s to a ram disk (two path MPIO with round robin). Thats with a xeon X5650 clocked down to 2.2GHz and 1500 byte MTUs

 

[diskdiddler]

This is as per my signature, (test machine) reading or writing from the SSD pool.

Destination is an Intel 8700 with an mx300 SSD. I'd expect at least 100MB more, maybe 200.

I'll try iperf, I've only used it once before, I recall it being kinda difficult to set up our find the exact right command. Considering what it actually does.

 

[Elliot Dierksen]

I'll try iperf, I've only used it once before, I recall it being kinda difficult to set up our find the exact right command.

It isn't that hard. One side is the server iperf -s and the other side is the client. You tell it to be a client and what that IP of the server is iperf -c <server-ip>. That should give you the basics of it.

 

[diskdiddler]

It isn't that hard. One side is the server iperf -s and the other side is the client. You tell it to be a client and what that IP of the server is iperf -c <server-IP>. That should give you the basics of it.

Ok yeah I am a bit dumb! Although I did have to find iperf2 binaries for Windows not iperf3.
Confirmed though, link is fine, 8.9Gbit, so time to try iSCSI to mess around and experiment.

You'd think mirror SSD, to destination SSD might be faster.
So it's either SMB or the disks. I am ok with 200MB/s but I'd love a few hundred more.