10GBE Peer to Peer Connection (TrueNAS Scale - Windows10)

[DigitalMinimalist]

Dear all,

today I received my 2x 10GBE Intel X710-DA2 NICs and quickly installed them in my TrueNAS Scale Server and my Windows 10 workstation and connected both with a DAC / Twinax cable.
The NICs are visible in TrueNAS and also Windows 10 and I want to establish a 10GBE Peer to Peer connection. I read several forum posts, but I'm stuck.
Green lights are lid up at the NICs and Windows recognizes an "unidentified network" with 10GBE

I currently set-up the NICs like that:

Windows:
IP = 10.10.10.2
Gateway = 255.255.255.252
Jumbo Packets enabled

TrueNAS
I tried multiple options, e.g. static route, but none worked (had to reset the whole config, because I messed up the network config)

Pinging TrueNAS from Windows fails...

Any clues how to establish the peerto peer connection?

 

[danb35]

Any clues how to establish the peerto peer connection?

Configure each adapter with a distinct static IP address that isn't in any subnet you're already using for some other purpose. So, assuming you aren't using 10.10.10.0/24 anywhere else, you'd set the Windows machine to 10.10.10.2/24 (that means the subnet mask would be 255.255.255.0), and the NAS to, say, 10.10.10.3/24. That's really it.

The key is that the IPs on this adapter not be part of any other extant subnet. So if you're already using 10.10.10.0/24 for other devices, use a different subnet for this connection--perhaps 10.10.20.2/24 for the Windows machine, and 10.10.20.3/24 for the NAS.

 

[DigitalMinimalist]

Thank you for your quick reply

Windows:
IP = 10.10.10.2
Gateway = 255.255.255.0
Jumbo Packets enabled (9014)

TrueNAS:

enp1s0f0

Alias IP: 10.10.10.1/24

MTU 9014
Media Subtype is correctly recognized: 10000Mb/s Direct Attach Copper
DHCP & Autoconfigure IPv6 disabled



I probably should just get a SFP+ 10GBE Switch...

 

[danb35]

I probably should just get a SFP+ 10GBE Switch...

Probably so (and the little Mikrotik units are pretty attractive for that), but what you've described should work. Can you ping one machine from the other?

 

[DigitalMinimalist]

If the little Mikrotik boxes would be available (in Europe)…

I anyhow plan to buy the TP-Link TL-SG3210XHP-M2 to feed the future Wifi6 APs with its 2.5G PoE and it has at least 2x SFP+ (40mm Noctua fans are already here to mod it).
I just want to test different vDev setups to define the final layout

 

[phoenix]

I should think that the Mikrotik 'boxes' would be available in Europe as they're available in the UK. I have the MikroTik CRS309-1G-8S+IN on my network and they work a treat.

 

[DigitalMinimalist]

Already offtopic... :)
I was looking into the Mikrotik CSS610-8G-2S+ as a cheap switch until I convert to Wifi 6 - but not available and I would like to go mainly Wifi in my new house - therefore Wifi 6 for most devices and workstation and Server connected with 10GBE

 

[phoenix]

If you're in the UK then there's plenty of them available. A quick search for "Mikrotik CSS610-8G-2S+ UK" finds them at varying prices and from reputable distributors and specialist suppliers.

 

[danb35]

Back to the original issue, with the configuration you've now described, can you ping between the two machines?

 

[DigitalMinimalist]

an you ping between the two machines?

No - 100% packet losses
Not sure what elese I can try

I also added the server in the "host" file on Windows as many guides are describing

If you're in the UK then there's plenty of them available.

I'm located in Portugal - just found one shop who declares to have the switch on stock - just ordered one...

 

[danb35]

I also added the server in the "host" file on Windows as many guides are describing

That shouldn't have any effect on this. Is anything else using a 10.10.10.x IP?

 

[DigitalMinimalist]

That shouldn't have any effect on this. Is anything else using a 10.10.10.x IP?

No - for sure not. All my other devices are in the 192.168.8.x range

 

[Etorix]

If the little Mikrotik boxes would be available (in Europe)…

Mikrotik is a European company… (From Riga, Latvia.) They definitely sell in their home market.

 

[DigitalMinimalist]

Mikrotik is a European company… (From Riga, Latvia.) They definitely sell in their home market.

True, but they seem to have severe supply issues right now and half of their products are out of stock…

The CSS610 was anyhow on my list, as I will use it even with the TP-Link as a switch for my home video area later

 

[danb35]

All my other devices are in the 192.168.8.x range

OK, that shouldn't be a problem. I wonder if there's a problem with the DAC--either that the cable itself (or one or both of the SFP+ units) is damaged, or that it isn't recognized by the NICs. The latter is a somewhat-common problem with SFP+ NICs, that they can be vendor-locked to only work with optics from the NIC vendor. But I don't know how you'd check that, other than using known-compatible SFP+ optics and a fiber patch cable. @jgreco, any ideas on how you'd check for that (or if I'm completely off-base here)?

 

[DigitalMinimalist]

OK, that shouldn't be a problem. I wonder if there's a problem with the DAC--either that the cable itself (or one or both of the SFP+ units) is damaged, or that it isn't recognized by the NICs. The latter is a somewhat-common problem with SFP+ NICs, that they can be vendor-locked to only work with optics from the NIC vendor. But I don't know how you'd check that, other than using known-compatible SFP+ optics and a fiber patch cable. @jgreco, any ideas on how you'd check for that (or if I'm completely off-base here)?

That could be an issue for sure:

Both X710-DA2 are actually Dell H54810, which I bought used, but look like new. DAC cables are from Draytek which I bought new.
Let’s see if some firmware flashing is required or not…

 

[danb35]

Let’s see if some firmware flashing is required or not

I wouldn't expect so, though I seem to recall seeing something on the STH forums about flashing certain NICs to remove vendor locks. The output of ifconfig enp1s0f0 might help see what's happening.

I wonder if removing the jumbo frame configuration might help.

 

[jgreco]

But I don't know how you'd check that, other than using known-compatible SFP+ optics and a fiber patch cable. @jgreco, any ideas on how you'd check for that (or if I'm completely off-base here)?

My classic test, which assumes at least a dual port card, is to plug both ends of the cable into the card and then check for link. Or at least plug one end into a device known to work with the DAC or optics. This is why I favor buying all the optics for a switch or interface card at the time of purchase and then just never having to worry about it again.

Both X710-DA2 are actually Dell H54810, which I bought used, but look like new.

Dell spares sometimes end up on the used market.

As a fan of the Supermicro AOC-STG-I4S, which is one of the few quad SFP+ half height cards on the market, I will note that the "newer" Intel cards like the X710's have rather complicated onboard firmware which can affect the operation of the card. For example, the well known "X710 intercepts and handles LLDP on the card" issue, or the more obscure issues such as

https://kb.vmware.com/s/article/53749

More disconcertingly, the firmware for the -I4S isn't available from Intel, and Supermicro only makes it available by special request. Also a PITA to update.

I have not seen the Dell -DA2 variant of this card, but I assume that there may be Dell-specific firmware for it. I found the VMware Hardware Compatibility Guide to be very helpful in helping me sort this issue out at the time I ran into it.

 

[DigitalMinimalist]

After some re-installation of the X710 Intel (Pro) drivers...it works!!!!!
And I can start testing different vDev configurations :)

Workstation HDD to TrueNAS SSD RAIDZ2: 180MB/S
Workstation SSD to TrueNAS SSD RAIDZ2: 500 MB/S
Workstation NVME to TrueNAS SSD RAIDZ2: 600 MB/S

Workstation SSD to TrueNAS HDD (Striped Mirror + NVME special vDev for metadata): 400 MB/S
Workstation NVME to TrueNAS HDD (Striped Mirror + NVME special vDev for metadata): 400 MB/S

Thank you for all the help!!!!!!!

 

[jgreco]

After some re-installation of the X710 Intel (Pro) drivers...it works!!!!!

Yes, once you get it to the point where it works, they're very nice cards. I just wish it was a little less work getting there.