10Gbe woes - and upgrade time!

[jgreco]

The interesting findings with that are that the FreeBSD driver is fine with it and I can use any brand/type of module, but the Linux and ESXi drivers both detect the EEPROM shenanigans and won't load the interface.

It's pretty obvious that the ESXi driver won't work since it's a precompiled and signed driver. but are you sure the Linux driver doesn't work with it? Since the Linux driver can be compiled from source, I would have expected that some desperate Linux folks would have adopted whatever hackaround worked on FreeBSD. I have not tried this specific thing because, well, I guess I don't really care as it is so much easier just to buy some cheap $10 Intel optics and have it work reliably. However, I did recently (possibly prompted by this thread) reorganize the 10 Gig Networking Primer bits on vendor locking for these cards to make it clearer to newbies.

 

[sretalla]

It's pretty obvious that the ESXi driver won't work since it's a precompiled and signed driver. but are you sure the Linux driver doesn't work with it? Since the Linux driver can be compiled from source, I would have expected that some desperate Linux folks would have adopted whatever hackaround worked on FreeBSD.

You're probably right. I lost interest after one attempt where I found the function which checks the EEPROM in the code and set it to always return OK, but that didn't help.

I moved on and found another solution not relying on that card.

 

[Matthias]

I have similar problem with Mikrotik SFP switch - the link is not established independent of modules that I try on the switch.
I found that if I manually select network speed (not use auto negotiation) - for example 1GB in Mikrotik switch, the link is established and working without other issues.
If you have time for tests you can try to set manually 10GB for NAS SFP port on Mikrotik switch and try again different SFP modules combinations.

I can select the speeds when i use the working combo, but if i insert any other module, the dropdown dissapears and i can't de-select auto negotiation.

Will investigate further tonight, did a quick check now during lunch break

Are you running RouterOS or SwitchOS?
(SwitchOS here)

 

[Matthias]

It's pretty obvious that the ESXi driver won't work since it's a precompiled and signed driver. but are you sure the Linux driver doesn't work with it? Since the Linux driver can be compiled from source, I would have expected that some desperate Linux folks would have adopted whatever hackaround worked on FreeBSD. I have not tried this specific thing because, well, I guess I don't really care as it is so much easier just to buy some cheap $10 Intel optics and have it work reliably. However, I did recently (possibly prompted by this thread) reorganize the 10 Gig Networking Primer bits on vendor locking for these cards to make it clearer to newbies.

All current and new padawans are in your debt master jedi!

Might i make a suggestion for the page? If so, i'd just also mention the 'up' procedure.
So when changing any module, you'd better run the command 'ifconfig ix1 up' for example. + The screenshot from @Mlovelace somewhere above.

And maybe a short checklist:
10Gbe speed wanted?

1: pick one of these vendors for the card (insert vendor in preferred order)
2: pick the preferred modules, but is tricky as in my case..
3: insert hardware and check. Truenas should detect and be able to config.

Maybe keeping a matrix is a good idea? (Top of the page )
Because with my post i can recommend fs optics for Mikrotik in combo with Uptimed card and modules (intel x520)

If you encounter problems ("no link/no carrier")
see:
- Ifconfig -v ix1: check the trancievers
- ifconfig ix1: check the connection
- You may try setting a sysctl tunable for "hw.ix.unsupported_sfp" to "1"
- ifconfig ix1 up
- check fiber connection, hardware...

Thank you!

 

[jgreco]

All current and new padawans are in your debt master jedi!

Might i make a suggestion for the page? If so, i'd just also mention the 'up' procedure.
So when changing any module, you'd better run the command 'ifconfig ix1 up' for example. + The screenshot from @Mlovelace somewhere above.

And maybe a short checklist:
10Gbe speed wanted?

1: pick one of these vendors for the card (insert vendor in preferred order)
2: pick the preferred modules, but is tricky as in my case..
3: insert hardware and check. Truenas should detect and be able to config.

Maybe keeping a matrix is a good idea? (Top of the page )
Because with my post i can recommend fs optics for Mikrotik in combo with Uptimed card and modules (intel x520)

If you encounter problems ("no link/no carrier")
see:
- Ifconfig -v ix1: check the trancievers
- ifconfig ix1: check the connection
- You may try setting a sysctl tunable for "hw.ix.unsupported_sfp" to "1"
- ifconfig ix1 up
- check fiber connection, hardware...

Thank you!

Ugh, more work.

 

[sretalla]

Not sure if you saw the comment in the report you made for yourself to come back to it... maybe somebody can offer feedback or more help in addition to the start I made:

Card	Module Recommendations	Support/Tested On	Notes	Useful links/posts
Chelsio T3, T4 and T5 ASICs	??	CORE, SCALE	Supported/chosen by iX. The best option	Something.
Intel x520 / x540	??	CORE, SCALE	Known to work well despite being PCIe v2
Intel x710	Intel-compatible SFP's are required	CORE	Can be hacked to work with non-Intel modules, but not recommended to do so
SolarFlare SFN5162F/SFN6122F	Documentation from SolarFlare indicates tested modules, but says all should work	ESXi, CORE, SCALE	Tested and works well with ESXi
IBM NetXTREME II 10GBit (Broadcom BCM 57710)	??	Works in FreeNAS 9.3
Emulex "oce" cards	N/A	N/A	Beware these cards/Avoid
Mellanox ConnectX-2 cards	N/A	N/A	Avoid, regardless of low price

 

[Matthias]

Ugh, more work.

If you like, i'll re-write my comments to match your page more. So jedi master can copy paste.

Is the least i can do for the forum and all the help i get!

 

[Matthias]

Not sure if you saw the comment in the report you made for yourself to come back to it... maybe somebody can offer feedback or more help in addition to the start I made:

Card	Module Recommendations	Support/Tested On	Notes
Chelsio T3, T4 and T5 ASICs	??	CORE, SCALE	Supported/chosen by iX. The best option
Intel x520 / x540	??	CORE, SCALE	Known to work well despite being PCIe v2
Intel x710	Intel-compatible SFP's are required	CORE	Can be hacked to work with non-Intel modules, but not recommended to do so
SolarFlare SFN5162F/SFN6122F	Documentation from SolarFlare indicates tested modules, but says all should work	ESXi, CORE, SCALE	Tested and works well with ESXi
IBM NetXTREME II 10GBit (Broadcom BCM 57710)	??	Works in FreeNAS 9.3
Emulex "oce" cards	N/A	N/A	Beware these cards/Avoid
Mellanox ConnectX-2 cards	N/A	N/A	Avoid, regardless of low price

I think i'm missing something?
Don't quite follow, sorry.

As for the matrix you mention, indeed something like this could be helpful for everyone.

 

[sretalla]

I think i'm missing something?
Don't quite follow, sorry.

That note was to @jgreco, who made a report on his own post so he would remember to come back to it (only moderators can see that... so I probably shouldn't have mentioned it... apologies for the confusion)

 

[Matthias]

That note was to @jgreco, who made a report on his own post so he would remember to come back to it (only moderators can see that... so I probably shouldn't have mentioned it... apologies for the confusion)

No problem!

Edit: and i see your point. Maybe the solution is a read only matrix for us padawans, and just writable for the mods. In that way, you can eliminate 'false' inputs, or add a column 'tested by' (ix-Solutions, Jedi masters, padawans, klingons, ...). Then @jgreco 's strain in his quest for maintaining that thread/matrix might just be a little lighter or easier.

If an interesting thread comes across, might be useful to paste that in too.

 

[Matthias]

Testing will be for the weekend.. far to tired today!

I did test the speed, and a question rose:

When copying data from a 980pro Samsung SSD on my desktop, i get these results:

Could that be fixed with the 'offload hardware' tick in the settings on the NIC interface?

Or should i too, beware...

 

[sretalla]

Have you run iperf across that same path?

Measuring multiple factors including SMB in the chain may not be helpful in determining if the problem is with the NIC.

 

[sretalla]

If an interesting thread comes across, might be useful to paste that in too.

Good idea, updated my table to include a links column

 

[nKk]

I can select the speeds when i use the working combo, but if i insert any other module, the dropdown dissapears and i can't de-select auto negotiation.

Will investigate further tonight, did a quick check now during lunch break

Are you running RouterOS or SwitchOS?
(SwitchOS here)

If I remember correctly it was SwitchOS.
Currently I don't have access to the Mikrotik and can't check.

 

[Matthias]

Have you run iperf across that same path?

Measuring multiple factors including SMB in the chain may not be helpful in determining if the problem is with the NIC.

ran a couple of tests:

Code:

Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51012 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   738 MBytes  6.19 Gbits/sec                 
[  4]   1.00-2.00   sec   744 MBytes  6.24 Gbits/sec                 
[  4]   2.00-3.00   sec   753 MBytes  6.32 Gbits/sec                 
[  4]   3.00-4.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   4.00-5.00   sec   753 MBytes  6.31 Gbits/sec                 
[  4]   5.00-6.00   sec   750 MBytes  6.29 Gbits/sec                 
[  4]   6.00-7.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   7.00-8.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   8.00-9.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   9.00-10.00  sec   752 MBytes  6.31 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  7.32 GBytes  6.29 Gbits/sec                  sender
[  4]   0.00-10.00  sec  7.32 GBytes  6.29 Gbits/sec                  receiver

iperf Done.

Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51014 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   746 MBytes  6.26 Gbits/sec                 
[  4]   1.00-2.00   sec   384 MBytes  3.22 Gbits/sec                 
[  4]   2.00-3.00   sec   374 MBytes  3.14 Gbits/sec                 
[  4]   3.00-4.00   sec   376 MBytes  3.15 Gbits/sec                 
[  4]   4.00-5.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   5.00-6.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   6.00-7.00   sec   374 MBytes  3.14 Gbits/sec                 
[  4]   7.00-8.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   8.00-9.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   9.00-10.00  sec   695 MBytes  5.83 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.36 GBytes  3.74 Gbits/sec                  sender
[  4]   0.00-10.00  sec  4.36 GBytes  3.74 Gbits/sec                  receiver

iperf Done.


Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51016 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   723 MBytes  6.06 Gbits/sec                 
[  4]   1.00-2.00   sec   751 MBytes  6.30 Gbits/sec                 
[  4]   2.00-3.00   sec   751 MBytes  6.30 Gbits/sec                 
[  4]   3.00-4.00   sec   745 MBytes  6.25 Gbits/sec                 
[  4]   4.00-5.00   sec   751 MBytes  6.30 Gbits/sec                 
[  4]   5.00-6.00   sec   750 MBytes  6.29 Gbits/sec                 
[  4]   6.00-7.00   sec   752 MBytes  6.30 Gbits/sec                 
[  4]   7.00-8.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   8.00-9.00   sec   751 MBytes  6.30 Gbits/sec                 
[  4]   9.00-10.00  sec   752 MBytes  6.30 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  7.30 GBytes  6.27 Gbits/sec                  sender
[  4]   0.00-10.00  sec  7.30 GBytes  6.27 Gbits/sec                  receiver

iperf Done.

rebooted the switch and ran some more tests:

Code:

Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51018 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   376 MBytes  3.15 Gbits/sec                 
[  4]   1.00-2.00   sec   378 MBytes  3.18 Gbits/sec                 
[  4]   2.00-3.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   3.00-4.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec                 
[  4]   5.00-6.00   sec   364 MBytes  3.05 Gbits/sec                 
[  4]   6.00-7.00   sec   377 MBytes  3.16 Gbits/sec                 
[  4]   7.00-8.00   sec   372 MBytes  3.12 Gbits/sec                 
[  4]   8.00-9.00   sec   375 MBytes  3.14 Gbits/sec                 
[  4]   9.00-10.00  sec   369 MBytes  3.10 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.65 GBytes  3.14 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.65 GBytes  3.14 Gbits/sec                  receiver

iperf Done.


Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51076 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   421 MBytes  3.53 Gbits/sec                 
[  4]   1.00-2.00   sec   728 MBytes  6.11 Gbits/sec                 
[  4]   2.00-3.00   sec   725 MBytes  6.08 Gbits/sec                 
[  4]   3.00-4.00   sec   710 MBytes  5.96 Gbits/sec                 
[  4]   4.00-5.00   sec   734 MBytes  6.15 Gbits/sec                 
[  4]   5.00-6.00   sec   729 MBytes  6.11 Gbits/sec                 
[  4]   6.00-7.00   sec   744 MBytes  6.24 Gbits/sec                 
[  4]   7.00-8.00   sec   722 MBytes  6.06 Gbits/sec                 
[  4]   8.00-9.00   sec   742 MBytes  6.22 Gbits/sec                 
[  4]   9.00-10.00  sec   733 MBytes  6.15 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  6.82 GBytes  5.86 Gbits/sec                  sender
[  4]   0.00-10.00  sec  6.82 GBytes  5.86 Gbits/sec                  receiver

iperf Done.


Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51083 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   727 MBytes  6.10 Gbits/sec                 
[  4]   1.00-2.00   sec   686 MBytes  5.76 Gbits/sec                 
[  4]   2.00-3.00   sec   640 MBytes  5.37 Gbits/sec                 
[  4]   3.00-4.00   sec   367 MBytes  3.08 Gbits/sec                 
[  4]   4.00-5.00   sec   373 MBytes  3.13 Gbits/sec                 
[  4]   5.00-6.00   sec   362 MBytes  3.04 Gbits/sec                 
[  4]   6.00-7.00   sec   354 MBytes  2.97 Gbits/sec                 
[  4]   7.00-8.00   sec   649 MBytes  5.45 Gbits/sec                 
[  4]   8.00-9.00   sec   719 MBytes  6.03 Gbits/sec                 
[  4]   9.00-10.00  sec   698 MBytes  5.86 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  5.45 GBytes  4.68 Gbits/sec                  sender
[  4]   0.00-10.00  sec  5.45 GBytes  4.68 Gbits/sec                  receiver

iperf Done.

some '-bidir' tests:

Code:

Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51097 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   377 MBytes  3.16 Gbits/sec                 
[  4]   1.00-2.00   sec   378 MBytes  3.18 Gbits/sec                 
[  4]   2.00-3.00   sec   375 MBytes  3.15 Gbits/sec                 
[  4]   3.00-4.00   sec   377 MBytes  3.16 Gbits/sec                 
[  4]   4.00-5.00   sec   375 MBytes  3.15 Gbits/sec                 
[  4]   5.00-6.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   6.00-7.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   7.00-8.00   sec   378 MBytes  3.17 Gbits/sec                 
[  4]   8.00-9.00   sec   375 MBytes  3.15 Gbits/sec                 
[  4]   9.00-10.00  sec   379 MBytes  3.18 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.68 GBytes  3.16 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.68 GBytes  3.16 Gbits/sec                  receiver

iperf Done.


Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51113 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   373 MBytes  3.13 Gbits/sec                 
[  4]   1.00-2.00   sec   712 MBytes  5.97 Gbits/sec                 
[  4]   2.00-3.00   sec   722 MBytes  6.05 Gbits/sec                 
[  4]   3.00-4.00   sec   739 MBytes  6.20 Gbits/sec                 
[  4]   4.00-5.00   sec   746 MBytes  6.26 Gbits/sec                 
[  4]   5.00-6.00   sec   715 MBytes  6.00 Gbits/sec                 
[  4]   6.00-7.00   sec   725 MBytes  6.08 Gbits/sec                 
[  4]   7.00-8.00   sec   738 MBytes  6.19 Gbits/sec                 
[  4]   8.00-9.00   sec   734 MBytes  6.15 Gbits/sec                 
[  4]   9.00-10.00  sec   732 MBytes  6.14 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  6.77 GBytes  5.82 Gbits/sec                  sender
[  4]   0.00-10.00  sec  6.77 GBytes  5.82 Gbits/sec                  receiver

iperf Done.

Performance seems inconsistent to me.

Enabled the 'disable HW offloading' tick under interfaces, following result:

Code:

Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51216 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   739 MBytes  6.20 Gbits/sec                 
[  4]   1.00-2.00   sec   740 MBytes  6.21 Gbits/sec                 
[  4]   2.00-3.00   sec   750 MBytes  6.29 Gbits/sec                 
[  4]   3.00-4.00   sec   753 MBytes  6.32 Gbits/sec                 
[  4]   4.00-5.00   sec   752 MBytes  6.31 Gbits/sec                 
[  4]   5.00-6.00   sec   744 MBytes  6.24 Gbits/sec                 
[  4]   6.00-7.00   sec   743 MBytes  6.23 Gbits/sec                 
[  4]   7.00-8.00   sec   745 MBytes  6.25 Gbits/sec                 
[  4]   8.00-9.00   sec   752 MBytes  6.30 Gbits/sec                 
[  4]   9.00-10.00  sec   751 MBytes  6.30 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  7.29 GBytes  6.26 Gbits/sec                  sender
[  4]   0.00-10.00  sec  7.29 GBytes  6.26 Gbits/sec                  receiver

iperf Done.


Connecting to host 192.168.1.33, port 5201
[  4] local 192.168.1.86 port 51252 connected to 192.168.1.33 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   351 MBytes  2.93 Gbits/sec                  
[  4]   1.01-2.00   sec   711 MBytes  6.00 Gbits/sec                  
[  4]   2.00-3.00   sec   698 MBytes  5.86 Gbits/sec                  
[  4]   3.00-4.00   sec   713 MBytes  5.98 Gbits/sec                  
[  4]   4.00-5.00   sec   700 MBytes  5.87 Gbits/sec                  
[  4]   5.00-6.00   sec   706 MBytes  5.92 Gbits/sec                  
[  4]   6.00-7.00   sec   704 MBytes  5.91 Gbits/sec                  
[  4]   7.00-8.00   sec   716 MBytes  6.00 Gbits/sec                  
[  4]   8.00-9.00   sec   681 MBytes  5.72 Gbits/sec                  
[  4]   9.00-10.00  sec   636 MBytes  5.33 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  6.46 GBytes  5.55 Gbits/sec                  sender
[  4]   0.00-10.00  sec  6.46 GBytes  5.55 Gbits/sec                  receiver

iperf Done.

seems like similar results, so reverted that interface setting.

 

[Matthias]

I have similar problem with Mikrotik SFP switch - the link is not established independent of modules that I try on the switch.
I found that if I manually select network speed (not use auto negotiation) - for example 1GB in Mikrotik switch, the link is established and working without other issues.
If you have time for tests you can try to set manually 10GB for NAS SFP port on Mikrotik switch and try again different SFP modules combinations.

Tried setting the speed manually.
Could only select the speeds seen in the screenshots with all other modules.

Some of the modules went 'link on' very briefly if i set it to another speed, but couldn't get them to work. (checked/tried with cli with all the commands mentioned in this thread)

So either my Uptimed branded card and module are very picky, or Microtik doesn't like their own modules. I suspect the former.

 

[nKk]

The iperf results are inconsistent and lower then expected.
Can you connect directly desktop PC and NAS without switch and make new iperf tests?

 

[Matthias]

The iperf results are inconsistent and lower then expected.
Can you connect directly desktop PC and NAS without switch and make new iperf tests?

I did the tests, at least i tried..

I encountered very strange behaviour around here. Not quite sure what to make of it.
I ran iperf one more time, same results.

Connected the fiber straight to the NAS, but could not get it to work.
(i think? i changed the default gateway from my router, to the ip of my desktop. that's it on the NAS side. On the desktop, set a static ip, that's it.)
So i reverted every setting i did on the NAS side and desktop side.

That's when the ghosts started appearing..

So, not only i cannot for the life of me connect to anything on my network from my desktop.. But i can go to youtube etc.. (iperf, ping to google, literally nothing works, except browsing normally?)
I cannot go to my switch, nor the Truenas webgui, literally not a single device..

Rebooted every device off course, but cannot get anything to work.

Changed my desktop connection to my 1Gbe connection, everything works..

So i'm further away than yesterday

I took this screen from the console:

Normally, only the .33 adress should be there. But i cannot clear this.

I tried following:
- "ifconfig (every interface) down"
- "ifconfig ix1 up" -> ok, shows above screen

Tried, using igb0 and inserting my copper gbe link, disconnecting every other connection, no luck. Same screen persists.

I still cannot access the webgui on .33 (or .177), but plex seem to run fine.
Only thing i think that could be messing with the system is my adguard server? I cannot access this either.

So my desktop wont run the 10Gig fiber, needed to revert to copper Gbe to get things somewhat running.
My NAS is the exact opposite.. Won't run the copper Gbe, but insists on the 10G fiber.

Multiple switch reboot off course, and averything up and running accourding to the Mikrotik's dash, and my Asus routers dash. (also rebooted).

For now, i'm going to let it rest.. before i go nuts..

 

[jgreco]

Well, there's something terribly wrong because you appear to have two interfaces on the same network. That causes all sorts of problems.

 

[Matthias]

Well, there's something terribly wrong because you appear to have two interfaces on the same network. That causes all sorts of problems.

I know, right?
Though only the copper or fiber was connected, along with the ipmi slot from my mobo.

I'll try again disconnecting the ipmi.