SOLVED 10 Gbe recommendation / feasibility

[chuck32]

TrueNAS SCALE
Supermicro X10SRi-F
Xeon 2640v4
128 GB ECC RAM
Seasonic PX-750

Hello all,

since I'm finishing up my new build (still mitigating some problems) I thought I maybe dive into 10Gbe networking, since I also plan to switch from RAIDZ2 to a striped mirror.

The setup is:

2 Windows Machines (Win 11) and the NAS all in the same room. One win machine directly next to the NAS, the other across the room, currently connected with ~ 10m ethernet cable to a switch that connects to the NAS and other windows machine.

I do not need a switch, since no other devices would need to use the faster connection. So I thought since most 10Gbe NICs are dual port anyway, I could setup something like:

Basically the current setup for ethernet stays so every machine can still be connected to the internet. My idea is to use both ports on the NAS to connect to either WIndows machine for 10 Gbe transfer speeds. Other devices can still access SMB shares via WLAN because the NAS is still connected to the switch.

From https://www.truenas.com/community/threads/10-gig-networking-primer.25749/ I gathered that chelsio would be a good choice.

I could get

CC2- S320E-SR Dual-Port 10GB Chelsio ISCSI 10Gbps 2xSFP+ inkl 783025-101 42DV1 for 50 Eur each.
I couldn't find out what inkl 783025-101 42DV1 is. It looks like it's a transceiver already shipped with the NIC (783025-101 inkl. 2x10Gbps SFP+)

Intel X520-DA2 10 Gigabit 10GBe SFP+ for 80 Eur each.

For the connection WIN 1 and NAS a < 0.50 m cable would be sufficient (maybe a DAC cable?).

What about the connection to WIN 2?

Tranceivers seem to run about 25 Eur each if not included short optical cables run about 5-10 Eur. I don't seem to find a solution that breaks the bank for Win 2..

Integration of Win 2 is not necessary from the start but the hardware I buy should be able to support that.

Any insights are greatly appreciated!

Have a nice weekend!

 

[NugentS]

It will work - but both 10Gb links will need to be on seperate network addresses otherwise TrueNAS will complain.

 

[chuck32]

If my SMB shares are currently available under .100 (just an example) and I use a bridge in truenas:

How would I access my shares if both connections do get separate addresses?

Any recommendations on chelsio vs intel? Especially with chelsio, the linked guide talks about S320E-SR-XFP, would the model I linked work too?

I guess I need some further reading on what transceivers and cables to use.

 

[danb35]

That's not a valid configuration--as @NugentS said above, each interface (i.e., port) needs an address on its own network. So if your primary network is 10.10.10.0/24, the first 10G port might be 10.10.11.100 (and the first Windows PC's 10G interface could be 10.10.11.101), and the second port might be 10.10.12.100. And then the Windows PCs connect to those respective addresses for the SMB shares.

Or just skip all this nonsense, get a 10G switch, and be done with it. And then you can even bond those two 10G ports into a LAGG group if you like.

Edit: Chelsio vs. Intel is pretty much at your preference these days, but if you go with Chelsio, you'd really want at least a T4 card; the T3 cards are old enough that they can't get 10 Gbit/sec throughput. Solarflare is another valid choice. For transceivers, compatible optics from fs.com work perfectly and are about $20 each; they also sell nice OM4 patch cables in whatever length you might want. Or you can look on eBay for optics and probably pay a bit less--I usually buy from fs.com.

 

[chuck32]

The switch you linked only has 1 Gbe port, it's not apparent from my diagram, but I would to connect a few more devices.

That switch isn't much more expensive but features enough 1 Gbe ports and 4 SFP+ Ports, that seems like a good choice for me.

I'm from Germany so I'll need to look for another source for cables. I think I'll go with Intel then, I couldn't really source t3/t4 devices. Ordering from china has additional customs and taxes, so I stick with what is available in the EU on ebay to save myself from the headache.

For the windows machines, are there cheaper alternatives to the NICs?

My current shopping list (if someone could have a look if I missed something compatibility wise?):

Transceiver E10GSFPSR 33 Eur
Intel X520-DA2 70 Eur (found it cheaper)
850nm cable in 0.5 and 10 m (7 Eur and 20 Eur resp.)
MikroTik CRS310 180 Eur
----------------------------------------------

 

[ChrisRJ]

I bought 2 X520-DA2 incl. transceivers for 50 Euros the other day, so 70 Euros is not a bargain.

 

[chuck32]

Would you mind sharing where you bought it? 50 for everything seems like a steal :)

 

[ChrisRJ]

From https://www.piospartslap.de/ . The cards were priced for a bit more and I made an offer and they accepted it.

This is in no way an endorsement and I have no connection other than having bought from them.

 

[chuck32]

Thanks! Probably I misunderstood and it's 50 per, but still considering that the transceivers are included that's the much better deal.

 

[danb35]

The switch you linked only has 1 Gbe port, it's not apparent from my diagram, but I would to connect a few more devices.

You can always chain multiple switches together if needed.

I'm from Germany so I'll need to look for another source for cables.

fs.com seems to ship to Germany, as well as to many--perhaps most--other countries. But I can't speak to the tax/duty situation.

 

[Patrick M. Hausen]

They ship to EU customers from Germany, absolutely no problem/hassle whatsoever with customs, returns, etc.

HPC, Rechenzentrum, Unternehmen, Telekom - FS.com Deutschland

FS ist eine neue Marke im Bereich Rechenzentrum, Unternehmens- und Telekommunikationslösungen. Wir machen es für IT-Fachleuten einfach und kostengünstig, ihre Geschäftslösungen zu nutzen.

www.fs.com

 

[chuck32]

You can always chain multiple switches together if needed.

I know, but the version with more ports is only 30 Eur more and I could use my current small switch elsewhere in the house then.

fs.com seems to ship to Germany,

They ship to EU customers from Germany, absolutely no problem/hassle whatsoever with customs, returns, etc.

Good to know! When they showed up in my search results earlier, there were insane shipping costs displayed so I assumed they would ship from outside the EU. I signed up for an account and they offer reasonable shipping prices / free shipping to germany.

 

[chuck32]

After some more search, the intel x520 can be had for a good price but only with SFP transceivers included.

I found ebay offers that included this transceiver as compatible: Intel Finisar FTLX8571D3BCV-IT SFP+ 10 Gb 850nm Transceiver

They can be had for 10 Eur each.

So for the complete setup I would need:

3x Intel X520-DA2
4x Transceiver FTLX8571D3BCV-IT
850 nm cable (OM4) (LC UPC auf LC UPC)
1xMikroTik CRS310

Mikrotik states the transceiver should be compatible.

I'm pretty confident, all listed HW should be compatible?

Thank you so much for all your input!

 

[chuck32]

Or just skip all this nonsense, get a 10G switch, and be done with it.

Eventually I will, but for now I wanted to skip the switch since the second windows machine does not need 10G and the switch is currently not available in germany anyway. So while I'm still browsing for possible switches I though I'd go the direct connect route, but I can't seem to get it working.
I'd really appreciate if someone could guide me here.

I thought I understood what @NugentS and @danb35 wanted to tell me, then I googled and stumbled upon nugentS answer here.

So I even changed the IP addresses to exactly what is proposed there to no avail:

I cannot change the DNS server assigment to anything other than automatic on windows, it just won't save.

The cable I'm using is 1m LC UPC to LC UPC duplex OM4 Multimode 2.0mm Fiber Optic Patch Cable I received from fs.com (great tip! shipping costs are rather high but they only seem to offer express shipping which I didn't really need).

NIC Intel X520-DA2

Transceivers are HP Original 10Gb SR SFP+ 455885-001

I connected the fiber cables in what I felt was the only fitting way:

Do I need to physically seperate the cables and switch A and B since I used it as a crossover cable? With RJ45 it wouldn't matter but with fibre?

Any input is appreciated!

 

[Patrick M. Hausen]

I would create a bridge interface with the two 10G ports and the 1G port. Plug 1G into switch with local LAN and Internet Uplink, plug Windows PCs into 10G ports. Fixed IP address for NAS, DHCP for Windows, easy peasy.

I can explain in more detail how to do that tomorrow.

The NAS can easily bridge up to 1G of traffic of the Windows PCs with other devices in the network and traffic from PC to NAS and vice versa will terminate in the NAS' IP stack, so the bridge won't be a bottleneck.

 

[chuck32]

Do I need to physically seperate the cables and switch A and B since I used it as a crossover cable? With RJ45 it wouldn't matter but with fibre?

So the default state in which I received the cables seems to be:

One end has A-A and B-B the other end has A-B and B-A (this should be apparent from the pictures I posted before).

I then used my other cable and switched one end so both ends are A-A and B-B - still no connection.

I would create a bridge interface with the two 10G ports and the 1G port. Plug 1G into switch with local LAN and Internet Uplink, plug Windows PCs into 10G ports. Fixed IP address for NAS, DHCP for Windows, easy peasy.

Sounds like a great idea! Thanks!

With the default cable orientation (as in my first post) I tried it and still no activity on truenas or windows, both tell me "not connected" (windows written and treunas with the greyed out icon).

 

[chuck32]

What a way to waste an evening...

In the rare case you are sure you didn't do anything wrong, consider the possibility you didn't do anything wrong...

tl:dr: either HP Original 10Gb SR SFP+ 455885-001 is not compatible or I received two DOA transceivers.

I got curious because there were no LEDs active on either X520 card and basically every swtich, nic I know had some LEDs, altough they all were consumer grade. Also in windows it showed

What if I was plugged into #2?

When I ordered the cards they just put whatever SFP+ transceivers they had (although the card was sold as SFP) and put a matching type in each card. So I received the aforementioned HP Original 10Gb SR SFP+ 455885-001 and two Intel Finisar FTLX8571D3BCVIT1.

So I got them out (I put them away for future use so they won't collect dust, there were no dust caps) and hot plugged them into the free slot on the windows machine. Immediately LEDs on both NICs lighted up and in windows it was now an unknown network 10G/10G. After adding the second transceiver also to the server and swapping cables: voila!

I don't know why the speed drops and windows doesn't show correct SMB usage for ethernet 2, but bottom line is apparently the HP Original 10Gb SR SFP+ 455885-001 are both either broken or not compatible, I will contact the reseller and see if I can get them swapped for the finisar ones.

Thank you all for you help and @Patrick M. Hausen I'd still be interested in your input about the bridge adapter. Ideally I would want to route all traffic from the server to my windows machine over the new 10g link. All web guis are loading slowly (on this hardware and on the previos server). I know that it can't be that 1g isn't enough but maybe it helps to some extent.

 

[ChrisRJ]

The alternative approach I had taken (having bought the same package from the same seller) was to disable the check for compatible SFP transceivers.

For Linux (on my XCP-ng machine) I appended

ixgbe.allow_unsupported_sfp=1,1

to the first module2 line of /boot/grub/grub.cfg and it worked.

And on for FreeBSD (on my TrueNAS Core machine) I added a tunable

hw.ix.unsupported_sfp=1

 

[chuck32]

I can explain in more detail how to do that tomorrow.

Could you elaborate a bit more if you get to it? I went ahead and tried it myself now that the connection is working:

On my windows machine:

In this hardware configuration:

And on truenas I added the 10g link to the bridge:

and removed the fixed IP for ens4f0 (the 10g link)

But then I had no connection from my windows machine to either the NAS nor the internet.

The alternative approach I had taken (having bought the same package from the same seller) was to disable the check for compatible SFP transceivers.

Thanks! I'd still try to get them to swap them so I don't have to work around if possible.

 

[Patrick_3000]

You don't need really a 10GB switch. SCALE makes an excellent switch using a bridge adapter. Just make sure there are no IPv4 settings on any physical adapter. Then go to the Network tab and create a bridge adapter (call it "br0" for example). Add whatever physical adapters you want to the bridge adapter. Then when you're done, add IPV4 to the bridge adapter.

Now, you can connect one of the physical adapters in the bridge to your router and connect the other physical adapters in the bridge to whatever other devices on the network you want to network at 10GB, and you're done.

The only question is what network card to use in SCALE for 10GB connections. I would consider getting a quad port card rather than a dual port card.

The only caveat I would add is that this setup will be slightly less robust than a switch because if you have to take the SCALE server offline, you will lose networking on the other devices connected to the bridge. But SCALE is so stable that I leave it running for months, and this is rarely a problem. Still, a switch would be slightly more robust, and 10Gb switches aren't that expensive anymore, but they take up shelf space and are an extra power cable to plug in. It's just a matter of your priorities.

The other difference is latency. A switch has lower latency than a bridge on SCALE due to ASIC technology. When I used to run a SCALE bridge for my 10GB networking, pinging from a connected device to the router took around 800-900 microseconds. Now that I have switch, it takes around 400-500 microseconds. I doubt, however, that that difference would ever matter on a small or home network.