Daisy Chaining 40Gb Nics

Phase

Explorer
Joined
Sep 30, 2020
Messages
63
Hi guys. I need a sanity check creating a second subnet connecting 2 TrueNAS servers and a Desktop. The current setup has all computers as part of a regular 1Gb network (192.168.1.xxx). That will remain in place.

In addition to that I have 3 40Gb dual port cards and I would like to daisy chain them across the 3 computers on a new subnet (192.168.3.xxx). There are no other devices on this subnet. The purpose of this subnet is (1) to have fast access between the Desktop and NAS#1 and (2) to have fast access between the NAS#1 and NAS#2 for replication. All other access “paths” between these machines or between these machines and others can take place through the existing 1 Gb 192.169.1.xxx subnet.

I apologize in advance as I’m not a network person at all… Can you help me figure out if this is the correct hardware list to make this work?

- 3 x Chelsio T580-LP-CR Dual Port 40GbE QSFP HBA 110-1164-50 MSIP-REM-CC2-T580

- 4 x Chelsio SM40G-SR QSFP+ Transceivers

- 1 x 30-meter “Cable Matters 40Gb OFNP Plenum Rated Multimode Duplex 50/125 OM4 Fiber Cable (Fiber Optic Cable, LC to LC Fiber Patch Cable)”

- 1 x 3-meter “Cable Matters 40Gb OFNP Plenum Rated Multimode Duplex 50/125 OM4 Fiber Cable (Fiber Optic Cable, LC to LC Fiber Patch Cable)”

Any tips or caveats I should keep in mind while configuring this?

Thanks!!
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
Any tips or caveats I should keep in mind while configuring this?

Thanks!!

You won't be able to daisy chain them, you're missing a switch to get them all on the same subnet. You can create point to point networks between each machine. You might be able to spool up a router VM on one of the NAS boxes to allow the client to talk to the NAS it isn't directly connected to. But you are going to need two private networks to make it work, and knowledge of how to set up a static routing table, etc...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
If you don't actually need to talk at 40 Gb/s between desktop and NAS2, you can get away with this by putting each link in a separate network, statically configured of course.
 

Phase

Explorer
Joined
Sep 30, 2020
Messages
63
Thanks guys!! I can live with that :)

The OM4 Multimode Fiber... Is that good, meaning if it has those 3 words it is the right cable for this application?
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
Thanks guys!! I can live with that :)

The OM4 Multimode Fiber... Is that good, meaning if it has those 3 words it is the right cable for this application?

Those transceivers should be good for 150 meters on OM4 multimode fiber, or 100m on OM3.
 
Top