General network considerations

[Allan_M]

Hi

I think I'm in dire need of a sanity check - again.

My current situation:

• I use a 2020 M1 MacBook Pro in clamshell-mode for most of my internet browsing, photoshop'ing, checking mail'ing and video editing.
• I have a TrueNAS-system on the network (X10SLL+-F)
• Soon I'll be adding:
  • A workstation for larger projects, and yes - some gaming too, but primarily photo and video editing, motion graphics and some entry level 3D-work. In the long run I would also like to do some Stable Diffusion, but that's another talk entirely.
  • CrazyNAS 4 th3 lulz (X10SDV-4C-TLN2F)

Our entire topology is currently 1 GbE but some time ago I bought a MikroTik CRS305 (1x 1Gb + 4x SFP+).

I would very much like your input on the best ways to connect: MacBook Pro, worktstation, TrueNAS-system and CrazyNAS 'together'.

The main purpose is faster file transfers and the ability to work directly off of the NAS.

I have no opinion on nics other than it should 'just work'. My primary concern is the mix of interfaces.

MacBook Pro is Thunderbolt. Fine, I'll be adding a thunderbolt adapter - was thinking of the Qnap QNA-T310G1S (DKK 1.800,- ~ $260)​

Workstation has plenty of PCIe. Since I'll be running Windows I considered anything "from":​

Mellanox ConnectX-2 (DKK 230,- ~ $33)​

Mellanox ConnectX-3 (DKK 250,- ~$36)​

Intel X520 (DKK 280 ~ $41)​

Intel X540-DA2 (DKK 350,- ~ $50)​

Intel X520-SR (DKK 400,- ~ $58)​

Chelsio T520-CR (DKK 420,- ~ $61)​

Chelsio T520-SO-CR (DKK 480,- ~ $70)​

TrueNAS. Chelsio? Mellanox is heavily discouraged in the forum and I don't want to deal with fake Intel cards.​

CrazyNAS has built in 10GbE (RJ45).​

​

Four systems, that somehow needs to be connected. I would prefer DACs or fibre. I'm absolutely not married to RJ45 and to be frank. The idea of fiber for my SOHO-network sounds plain awesome to me.

I also considered dual SFP28 / 25GbE for a while, but went away from that again. My thinking was 25 GbE between the servers (with some PCIe splitting on the CrazyNAS) to enable fast transfers or just being able to connect the workstation directly to the NAS. I didn't make much practical sense and to be honest, 10 Gig is probably just fine for now.

So. How would you tackle this? My main concern is compatability.

Oh. And everything connects back to a UDM Pro fwiw.

 

[ChrisRJ]

What is this "CrazyNAS" thing?

If you want to do video editing directly on the NAS your random I/O will likely not be sufficient. You probably need something SSD-based for this task.

Dual 25 Gbps would indeed be a massive mismatch for the disks you have (even a single 10 Gbps link will often not be fully utilized).

 

[Allan_M]

What is this "CrazyNAS" thing?

If you want to do video editing directly on the NAS your random I/O will likely not be sufficient. You probably need something SSD-based for this task.

Dual 25 Gbps would indeed be a massive mismatch for the disks you have (even a single 10 Gbps link will often not be fully utilized).

CrazyNAS is my +48 drive all flash based NAS. It took the name after jgreco "... for your little crazyNAS experiment".

I've made a very poor "build log": DIY all flash/SSD NAS (CrazyNAS issues)

It should have been clearer. I’m sorry

Dual 25 GbE was for between the NAS'es, with the added benefit of also having an available 25 GbE to hook back to the 10 GbE

 

[Etorix]

I have no opinion on nics other than it should 'just work'. My primary concern is the mix of interfaces.

There's not much of a mix. You need one RJ45 transceiver to connect CrazyNAS to the CRS-305 switch and the rest can be all fibre (or DAC for short runs).

MacBook Pro is Thunderbolt. Fine, I'll be adding a thunderbolt adapter - was thinking of the Qnap QNA-T310G1S (DKK 1.800,- ~ $260)​

Workstation has plenty of PCIe. Since I'll be running Windows I considered anything "from":​

Mellanox ConnectX-2 (DKK 230,- ~ $33)​

Mellanox ConnectX-3 (DKK 250,- ~$36)​

Intel X520 (DKK 280 ~ $41)​

Intel X540-DA2 (DKK 350,- ~ $50)​

Intel X520-SR (DKK 400,- ~ $58)​

Chelsio T520-CR (DKK 420,- ~ $61)​

Chelsio T520-SO-CR (DKK 480,- ~ $70)​

TrueNAS. Chelsio? Mellanox is heavily discouraged in the forum and I don't want to deal with fake Intel cards.​

T520-CR indeed (SO-CR taps into CPU power, if on top of that it comes out more expensive…).
And anything you want with Windows—I would personally get a batch of Chelsio cards with a corresponding batch of SFP+ modules and not bother with matching the right module to the right card.
You may even use any ConnectX-3 (and later), Intel X500/X700 or Chelsio card in a Thunderbolt enclosure with the MacBook Pro, but a Thunderbolt adapter is smaller, more convenient and likely cheaper.

 

[Allan_M]

There's not much of a mix. You need one RJ45 transceiver to connect CrazyNAS to the CRS-305 switch and the rest can be all fibre (or DAC for short runs).

I phrased that poorly, but you seem to get my thinking:

• MacBook Pro (MBP): Thunderbolt -> something, preferably SFP+ (since everything 10 Gig RJ45-based, gets hot?)
• Workstation (WS) (has 1 GbE, wants 10 GbE, so): PCIe -> SFP+
• TrueNAS (TN) (X10SLL+-F, has 1 GbE, wants 10 GbE): PCIe -> SFP+
• CrazyNAS (CN) (X10SDV-4C-TLN2F, has dual 10 GbE): RJ45 -> ?
• CRS305: 1x 1GbE RJ45 and 4x SFP+

That's my poor definition of "mix of interfaces" (thunderbolt, SFP+, PCIe and RJ45). I thought I could do with just one transceiver, but was unsure - you just confirmed it for me. Thanks

T520-CR indeed [...] I would personally get a batch of Chelsio cards with a corresponding batch of SFP+ modules and not bother with matching the right module to the right card.

I was hoping I could do just that. I think I'll buy two Chelsio T520-CR's for the workstation and the TrueNAS-system.

Regarding the physical connection:
MBP: Qnap (SFP+) <- DAC -> CRS305
WS: T520-CR (SFP+, transceiver) <- DAC/fibre -> (transceiver) CRS305
TN: T520-CR (SFP+, transceiver) <- fibre -> (transceiver) CRS305
CN: built-in (RJ45) <- CAT6 -> (transceiver) CRS305

So, my "shopping" list, should [at least] be:
1x Qnap QNA-T310G1S
2x Chelsio T520-CR
1x DAC / 2x fibre cables or 2x DACs / 1x fibre cable
4x MMF transceivers (compatible Chelsio-something, I quess?)
1x RJ45 transceiver (MikroTik, compatible?)

Is that all or is there something I should get at least "one extra" of, "just in case"? I was thinking extra transceivers, DACs or fibre cables. For instance (added to the above):

1x Chelsio T520-CR
1x DAC / 1x fibre
2x MMF transceivers (perhaps for connecting something directly to one another, like peer-to-peer / ring-like topology)
1x RJ45 transceiver (link aggregation on the CrazyNAS, perhaps)

 

[Allan_M]

Update:

Since last - I know, I'm not moving fast - I've lucked into some gear.

For clearity; The current (wired) setup:

• Internet (1 Gbps fibre, office)
  • Router: UDM Pro (office)
    • Switch: Cisco Catalyst 2960-CG (office)
      • Switch: Netgear GS108 (attic)
        • Switch: Unifi US-8-60W (attic)
          • UAP-AC-Pro #1
          • UAP-AC-Pro #2
        • TrueNAS (Supermicro X10SLL+-F, entryway / utility room)
        • Desktop-PC (oldest kids gaming PC, bedroom)
        • Switch: TP-Link TL-SG105 (living room)
          • Apple TV
          • Xbox
          • Playstation
          • Nintendo Switch
      • MacBook Pro (office)
      • Lenovo Tiny (office)
      • Workstation (office)
      • What-ever device I'm currently working on (office)
      • Power-line Adapter: TP-LinkTL-WPA4220 (office, connected to receiver in another bedrom with the printer connected directly into it)

I know it's not the most elegant network topology - but it works.

Anywho. The office is right next to the entryway / utility room, less than 10m across the attic. I'm running a single cable through the ceiling from the Cisco 2960-CG to the Netgear GS108 on the attic. The GS108 is right next to the US-8-60W. Most cables are copper (cat5e) except for the bundle of fibre between the UDM and 2960-CG.

As indicated above I've been thinking of 10gig-ing for a while.

I've narrowed it down to:
- For the TrueNAS-box: Intel X520-DA2 / Chelsio T520-CR or Mellanox ConnectX-3
- For the MacBook Pro: Qnap QNA-T310G1S

Simple enough, right? Well. Not so fast.

As mentioned, I've lucked into some gear and also have made some purchases along the way. Gear I own, but isn't utilizing at the moment:

• Switches:
  • UniFi US-16-XG
  • MikroTik CRS305-1G-4S+IN
  • Cisco SG100D-08
• NICs:
  • Sonology E10G18-T2
• Cables:
  • UACC-DAC-SFP10-1M
  • UC-DAC-SFP+ (0.5M)
• Other:
  • Ubiquiti UF-MM-10G (like, a lot of them, at least 10!)

Where I'm at right now:
I have a lot of parts and things I want to connect. Most pressing right now is a desire to 10gig the MacBook to the TrueNAS-box.
I'm not sure if the Sonology nic will even work with TrueNAS but at least I can perhaps use it in another build or save it for something else.
I'd rather use an Intel / Chelsio or perhaps even Mellanox instead of fighting with the Sonology - but: If it worked in TrueNAS, that'd be awesome.

Anywho.

My questions:

1. Intel nics prefers Intel branded DACs, but also transceivers?
   • Haven't been able to find a specific list with/without the UF-MM-10G, but according to this post (link: "Advice wanted - 10G link between...") it works?
   • Others say the Intel card has to be told to ignore "supported" transceivers, but that post also mentioned transceivers from FS.com or 10Gtek, I think
2. According to the 10G primer and a lot of forum posts, Mellanox ConnectX-2 (or Mellanox in general) is discouraged.
   • I read this as in "TrueNAS + Mellanox = not good / you're on your own"
   • Also: It's not necessarily an issue with Windows - so a Mellanox card is good to go on that end?
3. A lot of NICs are listed as either "Intel", "Dell", "Lenovo" or "HP". I get, that Intel cards are Intel-cards. But, does Dell-branded Intel X520 prefer to be installed into a Dell-server? As in: It expects Dell-firmware/BIOS?
4. With the above mentioned gear in mind, how would/could one re-organize the network?
5. I don't mind buying Intel NICs with Intel-compatible transceivers, but how feasible would it be to run fibre around the house and is it even worth it vs. installing cat6a instead (no run would be longer than 20-30m/100ft)?
   • Dream scenario:
     • fibre to the living room, because, why not?
     • fibre to kids' bedroom, again, why not?