SOLVED Direct connect Macs to Freenas NAS NIC cards (no router or switch)

[VictorR]

We have a new 30-drive NAS running 9.3.1 with 3x dual 10GbE NIC cards that will be used as shared storage for our film post-production/editing room. The on-motherboard 1GbE port is receiving a 192.168.0.x address from an Apple Airport Extreme router(for internet access). Via the web GUI, I have the 30 x 4TB drives in RAID Z2. And have created a temporary AFP share for doing some bandwidth testing.

I configured the one of interfaces (ix0) as 10.0.0.1/24.

For testing purposes, right now, I manually configured my Macbook Pro's 1GbE NIC to 10.0.0.7/24. I was unsure what to put in the router field as there is none. Aiming my browser at 10.0.0.9, I get the Freenas GUI

Running Black Magic's Disk Speed Test, I am getting 110MB/sec read & write. All good.
But, when I try to add any of the other interfaces as 10.0.0.x /24 (netmask of 255.255.255.0), it says "The network 10.0.0.0/24 is already in use by another NIC"

I don't know why I am getting a brainfart on this, but I am at a loss why it isn't allowed. Ultimately, we would like to LAGG (link aggregate) each of the dual-NICs when this goes live for production.

 

[danb35]

Try deactivating all but one of the 10G NICs, and assigning the remaining one as 10.0.0.1/24. Configure your Mac to 10.0.0.2/24. See if that works.

When you use a /32 netmask, every single IP is its own network, and you'd need to route between networks to get to any other IP (which wouldn't be possible, as there's no other valid IP within the subnet for the router/gateway to occupy). When you use /24 and 10.0.0.X IPs, FreeNAS sees that you're trying to assign a bunch of interfaces to the same subnet, which is bad, so it won't let you. If the test above works, I think you'd need to set the NICs to 10.0.0.1/24, 10.0.1.1/24, 10.0.2.1/24, etc. (you could really get away with using /31 subnets, but you'd need to figure out the binary -> decimal for all the hosts--a /24 subnet just makes it easier).

 

[pirateghost]

put each interface on a completely different subnet (10.0.1.1, 10.0.2.1, etc) and attach to clients. Put client machine in same subnet as it's connection. There should be no route at all.

 

[danb35]

Oh, and I hope you don't mean your pool consists of a single, 30-disk RAIDZ2 vdev.

 

[VictorR]

Thanks everyone. This is a classic example of not stepping back and thinking something through
danb35, that is exactly what I did right after I posted and it worked. I was re-editing my OP when all of you posted.

As my revised post says, I now get 110MB/sec read/write over 1GbE and CAT5. Next will be connecting the Thunderbolt to 10GbE converter and testing throughput.

It now makes total sense to have each LAGG on a different subnet...and that's what the Freenas Interface config was trying to tell me

 

[jgreco]

but I am at a loss why it isn't allowed.

https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/

 

[Dennis.kulmosen]

So my 2 cent on this.
Use subnets as described by others.
Use 5 x 6 drive Raidz2, that gives you 5 times the IOPS.
Don't use LAGG you wont benefit from it the way you think.
Use the Sonnettech Twin 10GbE thunderbolt interface. Its pretty cheap and high performant.

Sendt fra min SM-G925F med Tapatalk

 

[VictorR]

So my 2 cent on this.
Use 5 x 6 drive Raidz2, that gives you 5 times the IOPS.

Thanks for the suggestion, I was thinking through some different configs to test this morning. I will definitely try your suggested 5 x 6

Don't use LAGG you wont benefit from it the way you think.

I was hoping it would boost ingest speed when multiple large ultra hi-def video files are being transferred to the RAID array. After doing some reading last night, it looks like it doesn't really help for that. The only benefit would be fault tolerance.

Use the Sonnettech Twin 10GbE thunderbolt interface. Its pretty cheap and high performant.

Yes, we've got 6 of them. Very easy to set up. I need to pick up some CAT6 cables on the way to work.
I got a weird test result using CAT5 last night via the BlackMagic Test (10 x 3 drive RAID):
late 2011 MacBook Pro > Sonnet 10G > NAS - 362MB/s write, 298MB/s read
2014 Mac Pro II > Sonnet 10G > NAS - 128MB/s write, 412MB/s read

I need to pick up some CAT6 cables on the way to work to rule out the 5' CAT5 patch cable as the problem. Although, at that length, latency shouldn't be that big an issue. And the variance between the two is odd. The Mac Pro write speed makes no sense. I tried all the 6 Thunderbolt ports and the performance was the same

The NAS manufacturer did 3 x 10 drives in Z2 and got 724MB/s write, 598MB/s read via the Sonnet into a Mac Mini client. With iSCSI target and SANmp iSCSI driver in the same setup, they got ~825MB/s write/read. Using a Linux client with the native iSCSI driver they were able to saturate the line at 1.1 GB/s