Supported 40GbE cards?

[friolator]

Which model switch btw?

It's an IBM RackSwitch G8316R 16-Port. Got it on ebay for $1200, brand new and still shrinkwrapped.

One nice thing is that it can use a single 40GbE to 4x 10Gbe breakout cables, so eventually we can upgrade most of the PCs that don't need crazy bandwidth, without using up all the ports on the switch.

 

[JustinClift]

Interestingly. Digging around for info on compatibility of that model isn't showing huge amount of stuff right away, but there's a mention of the IBM RackSwitch's being OEM'd Juniper ones:

https://forums.servethehome.com/index.php?threads/ibm-blade-rackswitch-g8124-g8124r.4123/ (different model to yours)

Might be a useful data point while getting it working. :)

 

[friolator]

The switch itself seems to be functioning just fine. It's communicating with the FreeNAS box and there are no issues there. The problem has been getting it to talk to the workstations, which have the ConnectX cards. This discussion got me curious again, so I set aside the afternoon to mess with it and see if I can get one of the workstations on the switch...

I mean, worst case we'll put different cards in there, but we got three of the ConnectX-3 cards for $200 total, so my inclination is to spend some more time trying to make this work. I'll hit up the servethehome folks if I don't get anywhere.

 

[JustinClift]

Sounds like a plan. :)

If you haven't already, it's a good idea to flash the firmware of Mellanox cards to the latest available. Some weird problems really do go away that easily.

That's from experience with older ConnectX-1 cards from ebay. Not had a personal use for anything faster. ;)

 

[friolator]

Ok, it's all working now. Took some doing but basically the issue was that my firmware *was* too old, only I couldn't update the firmware with the current Mellanox binary, I had to build a custom OEM binary and burn that. My cards are IBM branded so the firmware tools just wouldn't do it. It was actually simple though - download the source, change the PSID in the .ini file to the one used by the card, and then burn that. Worked like a charm, and the card and switch are now talking at 40Gbps. As of right now we have a very small network consisting of the FreeNAS box, one Windows 7 workstation and the switch, all functioning normally, as far as I can tell.

Next step is to get some drives in there and start performance testing. This should be the fun part.

 

[JustinClift]

Very awesome. :D

 

[CookiesLikeWhoa]

Can you post some iperf results by any chance?

I'm thinking of taking the 40Gbe plunge and have been looking at the different NICs to see how they all perform.

 

[friolator]

I'm working on some other network issues right now (dropped connections at the FreeNAS machine), but basically, I'm getting about 14.1Gbps on iperf tests that run about 20 minutes. It's consistently capped at that rate. I haven't done much tuning yet though, since we're still trying to figure out what's causing the dropped connections. Once that's settled, I'll be able to run more tests and try to fine tune the speed.

I'm a bit disappointed in the disk speed tests so far, though. We're only getting about 600MB/Second writes and 1200MB/Second reads. I pretty much need to double the writes and get the reads up closer to 1500 for this setup to be viable for us. I'm not sure at this point if that's a FreeNAS thing or not, but we're easily achieving those speeds on direct-attached SATA raids with a similar number of drives. Even if our network speed is capped at 14Gbps, the bigger issue seems to be drive speed.

Granted, these tests are via windows shares, so that may be a factor as well. We'll also be testing iSCSI, which should remove some of the overhead that SMB/CIFS introduces.

 

[CookiesLikeWhoa]

Yeah on my 10Gbe network CIFS/SMB writes are really poor. iSCSI everything is a prefect 10Gbe each direction though.

Good luck with the network issues! Might want to check the MTU's across the board. Caused me a headache with dropped connections.

 

[depasseg]

Granted, these tests are via windows shares,

Which is limited to a single thread, which means CPU speed is crucial.

 

[friolator]

I just increased the MTUs to 9614 (the max the Mellanox card will accept), and the speed on the iperf test jumped to about 21Gbps, so that's pretty substantial!

Which is limited to a single thread, which means CPU speed is crucial.

Yeah, the hope was that we'd be able to do everything through standard shares like this for simplicity's sake, but in the back of my mind, I've been planning to use iSCSI if the performance wasn't up to snuff. Still more testing to do, so I haven't ruled this out yet, but in real-world terms, we're able to play back 4k 10bit image sequences at almost real time speed (23fps), and write it at half of that. It's pretty close to what we need for reads, but writes are way off at this point.

In the end, I suspect we'll have a combination: some volumes will be iSCSI, and others will be SMB/CIFS, each with different end-user purposes.

 

[CookiesLikeWhoa]

The ports on the switch are set to something over 9614 correct? Just making sure because I've made that mistake before. 21Gbs is a huge jump though. At least a simple change netted a 33% increase.

 

[friolator]

The ports on the switch are set to something over 9614 correct? Just making sure because I've made that mistake before. 21Gbs is a huge jump though. At least a simple change netted a 33% increase.

Hard to tell what the ports are set to on the switch. The UI for the admin interface is ...less than intuitive. I'll have to dig into that soon. Still having dropped connections, and this may be related to that, so maybe sooner than later!

 

[Mlovelace]

Yeah on my 10Gbe network CIFS/SMB writes are really poor. iSCSI everything is a prefect 10Gbe each direction though.

Good luck with the network issues! Might want to check the MTU's across the board. Caused me a headache with dropped connections.

While I agree iSCSI/NFS has less overhead you can still saturate a 10Gbe link with CIFS. Here's a link to previous thread where I demonstrate it. https://forums.freenas.org/index.ph...rdware-smb-cifs-bottleneck.42641/#post-277479

 

[friolator]

BTW - the change in MTU bumped up the disk speed tests as well - we went from 600 to 880 MB/s on the reads and 1200 to 1500 on the writes. So maybe this is less about disk speed than I thought.

 

[CookiesLikeWhoa]

While I agree iSCSI/NFS has less overhead you can still saturate a 10Gbe link with CIFS. Here's a link to previous thread where I demonstrate it. https://forums.freenas.org/index.ph...rdware-smb-cifs-bottleneck.42641/#post-277479

I don't doubt it as I can get reads at wire speed. I just think I don't have the skillz required to get the write speeds. I've dug in as far as I can figure out how to but still can't get CIFS to saturate on writes. From what I've gathered its a mix of cards I'm using (Intel X520-DA2) on Windows 10 and lack of ability to truly tune it.

Eventually I'm going to move to Chelsio t420's and see if I can't get wire speed there.

BTW - the change in MTU bumped up the disk speed tests as well - we went from 600 to 880 MB/s on the reads and 1200 to 1500 on the writes. So maybe this is less about disk speed than I thought.

With more tuning you should be able to hit your requirements it seems.

 

[friolator]

Short of swapping DACs out (which I'm going to do tomorrow - waiting on a network engineer friend to bring over a few brands from his work so we can test them to see if there's improvement), does anyone have suggestions on the software side for what we can do to stop the dropped connections we're seeing on the 40Gb network? We're getting them every few minutes, and only on the FreeNAS box, never on the workstation. I've been running long iperf tests, and every 8-10 minutes, we drop the connection for 30 seconds to a minute, then it resumes.

Here's what I've tried:

1) Various tunables:

kern.ipc.somaxconn 2048
net.inet.ip.ifq.maxlen 4096
net.inet.tcp.always_keepalive 0
net.inet.tcp.recvbuf_auto 1
net.inet.tcp.recvbuf_inc 524288
net.inet.tcp.recvbuf_max 16777216
net.inet.tcp.sendbuf_auto 1
net.inet.tcp.sendbuf_max 1677216

2) MTU size 9000 (Previously had 9614, but found out when setting this at the CLI that the driver only takes up to 9000 even though the GUI let me set it higher). This resulted in approximately 33% boost in throughput, according to both iperf and real-world tests, BTW

3) I've also tried checking the autotune box in the System/Advanced settings screen


Again, the switch seems to be fine - I've tried multiple ports and different cables (same brand, different lengths), and the workstation on the other end of the switch has had no dropped connections according to the switch logs. it's just at the FreeNAS side. Wherever we could find a windows equivalent in the Mellanox Driver settings, we set it to match those on the FreeNAS box.

Thanks!

 

[friolator]

different DAC cables made no difference. I started digging into the switch and saw that there were some firmware updates for it, so I took care of that today, and it seems to be much happier. It's significantly more stable now (the connection between Chelsio Card and switch) and has been running iperf tests for the past 20 minutes with no dropped connections. I'm going to keep that going for a few more hours, then check the logs tomorrow AM to see if it had any problems.

 

[CookiesLikeWhoa]

Wow that's excellent news! Hopefully everything is consistent now with the connection.


Sent from my iPhone using Tapatalk

 

[friolator]

So far it's been running over an hour at 21Gbps with no dropped links. so I'm reasonably confident that this solved that problem at least. Now I can get back to tuning it to try to get closer to 40gbps...