Hey guys!
I just want to start off by saying I'm new to the forums, and kinda new to TrueNAS, and infiniband.
Long, long, long story short, we have 2 servers, 1 TrueNAS server, 1 Host server running proxmox both with Infiniband ConnectX-3 40GB Cards, and an Infiniband IS5022 switch.
After a very long time of figuring how to get infiniband to just show up on everything, We finally got everything communicating, however very poorly. iSCSI is slow, / unreliable, lots of packet drops, and iPerf usually shows a bandwith of 4gbit/s.
After some research, It seems like the cards should be in connected mode, instead of datagram, which allows MTU to go higher than 2044, in our case, since it's infiniband, we need an MTU of 65520. Increasing the MTU to 65520 on Proxmox / host end, and 9216 on TrueNAS (highest it can be set) increases iPerf performance to 10gigabit/s, but way more packet loss, because of the mismatched MTU I would assume.
So my questions are:
1.) How can we set MTU to 65520 on the TrueNAS end
2.) How can we verify the cards are in Connected mode (not datagram mode) on TrueNAS
3.) The monitoring for the cards on the dashboard show the link state as unknown, even though they are up and connected. Is there a fix for this?
I just want to start off by saying I'm new to the forums, and kinda new to TrueNAS, and infiniband.
Long, long, long story short, we have 2 servers, 1 TrueNAS server, 1 Host server running proxmox both with Infiniband ConnectX-3 40GB Cards, and an Infiniband IS5022 switch.
After a very long time of figuring how to get infiniband to just show up on everything, We finally got everything communicating, however very poorly. iSCSI is slow, / unreliable, lots of packet drops, and iPerf usually shows a bandwith of 4gbit/s.
After some research, It seems like the cards should be in connected mode, instead of datagram, which allows MTU to go higher than 2044, in our case, since it's infiniband, we need an MTU of 65520. Increasing the MTU to 65520 on Proxmox / host end, and 9216 on TrueNAS (highest it can be set) increases iPerf performance to 10gigabit/s, but way more packet loss, because of the mismatched MTU I would assume.
So my questions are:
1.) How can we set MTU to 65520 on the TrueNAS end
2.) How can we verify the cards are in Connected mode (not datagram mode) on TrueNAS
3.) The monitoring for the cards on the dashboard show the link state as unknown, even though they are up and connected. Is there a fix for this?