Hi all
I have a strange problem that I just noticed. I have 1 Freenas and 3 proxmox servers. All have a Mellanox ConnectX MT26448 card (10GBit). I have a switch with 4SFP+ ports. All ports on the switch show 10Gbit. All cards are, at least it seems to me, installed correctly. I did iperf tests between all 3 proxmox hosts where I always receive and can send 10Gbit.
When I connect from 1 proxmox host to freenas I also get 10Gbit/s:
but whenever I connect from the freenas to a proxmox host I only get 1GBit:
This is what I get when i do a ifconfig on the freenas:
I have a 1GB card on freenas which is on a seperate subnet. I did a traceroute to see if it was going the right way:
I'm really confused. What could the problem be? Could this actually be a faulty card, port or cable? If so, wouldn't it either not work at all or be 1Gig all the way or the switch wouldnt show 10GBit? I don't know where I should start debugging.
Thanks for your tips!
Edit:
When I pull a file from the freenas at the same time from all 3 proxmox hosts I get about 110MB/s for each host. Meaning the card is capable of actually transferring more than 1 GBit/s simultaniously. It just seems it is "capped" at 1GBit/s for each host. I haven't found such a setting in freenas.
My freenas Version btw is:
FreeNAS-11.1-RELEASE
and when I monitor the traffic I can see it all going through the mellanox card. It shows up to 3G on the monitor.
I have a strange problem that I just noticed. I have 1 Freenas and 3 proxmox servers. All have a Mellanox ConnectX MT26448 card (10GBit). I have a switch with 4SFP+ ports. All ports on the switch show 10Gbit. All cards are, at least it seems to me, installed correctly. I did iperf tests between all 3 proxmox hosts where I always receive and can send 10Gbit.
When I connect from 1 proxmox host to freenas I also get 10Gbit/s:
root@proxmox03:~# iperf -c 192.168.111.5
------------------------------------------------------------
Client connecting to 192.168.111.5, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.111.12 port 60618 connected with 192.168.111.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.9 GBytes 9.37 Gbits/sec
but whenever I connect from the freenas to a proxmox host I only get 1GBit:
root@freenas:~ # iperf -c 192.168.111.12
------------------------------------------------------------
Client connecting to 192.168.111.12, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.111.5 port 21453 connected with 192.168.111.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
This is what I get when i do a ifconfig on the freenas:
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:54:f0:c8
hwaddr 00:02:c9:54:f0:c8
inet 192.168.111.5 netmask 0xffffff00 broadcast 192.168.111.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
I have a 1GB card on freenas which is on a seperate subnet. I did a traceroute to see if it was going the right way:
root@freenas:~ # traceroute 192.168.111.12
traceroute to 192.168.111.12 (192.168.111.12), 64 hops max, 40 byte packets
1 192.168.111.12 (192.168.111.12) 0.157 ms 0.110 ms 0.076 ms
I'm really confused. What could the problem be? Could this actually be a faulty card, port or cable? If so, wouldn't it either not work at all or be 1Gig all the way or the switch wouldnt show 10GBit? I don't know where I should start debugging.
Thanks for your tips!
Edit:
When I pull a file from the freenas at the same time from all 3 proxmox hosts I get about 110MB/s for each host. Meaning the card is capable of actually transferring more than 1 GBit/s simultaniously. It just seems it is "capped" at 1GBit/s for each host. I haven't found such a setting in freenas.
My freenas Version btw is:
FreeNAS-11.1-RELEASE
and when I monitor the traffic I can see it all going through the mellanox card. It shows up to 3G on the monitor.
Last edited: