Problem with 10G Chelsio card

[KenY21]

Hi,

I recently bought 10G Network Kit (2x Chelsio N320E 10Gbit + 1x3m SFP+ 10GBe Cisco cable) for my FreeNAS box (Supermicro H8SCM-F).
At the moment my router (MikroTik RB4011iGS+RM) has one SFP+ port so tried to connect both, so I could backup few of my computers at the same time through the RJ45 gigabit ports.
Unfortunately, I wasn't even able to test how many 1G ports I could use to max out my FreeNAS RAID read/write speeds because after I add 10G interface into FreeNAS GUI (cxgb0) it starts to behave strangely.

Setup:
Network - 192.168.1.0/24
Onboard 1G NIC (IPMI) - 192.168.1.2/24 (DHCP with static IP based on mac address)
Onboard 1G NIC (em0) - 192.168.1.3/24 (DHCP with static IP based on mac address)
Chelsio N320E 10Gbit NIC (cxgb0) - 192.168.1.10/24 (static IP)

After I add 10G (cxgb0) interface in FreeNAS GUI it starts to behave strangely.
I can open FreeNAS GUI using both interfaces, but only through 1G (em0) ip address I can copy files from SMB share. If I try to copy from/to FreeNAS SMB share using 10G ip address copy dialog (Windows 10) just stuck and newer starts copying. If I copy trough 1G everything works as expected.

So at that moment, I thought, that maybe FreeNAS don't like that both interfaces are in the same network (usually all setups, that I have seen in YT or blogs use 10/40G directly connected to VMware/Citrix/Proxmox hosts to provide iSCSI storage for VM's using different network), so using Supermicro IPMI console I connect to FreeNAS console and deleted both interfaces and added only 10G interface using DCHP. 10GB interface got valid DCHP address from my router, but when I try to open FreeNAS GUI it just loades forever without success. I tried to connect to SMB share. I was able to open window with all SMB shares, but if I try to copy files from the share it fails as before (copy dialog (Windows 10) just stuck and newer starts copying). What's interesting as soon, as I add 1G interface with static IP (because DHCP is used for 10G) FreeNAS GUI starts working again with 1G or 10G ip address.

As I have two of the Chelsio cards I tried both (though maybe bad hardware), tried in different PCI slots and even upgraded to FreeNAS-11.2-RC2, but without luck. My ultimate goal would be to use only one 10G interface for all the data transfers, VM, plugins and accessing FreeNAS GUI and 1G interface keep only for IPMI access, both in the same network - 192.168.1.0/24.
Is it possible? What could cause this strange behavior? I would greatly appreciate any ideas or input.

 

[Elliot Dierksen]

The NIC's need to be on different IP networks. If you want to use 10G, I would not connect the 1G NIC's and see how things perform. Routing tables are global, and you have 2 NIC's on the IP network. It may be load balancing between the two of the even though one NIC should be faster than the other. Disconnect em0, reboot, and try your tests again. Then we can troubleshoot from there if necessary.

 

[KenY21]

The NIC's need to be on different IP networks. If you want to use 10G, I would not connect the 1G NIC's and see how things perform. Routing tables are global, and you have 2 NIC's on the IP network. It may be load balancing between the two of the even though one NIC should be faster than the other. Disconnect em0, reboot, and try your tests again. Then we can troubleshoot from there if necessary.

Steps I took:

1. Login in GUI --> Shutdown VM --> Un-check "Start on boot" as this VM used em0 interface
2. Login in Supermicro IPMI console --> Directly trough FreeNAS console deleted only interface (em0)
3. Shutdown FreeNAS box --> Un-plug all network cables from FreeNAS except 10G SFP+ cable (cxgb0) and RJ45 for IPMI management (192.168.1.2/24)
4. Turn on --> FreeNAS using DHCP added cxgb0 interface with DHCP address (192.168.1.119/24)
5. Ping from the router (192.168.1.1/24) - success
6. Ping from my PC (192.168.1.200/24) - success
7. Open FreeNAS GUI - fail (continuously loads login page without success)
8. Open SMB share - success
9. Copy some files from SMB share - fail (hangs up on "0% complete")

Other ideas?

 

[melloa]

1. Login in GUI --> Shutdown VM --> Un-check "Start on boot" as this VM used em0 interface

VM? What hypervisor?

2. Login in Supermicro IPMI console --> Directly trough FreeNAS console deleted only interface (em0)

I might have missed ..,. what your server hardware?

3. Shutdown FreeNAS box --> Un-plug all network cables from FreeNAS except 10G SFP+ cable (cxgb0) and RJ45 for IPMI management (192.168.1.2/24)

Depends how you have set it up.

Just to give you an idea, my ESXi has one 10Gb card and 3 1Gb on board interface. Using VM you can add the vSwitch (10Gb or 1Gb) to it and configure FreeNAS interface from the console to use it.

My FreeBSD has one dual 10Gb Chelsio t320: One assigned to the server it self, so my samba and NFS exports will use that and the other assigned to the bhyve vSwitch, so the VMs will use the other one.

7. Open FreeNAS GUI - fail (continuously loads login page without success)

FN version? ifconfig from it ...

9. Copy some files from SMB share - fail (hangs up on "0% complete")

testparm please ...

 

[KenY21]

VM? What hypervisor?

FreeNAS itself (bhyve)

I might have missed ..,. what your server hardware?

MB: Supermicro H8SCM-F
CPU: AMD Opteron(tm) Processor 4174 HE
RAM: 24GB ECC
NIC (IPMI): Integrated IPMI 2.0 + KVM with dedicated LAN
NIC (2x1G): Two Intel® 82574L controllers
NIC (10G): Chelsio N320E
HDD Pool1: 2x2TB WD 7200 in MIRROR
HDD Pool2: 2x1TB WD 7200 in MIRROR

FN version? ifconfig from it ...

FN version:
Tried FreeNAS-11.1-U6
In a testing process upgraded to FreeNAS-11.2-RC2

ifconfig:

Code:

% ifconfig
cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS																			 UM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:0c:31:74
		hwaddr 00:07:43:0c:31:74
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet none
		status: no carrier
cxgb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS																			 UM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:0c:31:75
		hwaddr 00:07:43:0c:31:75
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet none
		status: no carrier
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 150																			 0
		options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
		ether 00:25:90:57:ef:5c
		hwaddr 00:25:90:57:ef:5c
		inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC																			 >
		ether 00:25:90:57:ef:5d
		hwaddr 00:25:90:57:ef:5d
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:3e:9f:bf:18:00
		nd6 options=1<PERFORMNUD>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 9 priority 128 path cost 2000000
		member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 8 priority 128 path cost 2000
		member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000
		member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 3 priority 128 path cost 20000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu																			  1500
		description: associated with jail: transmission
		options=8<VLAN_MTU>
		ether 02:ff:60:03:aa:46
		hwaddr 02:24:10:00:07:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu																			  1500
		description: associated with jail: plex
		options=8<VLAN_MTU>
		ether 02:ff:60:14:fa:09
		hwaddr 02:24:10:00:08:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 15																			 00
		options=80000<LINKSTATE>
		ether 00:bd:45:43:ff:00
		hwaddr 00:bd:45:43:ff:00
		inet 192.168.1.120 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=1<PERFORMNUD>
		media: Ethernet autoselect
		status: active
		groups: tap
		Opened by PID 5748

testparm please ...

Code:

% testparm
Load smb config files from /usr/local/etc/smb4.conf
Processing section "[Downloads]"
Processing section "[FreeNAS]"
Processing section "[LiveCD]"
Loaded services file OK.
Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

# Global parameters
[global]
		deadtime = 15
		disable spoolss = Yes
		dns proxy = No
		dos charset = CP437
		hostname lookups = Yes
		kernel change notify = No
		lm announce = Yes
		load printers = No
		logging = file
		map to guest = Bad User
		max log size = 51200
		max open files = 706556
		nsupdate command = /usr/local/bin/samba-nsupdate -g
		obey pam restrictions = Yes
		panic action = /usr/local/libexec/samba/samba-backtrace
		printcap name = /dev/null
		security = USER
		server min protocol = SMB2_02
		server role = standalone server
		server string = nas.xxx.xxx.lv
		time server = Yes
		workgroup = XXX.XXX.LV
		idmap config *: range = 90000001-100000000
		idmap config * : backend = tdb
		acl allow execute always = Yes
		create mask = 0666
		directory mask = 0777
		directory name cache size = 0
		dos filemode = Yes
		ea support = Yes
		store dos attributes = Yes
		strict locking = No


[Downloads]
		path = "/mnt/FreeNAS/Downloads"
		read only = No
		veto files = /.snapshot/.windows/.mac/.zfs/
		vfs objects = zfs_space zfsacl streams_xattr
		zfsacl:acesort = dontcare
		nfs4:chown = true
		nfs4:acedup = merge
		nfs4:mode = special


[FreeNAS]
		path = "/mnt/FreeNAS"
		read only = No
		veto files = /.snapshot/.windows/.mac/.zfs/
		vfs objects = zfs_space zfsacl streams_xattr
		zfsacl:acesort = dontcare
		nfs4:chown = true
		nfs4:acedup = merge
		nfs4:mode = special


[LiveCD]
		path = "/mnt/VirtualMachines/LiveCD"
		read only = No
		veto files = /.snapshot/.windows/.mac/.zfs/
		vfs objects = zfs_space zfsacl streams_xattr
		zfsacl:acesort = dontcare
		nfs4:chown = true
		nfs4:acedup = merge
		nfs4:mode = special

 

[Elliot Dierksen]

Code:

cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:0c:31:74
		hwaddr 00:07:43:0c:31:74
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet none
		status: no carrier
cxgb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:0c:31:75
		hwaddr 00:07:43:0c:31:75
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet none
		status: no carrier

You have no carrier (physically not up) on either port of your Chelsio NIC.

 

[KenY21]

You have no carrier (physically not up) on either port of your Chelsio NIC.

I did "ifconfig" when I have already rolled back to 1G. I disconnected SFP+ cable and configured my RJ45 again, that's why it shows, that there isn't any physical connection, if it helps I will reconfigure it once more after I get back from work and run "ifconfig" again.

 

[Elliot Dierksen]

I did "ifconfig" when I have already rolled back to 1G. I disconnected SFP+ cable and configured my RJ45 again, that's why it shows, that there isn't any physical connection, if it helps I will reconfigure it once more after I get back from work and run "ifconfig" again.

The other thing is that neither of the cxgb ports are part of your bridge0 interface. That would cause the jails to get cut off.

 

[kdragon75]

FreeNAS networking is a hot mess... Under System > General look for WebGUI ipv4 Address and set this to 0.0.0.0 before removing your 1gb interface. This will allow the WebGUI to bind/listen on all local addresses. If this solves it, we need to file a bug report.

 

[KenY21]

The other thing is that neither of the cxgb ports are part of your bridge0 interface. That would cause the jails to get cut off.

Here is ifconfig with only 10G interface configured:

Code:

cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=2c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
		ether 00:07:43:0c:31:74
		hwaddr 00:07:43:0c:31:74
		inet 192.168.1.119 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet 10Gbase-Twinax <full-duplex>
		status: active
cxgb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:0c:31:75
		hwaddr 00:07:43:0c:31:75
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet none
		status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
		ether 00:25:90:57:ef:5c
		hwaddr 00:25:90:57:ef:5c							 
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
		ether 00:25:90:57:ef:5d
		hwaddr 00:25:90:57:ef:5d
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:3e:9f:bf:18:00
		nd6 options=1<PERFORMNUD>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0													
		member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 8 priority 128 path cost 2000
		member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000
		member: cxgb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 2000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: associated with jail: transmission
		options=8<VLAN_MTU>
		ether 02:ff:60:03:aa:46
		hwaddr 02:24:10:00:07:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: associated with jail: plex
		options=8<VLAN_MTU>
		ether 02:ff:60:14:fa:09
		hwaddr 02:24:10:00:08:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair

cxbg0 is part of the bridge0, so thats not a problem.
Here I compared ifconfig setting with em0 connected and with cxbg0 connected. With cxbg0 there is now tap0 interface. Is tap0 for my only VM? If yes, then that explains why there is no while cxbg0 is connected.

 

[KenY21]

FreeNAS networking is a hot mess... Under System > General look for WebGUI ipv4 Address and set this to 0.0.0.0 before removing your 1gb interface. This will allow the WebGUI to bind/listen on all local addresses. If this solves it, we need to file a bug report.

Thank you, kdragon75 for an idea, but I thought about that setting already. As far, as I know, when em0 is connected WebGUI ipv4 is set to 0.0.0.0, I don't know what value is there while cxbg0 is connected, because I can't access GUI and wasn't able to find console command to see that value. And just for a sanity check, I did disable all my Firewall Drop rules, just to make sure, that something isn't blocking it.

 

[Elliot Dierksen]

cxbg0 is part of the bridge0, so thats not a problem

If you are going to bridge all those interfaces together, then the IP should be assigned to the bridge0 and not one of the physicals. That could cause all kinds of wonky behavior. Also as @kdragon75 pointed out, there are some issues with the services being bound to particular IP's that aren't there later. You can save your existing config to be able to revert, but my suggestion would be to reset to factory defaults from the console and start over. I would also strongly suggest that you give FreeNAS a fixed IP and only use cxgb0 to start. If that resolves your GUI and SMB issues, then you can move on to the jails/VM's. I really wanted to stress that if all the things that are part of the bridge0 interface want to access FreeNAS via a single IP, that IP MUST assigned to bridge0 and not to a physical interface.

Edit: If you do reset the config, DO NOT run the config wizard. That would wipe your disks. Exit the wizard, and then set the networking configuration from the console. It will find your existing pools automatically.

 

[KenY21]

I would also strongly suggest that you give FreeNAS a fixed IP and only use cxgb0 to start.
I really wanted to stress that if all the things that are part of the bridge0 interface want to access FreeNAS via a single IP, that IP MUST assigned to bridge0 and not to a physical interface.

Thank you for your response Elliot, but I'm bit confused.
So my setup is pretty simple:
1) FreeNAS Box
2) Two RAIDS
3) Few SMB shares
4) Two Jails (Plex and Transmission)
5) One Ubuntu VM using Bhyve.

What I want is to use only 10G interface:
1) To access my SMB shares
2) FreeNAS GUI
3) Provide internet connection to my two jails
4) Provide internet connection to my single VM

So should I use fixed IP for cxbg0 or assign single (static) IP to bridge0? Or assign static ip for cxbg0 and different IP for bridge0?
How can I assign IP to bridge0? I tried to google command but without luck.

 

[Elliot Dierksen]

So should I use fixed IP for cxbg0 or assign single (static) IP to bridge0? Or assign static IP for cxbg0 and different IP for bridge0?

If you are going to do a bridge, then you must assign the IP to bridge0. You may have to do that with a startup script to create the bridge and assign the IP address. I can't help you there as I only use my FreeNAS for storage. I do the virtualization on separate ESXi boxes. Just as a test, try resetting the config and assigning a static IP to cxgb0 (with no bridge configured). See if you can access the SMB shares and GUI from an external client. Once that part is resolved, then we can move on to more complex issues.

 

[melloa]

So should I use fixed IP for cxbg0 or assign single (static) IP to bridge0? Or assign static IP for cxbg0 and different IP for bridge0?
How can I assign IP to bridge0? I tried to google command but without luck.

You won't max out a 10Gb connection with a home setup in most cases, so why complicate your life? Get only the 10Gb NIC for all...

 

[KenY21]

If you are going to do a bridge, then you must assign the IP to bridge0. You may have to do that with a startup script to create the bridge and assign the IP address. I can't help you there as I only use my FreeNAS for storage. I do the virtualization on separate ESXi boxes. Just as a test, try resetting the config and assigning a static IP to cxgb0 (with no bridge configured). See if you can access the SMB shares and GUI from an external client. Once that part is resolved, then we can move on to more complex issues.

Well, at the moment I don't even worry about jails/VM, the main focus is to find out why I can't open Web GUI and why using SMB I can open shares, but when I try to copy files it stucks on 0%. And as my em0 works fine I really don't see how reinstalling FreeNAS would help in this case. I guess if that's my last option, then I will do it...

Yesterday I tried to copy files using WinSCP because SSH (sort-off) works with cxgb0 interface. I started to copy some files and after few seconds it failed as well.

Then I started to look through /var/logs and found out, that if I tried to "cat" some bigger file connection would drop, but if I "cat'ed" some small file it would output information. I added some information from log files, that I was able to get when I tried to access Web GUI.

1) /var/logs/auth.log

Code:

Nov 27 11:21:31 nas sshd[15091]: Accepted password for XXX from 192.168.1.231 port 64003 ssh2
Nov 27 11:25:29 nas sshd[2511]: Received signal 15; terminating.
Nov 27 11:28:31 nas sshd[2519]: Server listening on :: port 22.
Nov 27 11:28:31 nas sshd[2519]: Server listening on 0.0.0.0 port 22.
Nov 27 11:30:23 nas sshd[5048]: Accepted password for XXX from 192.168.1.231 port 63758 ssh2
Nov 27 11:37:55 nas sshd[5068]: Timeout, client not responding.
Nov 27 11:38:28 nas sshd[5950]: Accepted password for XXX from 192.168.1.231 port 64295 ssh2
Nov 27 11:43:17 nas sshd[5970]: Timeout, client not responding.
Nov 27 12:02:15 nas sshd[8477]: Accepted password for XXX from 192.168.1.231 port 49899 ssh2
Nov 27 12:04:40 nas sshd[8698]: Accepted password for XXX from 192.168.1.231 port 50034 ssh2
Nov 27 12:05:17 nas sshd[8781]: Accepted password for XXX from 192.168.1.231 port 50085 ssh2
Nov 27 12:05:54 nas sshd[8479]: Timeout, client not responding.

2) /var/log/cron

Code:

Nov 27 12:01:00 nas /usr/sbin/cron[8348]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:02:00 nas /usr/sbin/cron[8422]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:03:00 nas /usr/sbin/cron[8518]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:04:00 nas /usr/sbin/cron[8610]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:05:00 nas /usr/sbin/cron[8721]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:05:00 nas /usr/sbin/cron[8722]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
Nov 27 12:06:00 nas /usr/sbin/cron[8806]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:07:00 nas /usr/sbin/cron[8880]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:08:00 nas /usr/sbin/cron[8965]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:09:00 nas /usr/sbin/cron[9055]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:10:00 nas /usr/sbin/cron[9154]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:10:00 nas /usr/sbin/cron[9156]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
Nov 27 12:11:00 nas /usr/sbin/cron[9238]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:11:00 nas /usr/sbin/cron[9239]: (operator) CMD (/usr/libexec/save-entropy > /dev/null 2>&1)
Nov 27 12:12:00 nas /usr/sbin/cron[9455]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:13:00 nas /usr/sbin/cron[9540]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:14:00 nas /usr/sbin/cron[9634]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:15:00 nas /usr/sbin/cron[9732]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:15:00 nas /usr/sbin/cron[9735]: (root) CMD (/usr/libexec/atrun > /dev/null 2>&1)
Nov 27 12:16:00 nas /usr/sbin/cron[9809]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
Nov 27 12:17:00 nas /usr/sbin/cron[9883]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)

3) /var/log/debug.log

Code:

Nov 27 12:00:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:01:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:02:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:02:20 nas sudo:   XXX: TTY=pts/0 ; PWD=/nonexistent ; USER=root ; COMMAND=/usr/bin/su
Nov 27 12:03:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:04:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:04:45 nas sudo:   XXXX: TTY=pts/1 ; PWD=/nonexistent ; USER=root ; COMMAND=/usr/bin/su
Nov 27 12:05:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:06:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:07:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:08:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''
Nov 27 12:09:01 nas /autosnap.py: [ws4py:360] Closing message received (1000) 'b'''

4) /var/log/messages

Code:

Nov 27 11:28:30 nas ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 2> on usbus4
Nov 27 11:28:30 nas ums0: 3 buttons and [Z] coordinates ID=0
Nov 27 11:28:31 nas ntpd[2600]: ntpd 4.2.8p11-a (1): Starting
Nov 27 11:28:35 nas root: /etc/rc: WARNING: failed precmd routine for minio
Nov 27 11:28:40 nas bridge0: Ethernet address: 02:3e:9f:bf:18:00
Nov 27 11:28:40 nas kernel: bridge0: link state changed to UP
Nov 27 11:28:40 nas kernel: bridge0: link state changed to UP
Nov 27 11:28:40 nas kernel: cxgb0: promiscuous mode enabled
Nov 27 11:28:40 nas epair0a: Ethernet address: 02:24:10:00:07:0a
Nov 27 11:28:40 nas epair0b: Ethernet address: 02:24:10:00:08:0b
Nov 27 11:28:40 nas kernel: epair0a: link state changed to UP
Nov 27 11:28:40 nas kernel: epair0a: link state changed to UP
Nov 27 11:28:40 nas kernel: epair0b: link state changed to UP
Nov 27 11:28:40 nas kernel: epair0b: link state changed to UP
Nov 27 11:28:40 nas kernel: epair0a: changing name to 'vnet0:1'
Nov 27 11:28:41 nas cxgb0: tso4 disabled due to -txcsum.
Nov 27 11:28:41 nas cxgb0: tso6 disabled due to -txcsum6.
Nov 27 11:28:41 nas cxgb0: enable txcsum first.
Nov 27 11:28:41 nas bridge0: error setting capabilities on cxgb0: 35
Nov 27 11:28:41 nas kernel: vnet0:1: promiscuous mode enabled
Nov 27 11:28:52 nas epair1a: Ethernet address: 02:24:10:00:08:0a
Nov 27 11:28:52 nas epair1b: Ethernet address: 02:24:10:00:09:0b
Nov 27 11:28:52 nas kernel: epair1a: link state changed to UP
Nov 27 11:28:52 nas kernel: epair1a: link state changed to UP
Nov 27 11:28:52 nas kernel: epair1b: link state changed to UP
Nov 27 11:28:52 nas kernel: epair1b: link state changed to UP
Nov 27 11:28:52 nas kernel: epair1a: changing name to 'vnet0:2'
Nov 27 11:28:52 nas kernel: epair1b: changing name to 'epair0b'
Nov 27 11:28:52 nas kernel: vnet0:2: promiscuous mode enabled

Maybe this somehow helps.

 

[KenY21]

You won't max out a 10Gb connection with a home setup in most cases, so why complicate your life? Get only the 10Gb NIC for all...

Sorry, I didn't cache the idea.

 

[melloa]

Sorry, I didn't cache the idea.

I thought I saw you trying to use your 1Gb NICs and the 10Gb NIC. The thought was to use only the 10Gb NIC.

why I can't open Web GUI and why using SMB I can open shares, but when I try to copy files it stucks on 0%.

Also that time out happened with me when I had my workstation set to MTU 1500 and the server set to MTU 9000. I wasn't able to copy, test iperf, etc. I'm using MTU 1500 for both.

During my tests I removed the interfaces from the console and re-added only the Chelsio and was able to access the GUI without any problems.

 

[KenY21]

Finally, with friends help figure it out...
As I said in my first post, on the other side I had Mikrotik RB4011 router with SFP+ port.
Digging into compatibility requirements friend pointed out, that SFP+ port only works with active DAC modules (https://wiki.mikrotik.com/wiki/MikroTik_SFP_module_compatibility_table - RB4011 -- Only active Direct Attach Cables are supported. MikroTik S+DA0001/S+DA0003 passive DAC are not supported.) and you guessed it - in my case I had Cisco SFP-H10GB-CU (https://shop.fiber24.net/index.php/...CO-DAC-AOC/a-F24-CI-SFP-H10GB-CU?sslCall=true - Cisco Direct Attached Copper Cable, Twinax, passive, up to 10.5 Gb/s).
So a friend gave me spear Mikrotik 1M DAC active modules and Supermicro 10G network card to troubleshoot if FreeNAS, Chelsio NIC, Cisco DAC or Mikrotik router is at fault. As soon, as I switched from Cisco passive to Mikrotik active DAC everything started to work. :)
So thank everyone for help and ideas. As I don't face fiber in my day-to-day life it was a good lesson, that even if you don't use compatible wire it could partly work (SSH, ping, DHCP, loading Web GUI)...