Slow ssh/network Performance on 11.2-Beta3

[lulu]

Hello,

I've installed a fresh install of 11.2-BETA3 and I've ended up with freezing ssh I made a little video to show it.

Earlier I've had 11.1 on the same hardware and the same network and it operated normally.

Here is the video

 

[wblock]

I do not have speakers on this desktop and could not tell what was going on in that video. Possibly it was pauses in SSH output, but it was difficult to tell. Please attempt to describe the problem in text.

 

[lulu]

When I’m inputting something to the ssh it freezes for 1-5 seconds and it is incredibly slow.
In the video, I was constantly inputting something and you can see that it froze for several seconds and then it inputted all at character at once.
It occurs via SSH and GUI shell.
It works normally when directly inputted to nas with keyboard connected to it.

 

[wblock]

What are the hardware specifications of the machine running FreeNAS? Specifically, how much RAM does it have?

 

[lulu]

@wblock It is build from a gaming pc

I7-3960x
16gb ddr 3 ram

I was running 11.1 on the same machine and it was operating super fast

 

[lulu]

this is from logging into ssh in debug3 mode

Code:

OpenSSH_7.8p1, LibreSSL 2.7.3

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 48: Applying options for *

debug2: resolve_canonicalize: hostname 192.168.1.24 is address

debug2: ssh_connect_direct

debug1: Connecting to 192.168.1.24 [192.168.1.24] port 22.

debug1: Connection established.

debug1: identity file /Users/lucjangrzesik/.ssh/id_rsa type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_rsa-cert type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_dsa type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_dsa-cert type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_ecdsa type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_ecdsa-cert type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_ed25519 type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_ed25519-cert type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_xmss type -1

debug1: identity file /Users/lucjangrzesik/.ssh/id_xmss-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_7.8

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5-hpn14v5

debug1: match: OpenSSH_7.5-hpn14v5 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to 192.168.1.24:22 as 'lulu'

debug3: hostkeys_foreach: reading file "/Users/lucjangrzesik/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /Users/lucjangrzesik/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from 192.168.1.24

debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c

debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none,zlib@openssh.com,zlib

debug2: compression stoc: none,zlib@openssh.com,zlib

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug2: peer server KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1

debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,none

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none

debug2: compression stoc: none

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug1: kex: algorithm: curve25519-sha256

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug3: send packet: type 30

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug3: receive packet: type 31

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:D7dBr9EFsbaetw0buZL5LmPSglufp4Sa9+SdJrnclBk

debug3: hostkeys_foreach: reading file "/Users/lucjangrzesik/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /Users/lucjangrzesik/.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from 192.168.1.24

debug1: Host '192.168.1.24' is known and matches the ECDSA host key.

debug1: Found key in /Users/lucjangrzesik/.ssh/known_hosts:1

debug3: send packet: type 21

debug2: set_newkeys: mode 1

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug1: SSH2_MSG_NEWKEYS received

debug2: set_newkeys: mode 0

debug1: rekey after 134217728 blocks

debug2: key: /Users/lucjangrzesik/.ssh/id_rsa (0x0)

debug2: key: /Users/lucjangrzesik/.ssh/id_dsa (0x0)

debug2: key: /Users/lucjangrzesik/.ssh/id_ecdsa (0x0)

debug2: key: /Users/lucjangrzesik/.ssh/id_ed25519 (0x0)

debug2: key: /Users/lucjangrzesik/.ssh/id_xmss (0x0)

debug3: send packet: type 5

debug3: receive packet: type 7

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey,password

debug3: start over, passed a different list publickey,password

debug3: preferred publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Trying private key: /Users/lucjangrzesik/.ssh/id_rsa

debug3: no such identity: /Users/lucjangrzesik/.ssh/id_rsa: No such file or directory

debug1: Trying private key: /Users/lucjangrzesik/.ssh/id_dsa

debug3: no such identity: /Users/lucjangrzesik/.ssh/id_dsa: No such file or directory

debug1: Trying private key: /Users/lucjangrzesik/.ssh/id_ecdsa

debug3: no such identity: /Users/lucjangrzesik/.ssh/id_ecdsa: No such file or directory

debug1: Trying private key: /Users/lucjangrzesik/.ssh/id_ed25519

debug3: no such identity: /Users/lucjangrzesik/.ssh/id_ed25519: No such file or directory

debug1: Trying private key: /Users/lucjangrzesik/.ssh/id_xmss

debug3: no such identity: /Users/lucjangrzesik/.ssh/id_xmss: No such file or directory

debug2: we did not send a packet, disable method

debug3: authmethod_lookup password

debug3: remaining preferred: ,password

debug3: authmethod_is_enabled password

debug1: Next authentication method: password

lulu@192.168.1.24's password:

debug3: send packet: type 50

debug2: we sent a password packet, wait for reply

debug3: receive packet: type 52

debug1: Authentication succeeded (password).

Authenticated to 192.168.1.24 ([192.168.1.24]:22).

debug1: channel 0: new [client-session]

debug3: ssh_session2_open: channel_new: 0

debug2: channel 0: send open

debug3: send packet: type 90

debug1: Requesting no-more-sessions@openssh.com

debug3: send packet: type 80

debug1: Entering interactive session.

debug1: pledge: network

debug3: receive packet: type 80

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

debug3: receive packet: type 91

debug2: channel_input_open_confirmation: channel 0: callback start

debug2: fd 3 setting TCP_NODELAY

debug3: ssh_packet_set_tos: set IP_TOS 0x48

debug2: client_session2_setup: id 0

debug2: channel 0: request pty-req confirm 1

debug3: send packet: type 98

debug1: Sending environment.

debug3: Ignored env TERM_PROGRAM

debug3: Ignored env SHELL

debug3: Ignored env TERM

debug3: Ignored env TMPDIR

debug3: Ignored env Apple_PubSub_Socket_Render

debug3: Ignored env TERM_PROGRAM_VERSION

debug3: Ignored env TERM_SESSION_ID

debug3: Ignored env USER

debug3: Ignored env SSH_AUTH_SOCK

debug3: Ignored env PATH

debug3: Ignored env PWD

debug3: Ignored env XPC_FLAGS

debug3: Ignored env XPC_SERVICE_NAME

debug3: Ignored env SHLVL

debug3: Ignored env HOME

debug3: Ignored env LOGNAME

debug1: Sending env LC_CTYPE = UTF-8

debug2: channel 0: request env confirm 0

debug3: send packet: type 98

debug3: Ignored env _

debug3: Ignored env __CF_USER_TEXT_ENCODING

debug2: channel 0: request shell confirm 1

debug3: send packet: type 98

debug2: channel_input_open_confirmation: channel 0: callback done

debug2: channel 0: open confirm rwindow 0 rmax 32768

debug3: receive packet: type 99

debug2: channel_input_status_confirm: type 99 id 0

debug2: PTY allocation request accepted on channel 0

debug2: channel 0: rcvd adjust 65536

debug3: receive packet: type 99

debug2: channel_input_status_confirm: type 99 id 0

debug2: shell request accepted on channel 0

Last login: Tue Oct 16 20:58:55 2018 from 192.168.1.21

 

[lulu]

netstats also

Code:

root@freenas[/]# netstat -s		 

tcp:

11442 packets sent

6391 data packets (1366605 bytes)

923 data packets (116809 bytes) retransmitted

48 data packets unnecessarily retransmitted

0 resends initiated by MTU discovery

3431 ack-only packets (1009 delayed)

0 URG only packets

0 window probe packets

0 window update packets

778 control packets

11179 packets received

5636 acks (for 1369153 bytes)

524 duplicate acks

0 acks for unsent data

5280 packets (1180244 bytes) received in-sequence

130 completely duplicate packets (38568 bytes)

0 old duplicate packets

4 packets with some dup. data (220 bytes duped)

138 out-of-order packets (13586 bytes)

0 packets (0 bytes) of data after window

0 window probes

396 window update packets

6 packets received after close

0 discarded for bad checksums

0 discarded for bad header offset fields

0 discarded because packet too short

0 discarded due to memory problems

369 connection requests

395 connection accepts

0 bad connection attempts

0 listen queue overflows

0 ignored RSTs in the windows

764 connections established (including accepts)

723 times used RTT from hostcache

723 times used RTT variance from hostcache

37 times used slow-start threshold from hostcache

771 connections closed (including 4 drops)

27 connections updated cached RTT on close

28 connections updated cached RTT variance on close

21 connections updated cached ssthresh on close

0 embryonic connections dropped

5636 segments updated rtt (of 5093 attempts)

789 retransmit timeouts

1 connection dropped by rexmit timeout

0 persist timeouts

0 connections dropped by persist timeout

0 Connections (fin_wait_2) dropped because of timeout

0 keepalive timeouts

0 keepalive probes sent

0 connections dropped by keepalive

1367 correct ACK header predictions

2468 correct data packet header predictions

396 syncache entries added

6 retransmitted

0 dupsyn

1 dropped

395 completed

0 bucket overflow

0 cache overflow

0 reset

1 stale

0 aborted

0 badack

0 unreach

0 zone failures

397 cookies sent

0 cookies received

3 hostcache entries added

0 bucket overflow

73 SACK recovery episodes

127 segment rexmits in SACK recovery episodes

7204 byte rexmits in SACK recovery episodes

603 SACK options (SACK blocks) received

113 SACK options (SACK blocks) sent

0 SACK scoreboard overflow

0 packets with ECN CE bit set

0 packets with ECN ECT(0) bit set

0 packets with ECN ECT(1) bit set

0 successful ECN handshakes

0 times ECN reduced the congestion window

0 packets with matching signature received

0 packets with bad signature received

0 times failed to make signature due to no SA

0 times unexpected signature received

0 times no signature provided by segment

TCP connection count by state:

0 connections in CLOSED state

6 connections in LISTEN state

0 connections in SYN_SENT state

0 connections in SYN_RCVD state

8 connections in ESTABLISHED state

0 connections in CLOSE_WAIT state

0 connections in FIN_WAIT_1 state

0 connections in CLOSING state

0 connections in LAST_ACK state

0 connections in FIN_WAIT_2 state

12 connections in TIME_WAIT state

udp:

3974 datagrams received

0 with incomplete header

0 with bad data length field

0 with bad checksum

0 with no checksum

234 dropped due to no socket

79 broadcast/multicast datagrams undelivered

0 dropped due to full socket buffers

0 not for hashed pcb

3661 delivered

3713 datagrams output

0 times multicast source filter matched

ip:

15170 total packets received

0 bad header checksums

0 with size smaller than minimum

0 with data size < data length

0 with ip length > max ip packet size

0 with header length < data size

0 with data length < header length

0 with bad options

0 with incorrect version number

0 fragments received

0 fragments dropped (dup or out of space)

0 fragments dropped after timeout

0 packets reassembled ok

15167 packets for this host

0 packets for unknown/unsupported protocol

0 packets forwarded (0 packets fast forwarded)

3 packets not forwardable

16 packets received for unknown multicast group

0 redirects sent

16355 packets sent from this host

0 packets sent with fabricated ip header

0 output packets dropped due to no bufs, etc.

4 output packets discarded due to no route

0 output datagrams fragmented

0 fragments created

0 datagrams that can't be fragmented

0 tunneling packets that can't find gif

0 datagrams with bad address in header

icmp:

230 calls to icmp_error

0 errors not generated in response to an icmp message

Output histogram:

destination unreachable: 230

0 messages with bad code fields

0 messages less than the minimum length

0 messages with bad checksum

0 messages with bad length

0 multicast echo requests ignored

0 multicast timestamp requests ignored

Input histogram:

destination unreachable: 4

0 message responses generated

0 invalid return addresses

0 no return routes

ICMP address mask responses are disabled

igmp:

14 messages received

0 messages received with too few bytes

0 messages received with wrong TTL

0 messages received with bad checksum

7 V1/V2 membership queries received

0 V3 membership queries received

0 membership queries received with invalid field(s)

7 general queries received

0 group queries received

0 group-source queries received

0 group-source queries dropped

7 membership reports received

0 membership reports received with invalid field(s)

7 membership reports received for groups to which we belong

0 V3 reports received without Router Alert

872 membership reports sent

carp:

0 packets received (IPv4)

0 packets received (IPv6)

0 packets discarded for wrong TTL

0 packets shorter than header

0 discarded for bad checksums

0 discarded packets with a bad version

0 discarded because packet too short

0 discarded for bad authentication

0 discarded for bad vhid

0 discarded because of a bad address list

0 packets sent (IPv4)

0 packets sent (IPv6)

0 send failed due to mbuf memory error

arp:

1174 ARP requests sent

43 ARP replies sent

315 ARP requests received

251 ARP replies received

567 ARP packets received

1601 total packets dropped due to no ARP entry

0 ARP entrys timed out

0 Duplicate IPs seen

ip6:

8 total packets received

0 with size smaller than minimum

0 with data size < data length

0 with bad options

0 with incorrect version number

0 fragments received

0 fragments dropped (dup or out of space)

0 fragments dropped after timeout

0 fragments that exceeded limit

0 packets reassembled ok

4 packets for this host

0 packets forwarded

0 packets not forwardable

0 redirects sent

8 packets sent from this host

0 packets sent with fabricated ip header

0 output packets dropped due to no bufs, etc.

16 output packets discarded due to no route

0 output datagrams fragmented

0 fragments created

0 datagrams that can't be fragmented

0 packets that violated scope rules

0 multicast packets which we don't join

Input histogram:

UDP: 4

ICMP6: 4

Mbuf statistics:

4 one mbuf

two or more mbuf:

lo0= 4

0 one ext mbuf

0 two or more ext mbuf

0 packets whose headers are not contiguous

0 tunneling packets that can't find gif

0 packets discarded because of too many headers

5 failures of source address selection

source addresses on a non-outgoing I/F

5 addresses scope=0xf

Source addresses selection rule applied:

5 same address

icmp6:

4 calls to icmp6_error

0 errors not generated in response to an icmp6 message

0 errors not generated because of rate limitation

Output histogram:

unreach: 4

0 messages with bad code fields

0 messages < minimum length

0 bad checksums

0 messages with bad length

Input histogram:

unreach: 4

Histogram of error messages to be generated:

0 no route

0 administratively prohibited

0 beyond scope

0 address unreachable

4 port unreachable

0 packet too big

0 time exceed transit

0 time exceed reassembly

0 erroneous header field

0 unrecognized next header

0 unrecognized option

0 redirect

0 unknown

0 message responses generated

0 messages with too many ND options

0 messages with bad ND options

0 bad neighbor solicitation messages

0 bad neighbor advertisement messages

0 bad router solicitation messages

0 bad router advertisement messages

0 bad redirect messages

0 path MTU changes

rip6:

0 messages received

0 checksum calculations on inbound

0 messages with bad checksum

0 messages dropped due to no socket

0 multicast messages dropped due to no socket

0 messages dropped due to full socket buffers

0 delivered

0 datagrams output

 

[lulu]

Anyone?
I've tried reinstalling it several times the result is always the same ssh is freezing when typing for several seconds.
I've installed 11.1 and ssh right out the box is operating normally.
Please help me resolve this bug

 

[wblock]

Please enter a bug ticket at https://redmine.ixsystems.com/projects/freenas.

 

[lulu]

https://redmine.ixsystems.com/issues/52452