TrueNAS 12 hangs/freezed

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
Hello,

I am running a TrueNAS over Hyper-V to storage kubernetes volumes. I am aware this is not a best architecture, we are already installing TrueNAS 13 on a Dell R610 + Perc310 IT mode.

Our tank pool is 95% full. I was logged via SSH when it suddenly freezed. UI is not working too.
I went to hyper-V console and I saw this message:

1662736266171.jpeg




Any suggestions? Thanks
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
You're getting disk timeouts, likely the result of several compounding issues:

1. Your pool is almost completely full (458M free!) so performance will be significantly impacted (free space fragmentation, metaslab allocator working extremely hard) - please post zpool list output in CODE tags, and try to curtail or stop additional writes as filling your pool to 100% will be extremely bad.

Code:
tank                                                      539G   458M      104K  /mnt/tank


2. You appear to be presenting a single virtual disk to TrueNAS, and the number of zvols/containers may be flooding the I/O queue (which is made worse by the almost-full status) of a single vdev.

Please describe your Hyper-V host hardware and TrueNAS guest configuration as much as possible.

I am running a TrueNAS over Hyper-V to storage kubernetes volumes. I am aware this is not a best architecture, we are already installing TrueNAS 13 on a Dell R610 + Perc310 IT mode.

Sounds like you're taking steps to rectify this problem already. Make sure you flash proper LSI firmware onto the H310 as the stock queue depth is very small (25). This may require special steps if the H310 is a "Mini/Mono" card.

Please post other hardware details if you can as well so we can investigate RAM/CPU amounts (as well as ensure that you're building in a sufficient amount of free space!)
 

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
Hello,

the problem was caused because our pool tank reached 100% ( no free disk space left). A team mate solved it and it is working fine.

I will post more information about our new server at Dell R610 asap.

Thanks
 

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
Hello. My TrueNAS 12 freezed again, I cant connect UI.
1663336949180.png


I connected via ssh:

Code:
login as: root
root@172.30.3.245's password:
Last login: Fri Sep 16 10:47:56 2022 from 172.30.3.180
FreeBSD 12.2-RELEASE-p12 ec84e0c52a1(HEAD) TRUENAS

        TrueNAS (c) 2009-2022, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license w                                                               ith some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go                                                                here:
        http://truenas.com
Welcome to TrueNAS
Traceback (most recent call last):
  File "/usr/local/sbin/hactl", line 171, in <module>
    main(args.command, args.q)
  File "/usr/local/sbin/hactl", line 17, in main
    client = Client()
  File "/usr/local/lib/python3.9/site-packages/middlewared/client                                                               /client.py", line 283, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.9/site-packages/middlewared/client                                                               /client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.9/site-packages/ws4py/client/__ini                                                               t__.py", line 216, in connect
    self.sock.connect(self.bind_addr)
ConnectionRefusedError: [Errno 61] Connection refused

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

root@truenas[~]#


root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Mon Sep 12 03:45:05 2022
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 00:46:52 with 0 errors on Sun Aug 14 00:46:57 2022
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE       0     0     0
          gptid/4806796c-8f06-11eb-a787-00155d032ec0  ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# tail -f /var/log/messages


Sep 16 10:59:53 truenas kernel: sonewconn: pcb 0xfffff8000ef42400: Listen queue overflow: 193 already in queue awaiting acceptan
ce (36 occurrences)
Sep 16 10:59:53 truenas kernel: pid 72160 (collectd), jid 0, uid 0: exited on signal 11 (core dumped)
Sep 16 11:00:53 truenas kernel: sonewconn: pcb 0xfffff80013aad7a0: Listen queue overflow: 193 already in queue awaiting acceptan
ce (30 occurrences)
Sep 16 11:00:54 truenas 1 2022-09-16T11:00:54.366256-03:00 truenas.triadesolucoes.com.br collectd 72264 - - Traceback (most rece
nt call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 43, in init
    with Client() as c:
  File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 283, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.9/site-packages/ws4py/client/__init__.py", line 216, in connect
    self.sock.connect(self.bind_addr)
ConnectionRefusedError: [Errno 61] Connection refused
Sep 16 11:04:58 truenas kernel: pid 72740 (collectd), jid 0, uid 0: exited on signal 11 (core dumped)

Sep 16 11:41:29 truenas kernel: sonewconn: pcb 0xfffff80013aad7a0: Listen queue overflow: 193 already in queue awaiting acceptance (13 occurrences)




Any suggestions?
 
Last edited:

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
Code:
root@truenas[~]# sysctl kern.ipc.somaxconn=512
kern.ipc.somaxconn: 128 -> 512

root@truenas[/var/db/system/cores]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
tcp6  0/0/128                          *.80
tcp4  0/0/128                          *.80
tcp6  0/0/128                          *.443
tcp4  0/0/128                          *.443
tcp4  0/0/128                          *.22
tcp6  0/0/128                          *.22
tcp4  0/0/128                          *.199
tcp4  0/0/5                            172.30.3.245.5357
tcp4  0/0/128                          *.139
tcp4  0/0/128                          *.445
tcp6  0/0/128                          *.139
tcp6  0/0/128                          *.445
tcp4  0/0/128                          *.649
tcp6  0/0/128                          *.782
tcp6  0/0/128                          *.2049
tcp4  0/0/128                          *.824
tcp6  0/0/128                          *.824
tcp4  0/0/128                          *.2049
tcp4  0/0/128                          *.684
tcp6  0/0/128                          *.684
tcp4  0/0/128                          *.3260
tcp4  0/0/128                          *.111
tcp6  0/0/128                          *.111
tcp4  193/0/128                        *.6000
unix  0/0/128                          /var/db/syslog-ng.ctl
unix  0/0/128                          /var/run/avahi-daemon/socket
unix  0/0/128                          /var/run/dbus/system_bus_socket
unix  0/0/128                          /var/run/rrdcached.sock
unix  0/0/5                            /var/agentx/master
unix  0/0/5                            /var/run/snmpd.sock
unix  0/0/128                          /var/run/samba4/ncalrpc/np/winreg
unix  0/0/128                          /var/run/samba4/ncalrpc/DEFAULT
unix  0/0/128                          /var/run/samba4/ncalrpc/np/srvsvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netlogon
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsarpc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsass
unix  0/0/128                          /var/run/samba4/ncalrpc/np/samr
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netdfs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/wkssvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/svcctl
unix  0/0/128                          /var/run/samba4/ncalrpc/np/ntsvcs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/plugplay
unix  0/0/128                          /var/run/samba4/ncalrpc/np/eventlog
unix  0/0/128                          /var/run/samba4/ncalrpc/np/initshutdown
unix  0/0/128                          /var/db/system/samba4/winbindd_privileged/pipe
unix  0/0/128                          /var/run/samba4/winbindd/pipe
unix  0/0/128                          /var/run/rpcbind.sock
unix  193/0/128                        /var/run/middlewared.sock
unix  0/0/128                          /var/run/middlewared-internal.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe
root@truenas[/var/db/system/cores]#





How Can I increase kern.ipc.somaxconn?
 

Attachments

  • 1663342093254.png
    1663342093254.png
    6.9 KB · Views: 74
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
How Can I increase kern.ipc.somaxconn?
Seems to me that you're using LOADER type tunables for a SYSCTL value... seems a simple solution to that... use the SYSCTL type instead.
 

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
Seems to me that you're using LOADER type tunables for a SYSCTL value... seems a simple solution to that... use the SYSCTL type instead.
Hello, thanks for your reply. I did this:
Code:
sysctl kern.ipc.somaxconn=512


But it didn´t worked.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I did this:
Code:
sysctl kern.ipc.somaxconn=512
And after that, when you type:

sysctl kern.ipc.somaxconn does it show that it's set or not?
 

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
It seems to be working. Thanks
But it only sets to port 80 and 443. How do I set it to port 6000?

But now my UI interface hangs again.

1664291684783.png

Code:
root@truenas[~]# service middlewared status
middlewared is running as pid 518
root@truenas[~]#

root@truenas[/etc]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
tcp6  0/0/4096                         *.80
tcp4  0/0/4096                         *.80
tcp6  0/0/4096                         *.443
tcp4  0/0/4096                         *.443
tcp4  0/0/128                          *.6000
tcp4  0/0/128                          *.22
tcp6  0/0/128                          *.22
tcp4  0/0/128                          *.199
tcp4  0/0/5                            172.30.3.245.5357
tcp4  0/0/128                          *.139
tcp4  0/0/128                          *.445
tcp6  0/0/128                          *.139
tcp6  0/0/128                          *.445
tcp4  0/0/128                          *.956
tcp6  0/0/128                          *.817
tcp6  0/0/128                          *.2049
tcp4  0/0/128                          *.938
tcp6  0/0/128                          *.938
tcp4  0/0/128                          *.2049
tcp4  0/0/128                          *.838
tcp6  0/0/128                          *.838
tcp4  0/0/128                          *.3260
tcp4  0/0/128                          *.111
tcp6  0/0/128                          *.111
tcp4  193/0/128                        *.6000
unix  0/0/128                          /var/run/middlewared.sock
unix  0/0/128                          /var/run/middlewared-internal.sock
unix  0/0/128                          /var/db/syslog-ng.ctl
unix  0/0/128                          /var/run/avahi-daemon/socket
unix  0/0/128                          /var/run/dbus/system_bus_socket
unix  0/0/128                          /var/run/rrdcached.sock
unix  0/0/5                            /var/agentx/master
unix  0/0/5                            /var/run/snmpd.sock
unix  0/0/128                          /var/run/samba4/ncalrpc/np/winreg
unix  0/0/128                          /var/run/samba4/ncalrpc/DEFAULT
unix  0/0/128                          /var/run/samba4/ncalrpc/np/srvsvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netlogon
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsarpc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsass
unix  0/0/128                          /var/run/samba4/ncalrpc/np/samr
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netdfs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/wkssvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/svcctl
unix  0/0/128                          /var/run/samba4/ncalrpc/np/ntsvcs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/plugplay
unix  0/0/128                          /var/run/samba4/ncalrpc/np/eventlog
unix  0/0/128                          /var/run/samba4/ncalrpc/np/initshutdown
unix  0/0/128                          /var/db/system/samba4/winbindd_privileged/pipe
unix  0/0/128                          /var/run/samba4/winbindd/pipe
unix  0/0/128                          /var/run/rpcbind.sock
unix  152/0/128                        /var/run/middlewared.sock
unix  0/0/128                          /var/run/middlewared-internal.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe
root@truenas[/etc]#

root@truenas[~]# sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 4096
root@truenas[~]#


 
Last edited:

sekroots

Explorer
Joined
Feb 28, 2022
Messages
61
I change my tunables to this:
1664308167109.png



But I still cannot set max connections:

Code:
root@truenas[~]# netstat -Lan
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
tcp4  0/0/512                          *.3260
tcp6  0/0/128                          *.80
tcp4  0/0/128                          *.80
tcp6  0/0/128                          *.443
tcp4  0/0/128                          *.443
tcp4  0/0/128                          *.22
tcp6  0/0/128                          *.22
tcp4  0/0/128                          *.199
tcp4  0/0/5                            172.30.3.245.5357
tcp4  0/0/128                          *.139
tcp4  0/0/128                          *.445
tcp6  0/0/128                          *.139
tcp6  0/0/128                          *.445
tcp4  0/0/128                          *.811
tcp6  0/0/128                          *.706
tcp4  0/0/128                          *.938
tcp6  0/0/128                          *.938
tcp6  0/0/128                          *.2049
tcp4  0/0/128                          *.2049
tcp4  0/0/128                          *.1017
tcp6  0/0/128                          *.1017
tcp4  0/0/128                          *.111
tcp6  0/0/128                          *.111
tcp4  0/0/128                          *.6000
unix  0/0/128                          /var/run/avahi-daemon/socket
unix  0/0/128                          /var/run/dbus/system_bus_socket
unix  0/0/128                          /var/run/rrdcached.sock
unix  0/0/5                            /var/agentx/master
unix  0/0/5                            /var/run/snmpd.sock
unix  0/0/128                          /var/run/samba4/ncalrpc/np/winreg
unix  0/0/128                          /var/run/samba4/ncalrpc/DEFAULT
unix  0/0/128                          /var/run/samba4/ncalrpc/np/srvsvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netlogon
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsarpc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/lsass
unix  0/0/128                          /var/run/samba4/ncalrpc/np/samr
unix  0/0/128                          /var/run/samba4/ncalrpc/np/netdfs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/wkssvc
unix  0/0/128                          /var/run/samba4/ncalrpc/np/svcctl
unix  0/0/128                          /var/run/samba4/ncalrpc/np/ntsvcs
unix  0/0/128                          /var/run/samba4/ncalrpc/np/plugplay
unix  0/0/128                          /var/run/samba4/ncalrpc/np/eventlog
unix  0/0/128                          /var/run/samba4/ncalrpc/np/initshutdown
unix  0/0/128                          /var/db/system/samba4/winbindd_privileged/pipe
unix  0/0/128                          /var/run/samba4/winbindd/pipe
unix  0/0/128                          /var/run/rpcbind.sock
unix  0/0/128                          /var/db/syslog-ng.ctl
unix  0/0/128                          /var/run/middlewared.sock
unix  0/0/128                          /var/run/middlewared-internal.sock
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe
root@truenas[~]#
 
Top