Middleware crashes during rsync

S

simdim

Explorer
Joined
Mar 12, 2019
Messages
75
Hi All,
I am running TrueNas Scale 22.02.1, and during manual execution of rsync from GUI, middleware crashes.
The only way to recover is to restart middleware, however this action kills running rsync.
Any one else had seen that?
 
S

simdim

Explorer
Joined
Mar 12, 2019
Messages
75
Can I run debug log collection from CLI? Want to let rsync finish before restarting middleware.
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,553
simdim said:
Can I run debug log collection from CLI? Want to let rsync finish before restarting middleware.
Click to expand...
Debug collection makes middleware calls

Can you maybe send me following via PM:
/var/log/messages
/var/log/middlewared.log

Output of dmesg
output of top -b -n 1
 
X

xjavr

Cadet
Joined
Jul 7, 2022
Messages
3
  • I have the same problem
middlewared.log
Print information every 2 seconds


[2022/07/12 09:33:12] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 825, in get_proc_inodes
for fd in os.listdir("%s/%s/fd" % (self._procfs_path, pid)):

[2022/07/12 09:33:14] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 827, in get_proc_inodes
inode = readlink("%s/%s/fd/%s" % (self._procfs_path, pid, fd))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 211, in readlink
path = os.readlink(path)

[2022/07/12 09:33:16] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 825, in get_proc_inodes
for fd in os.listdir("%s/%s/fd" % (self._procfs_path, pid)):
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,553
xjavr said:
  • I have the same problem
middlewared.log
Print information every 2 seconds


[2022/07/12 09:33:12] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 825, in get_proc_inodes
for fd in os.listdir("%s/%s/fd" % (self._procfs_path, pid)):

[2022/07/12 09:33:14] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 827, in get_proc_inodes
inode = readlink("%s/%s/fd/%s" % (self._procfs_path, pid, fd))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 211, in readlink
path = os.readlink(path)

[2022/07/12 09:33:16] (WARNING) middlewared._loop_monitor_thread():1663 - Task seems blocked:
File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1175, in call_hook
await fut
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 605, in check_permission
remote_addr, remote_port = get_remote_addr_port(app)
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 42, in get_remote_addr_port
if process := get_peer_process(remote_addr, remote_port):
File "/usr/lib/python3/dist-packages/middlewared/plugins/auth.py", line 25, in get_peer_process
for connection in psutil.net_connections(kind='tcp'):
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 2153, in net_connections
return _psplatform.net_connections(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1027, in net_connections
return _connections.retrieve(kind)
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 1001, in retrieve
inodes = self.get_all_inodes()
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 849, in get_all_inodes
inodes.update(self.get_proc_inodes(pid))
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 825, in get_proc_inodes
for fd in os.listdir("%s/%s/fd" % (self._procfs_path, pid)):
Click to expand...
That's not a crash, it's just an indication that a blocking call is being made on the main event loop of middlewared. What are the other symptoms?
 
X

xjavr

Cadet
Joined
Jul 7, 2022
Messages
3
anodos said:
That's not a crash, it's just an indication that a blocking call is being made on the main event loop of middlewared. What are the other symptoms?
Click to expand...
Prompt message when accessing Truenas：

Connecting to TrueNAS ... Make sure the TrueNAS system is powered on and connected to the network.
1657593418157.png
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Middleware crashes during rsync"

Similar threads

Rvby1
Rsync Commands Failing (Catalog and Remote PC Sync)
Replies
2
Views
1K
Rvby1
Rvby1
Dosch
Path from Nightlies to Stable
Replies
3
Views
2K
elvisimprsntr
E
danb35
Trying to upgrade from CORE, middleware doesn't start
Replies
8
Views
2K
morganL
M
Stickeris
Unsure Why Pool failed
Replies
13
Views
2K
Stickeris
Stickeris
pixelwave
Apps not showing up or working after reboot ...
2
Replies
32
Views
25K
Archetype
A
Top