Hi,
we upgraded our FreeNAS-11.3-U5 to TrueNAS-12.0-U2.1 a few days ago. The Upgrade/Install as such was smooth and done within about 10 to 20 minutes.
TrueNAS was running fine for a few days and than started acting up all of a sudden. We had no SSH access, no GUI access and our Storage backups could not be written anymore. At first I didn't think much of it and classed it as a one time only freeze and classically restarted the machine. But last night it happened again and this time, during a resilvering, as we are just about to upgrade some of the disks in our pool.
The Pool mainly consisted of 6x6TB disks and we want to replace those with 8TB disks, so we added two new 8TB disks and started replacing the first 6TB disk of the volume. The resilvering process was already at 60.xx% on Friday and I thought, I could start the replacement of the second one on Monday, until TrueNAS crashed.
I'm a little worried about the resilvering process, because when I issue the CLI command zpool status, it still says "resilvering" but when I look at the GUI, it doesn't reflect that status.
This is a snippet from /var/log/messages at the time, the behavior started:
I did do some research about the out of swap space message but could only find old threads from 2017/2018, which did only allow me to check a few things being mentioned but couldn't really help me to figure out what is actually going on:
About the machine:
CPU: 4 Cores
RAM: 12 GB
SYS HDD: 16GB
Autotune: This was and has never been enabled
Tunables: None (since Autotune was never enabled)
Jails: None
vfs.zfs.arc_max: This had been mentioned quite a few times in the other threads I could find but, this is what I get, when I issue the sysctl command:
Not sure what is going on, since we didn't have those problems with the FreeNAS-11.3-U5 version. Please could someone help us out? Any help is really appreciated.
If I missed any information, please let me know and I will provide that as soon as possible.
we upgraded our FreeNAS-11.3-U5 to TrueNAS-12.0-U2.1 a few days ago. The Upgrade/Install as such was smooth and done within about 10 to 20 minutes.
TrueNAS was running fine for a few days and than started acting up all of a sudden. We had no SSH access, no GUI access and our Storage backups could not be written anymore. At first I didn't think much of it and classed it as a one time only freeze and classically restarted the machine. But last night it happened again and this time, during a resilvering, as we are just about to upgrade some of the disks in our pool.
The Pool mainly consisted of 6x6TB disks and we want to replace those with 8TB disks, so we added two new 8TB disks and started replacing the first 6TB disk of the volume. The resilvering process was already at 60.xx% on Friday and I thought, I could start the replacement of the second one on Monday, until TrueNAS crashed.
I'm a little worried about the resilvering process, because when I issue the CLI command zpool status, it still says "resilvering" but when I look at the GUI, it doesn't reflect that status.
This is a snippet from /var/log/messages at the time, the behavior started:
Code:
Mar 13 00:00:01 vm-freenas-a-1 syslog-ng[1402]: Configuration reload request received, reloading configuration;
Mar 13 00:00:01 vm-freenas-a-1 syslog-ng[1402]: Configuration reload finished;
Mar 13 02:01:48 vm-freenas-a-1 1 2021-03-13T02:01:48.620077+00:00 vm-freenas-a-1 collectd 1730 - - Traceback (most recent call last):
File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
with Client() as c:
File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
self._ws.connect()
File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
rv = super(WSClient, self).connect()
File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
bytes = self.sock.recv(128)
socket.timeout: timed out
Mar 13 16:56:49 vm-freenas-a-1 1 2021-03-13T16:56:48.865154+00:00 vm-freenas-a-1 collectd 1730 - - Traceback (most recent call last):
File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
with Client() as c:
File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
self._ws.connect()
File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
rv = super(WSClient, self).connect()
File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
bytes = self.sock.recv(128)
socket.timeout: timed out
Mar 13 22:03:47 vm-freenas-a-1 kernel: pid 70919 (rsync), jid 0, uid 0, was killed: out of swap space
Mar 13 22:03:47 vm-freenas-a-1 kernel[1402]: Last message 'pid 70919 (rsync), j' repeated 1 times, suppressed by syslog-ng on vm-freenas-a-1
Mar 13 22:03:47 vm-freenas-a-1 kernel: pid 70920 (rsync), jid 0, uid 0, was killed: out of swap space
Mar 13 22:03:47 vm-freenas-a-1 kernel[1402]: Last message 'pid 70920 (rsync), j' repeated 1 times, suppressed by syslog-ng on vm-freenas-a-1
Mar 13 22:03:47 vm-freenas-a-1 kernel: pid 1730 (collectd), jid 0, uid 0, was killed: out of swap space
Mar 13 22:03:47 vm-freenas-a-1 kernel[1402]: Last message 'pid 1730 (collectd),' repeated 1 times, suppressed by syslog-ng on vm-freenas-a-1
Mar 13 22:03:47 vm-freenas-a-1 kernel: pid 277 (python3.8), jid 0, uid 0, was killed: out of swap space
Mar 13 22:03:47 vm-freenas-a-1 kernel[1402]: Last message 'pid 277 (python3.8),' repeated 1 times, suppressed by syslog-ng on vm-freenas-a-1
Mar 13 22:03:47 vm-freenas-a-1 kernel: pid 278 (python3.8), jid 0, uid 0, was killed: out of swap space
Mar 13 22:04:47 vm-freenas-a-1 kernel[1402]: Last message 'pid 278 (python3.8),' repeated 1 times, suppressed by syslog-ng on vm-freenas-a-1I did do some research about the out of swap space message but could only find old threads from 2017/2018, which did only allow me to check a few things being mentioned but couldn't really help me to figure out what is actually going on:
About the machine:
CPU: 4 Cores
RAM: 12 GB
SYS HDD: 16GB
Autotune: This was and has never been enabled
Tunables: None (since Autotune was never enabled)
Jails: None
vfs.zfs.arc_max: This had been mentioned quite a few times in the other threads I could find but, this is what I get, when I issue the sysctl command:
Code:
root@vm-freenas-a-1:~ # sysctl -a vfs.zfs.arc_max vfs.zfs.arc_max: 0
Not sure what is going on, since we didn't have those problems with the FreeNAS-11.3-U5 version. Please could someone help us out? Any help is really appreciated.
If I missed any information, please let me know and I will provide that as soon as possible.
