2 corrupted zpools after unexpected reboot (ECC memory in system)

[robert keefe]

I was putting the finishing touches on a rebuild of my old Freenas 10 server onto a TrueNas 12 Core U1 release with a newer 6 core Xeon v2 Platform with 4 x 8GB RDIMMS. The system had a 2x3TB mirror, 1x1TB SSD (bhyve) and 1x2TB HDD (non-important storage). It's been up and running for about 2 weeks now.

Yesterday I was about 4 out of 6 hours into a large SMB transfer to the 2x3TB pool. During this transfer, I never saw the amount of "Free" memory go lower than 0.7 GiB and it was hovering around 1 GiB, even after a few hours. I walked away for about 10 minutes when I heard my NAS beep like it does when it turns on. I turned on the console view and saw that it was bootlooping and finding this message for the SSD when importing the drive during boot ->

"panic: VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) fauiled (281481095389184 <= 858993592)"

I tried the drive in a new TrueNas install but that would not boot either. I had replication of the zvols to the 2X3TB array so I figured I would attempt to pull them off of the array. That array could not mount either because it would not pass integrity check during mount but at least TrueNas could still boot.

I was able to eventually mount the 2x3TB zpool in read-only mode. Luckily I could pull a recent snapshot name of a zvol and zfs SEND it to another drive zvol. But that data is also reporting corrupted.

At this point, I did not have any more replications configured yet because I just setup the box and literally was setting up external replications less than hour before this happened but did not finish. So I'm just going to start the process over from scratch sadly. Oh and the 1x2TB was sitting idle during the crash and has no issues.

Here is the message that TrueNAS reported when I was able to get into the GUI after the crash:

CRITICAL

Failed to check for alert VolumeStatus: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/alert.py", line 706, in __run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.8/site-packages/middlewared/alert/source/volume_status.py", line 31, in check for vdev in await self.middleware.call("pool.flatten_topology", pool["topology"]): File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call return await self._call( File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1206, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 438, in flatten_topology d = deque(sum(topology.values(), [])) AttributeError: 'NoneType' object has no attribute 'values'

 

[joeschmuck]

Have you submitted a bug report? If not, you should.

 

[robert keefe]

I just submitted one. Thanks for the recommendation.