SOLVED Testing Observation #1 - Dataset Disappearing

[kagbasi-wgsdac]

Hi Folks,

I trust you're all doing well. This is more of a sharing my observation than asking for support. I'm fairly new to TrueNAS and I'm testing TrueNAS-SCALE-22.12.3.1 in a virtualized environment. So far so good, however, I noticed a weird behavior and wasn't sure if this is a consequence of running it in a VM and using the VM disks (as opposed to doing a PCI passthrough of the raw disks). The first time it happened, I thought it was a fluke and simply rebooted the VM. But now that it's happened again, I figured I'd share - in case anyone can share some insights.

So, I've noticed - after some yet undetermined amount of time - that the only defined dataset disappears and I'm met with the following error whenever I attempt to access the Datasets menu from the GUI:

Code:

CallError
[ENOENT] Path /mnt/zPool1 not found

Here's additional details of the error:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 204, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1355, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1382, in nf
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1252, in nf
    res = f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/filesystem.py", line 363, in stat
    raise CallError(f'Path {_path} not found', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Path /mnt/zPool1 not found

Now, what's strange about this, while this error is present the Pool Status on the dashboard still shows that the pool is online and healthy. I'm using this dataset to host an SMB share, and consequently, while the share is visible from the Windows Explorer window, attempting to navigate into it leads to a path not found error.

Of course, I can resolve the problem by simply rebooting TrueNAS but I haven't done that yet. Gonna hold off to see what traction I get with this post. Thanks for taking any interest in this.

 

[joeschmuck]

You are missing some vital information:
1. What hypervisor are you using?
2. How is the hypervisor setup? How much RAM does your system have, how much did you allocate to the VM, how many CPU's core/threads, the type of virtual drive, capacity, etc.

Many of us have been virtualizing TrueNAS for years without any problem at all. I periodically switch between Core and Scale depending on what I'm doing. And I have run for several days Scale on a VM using virtual disks, it makes for a nice testbed. I use ESXi 7 for my hypervisor. I plan to update to ESXi 8 in a few months, maybe. Why mess with something that work?

 

[kagbasi-wgsdac]

You are missing some vital information:
1. What hypervisor are you using?
2. How is the hypervisor setup? How much RAM does your system have, how much did you allocate to the VM, how many CPU's core/threads, the type of virtual drive, capacity, etc.

Oh man, what a schmuck I am.

I have a rather simple setup. I'm running a single node XCP-ng version 8.2.1 as my hypervisor on a HP ProLiant DL360p Gen8 server with the following specs:

• Processor: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
• Processor Count: 2 (6 cores each)
• Memory: 768 GB (with ECC)
• RAM Allocated to TrueNAS VM: 8 GB
• vCPUs Allocated to TrueNAS VM: 4
• Virtual Disk Type: Local (ext4 I believe)
• Virtual Disk Count: 4
• Virtual Disk Capacity: 100GB each
• Virtual Disk Provisioning: Thin

Again, my apologies for not providing the above with my initial post.

 

[joeschmuck]

I've never used XCP-ng before so I can't speak to it including how reliable it is and I do know some folks are using it, but I would recommend you increase your TrueNAS VM RAM to 16GB. 8GB is the bare minimum and it's possible you are running out of RAM and using the Swap partition on the virtual drives. You can also drop the vCPU's down to 2 if you desire. One will work but I find it best to run two. The rest of your resources look good as a test bench.

 

[kagbasi-wgsdac]

@joeschmuck I've made the suggested changes to the VM and rebooted it. So far, the issue hasn't resurfaced. Will keep an eye on it. At least now I have a data point to keep in my notes.

Thank you.

 

[sretalla]

running it in a VM and using the VM disks (as opposed to doing a PCI passthrough of the raw disks)

If you're just using the VM for a test or to evaluate TrueNAS, no problem, but if the data you put on it is real, you need to change your approach.

Note that you can't really do a PCI passthrough of disks (unless they are NVME), rather of the entire disk controller.

"Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

If you're serious about the use of TrueNAS in a virtualized environment and want stability and security of your data, you should pay close attention to the notes in that post (which may have you concluding that you need to use ESXi instead).

 

[kagbasi-wgsdac]

If you're just using the VM for a test or to evaluate TrueNAS, no problem...

Yes, I'm only virtualizing TrueNAS for testing purposes. Want to make sure I document how to get it working in my Active Directory environment for smb/cifs and nfs shares.

Ultimately, I will be going with real hardware when the time comes to move into production. I may build my own or purchase an appliance. This is for a small church network by the way (single campus with maybe 50 Windows clients, and about 20 Debian Linux clients).

Not really a VMware ESXi fan anymore, since discovering XCP-ng. I have quite a bit of exposure to ESXi at work, so the article you shared is definitely of interest.

Thanks again.

 

[kagbasi-wgsdac]

Quick update. Thus far, the issue hasn't resurfaced. It appears this may have had something to do with the amount of RAM assigned to the VM (hmmm, interesting). I'm gonna mark this thread and solved, thank you all.