Disabling scrub on the boot pool

[Whattteva]

I'm pretty sure this is such a simple task but I can't seem to find this on the forums or Google.
I think I read somewhere in the past that when you're virtualizing TrueNAS, it's advisable to disable scrubs for the boot pool since it's on a virtual disk, but that scrub task doesn't seem to be listed anywhere in the GUI. Obviously, it's there somewhere because I notice the boot pool scrub task completion in the TrueNAS notifications area.

Anyone have some ideas on where to find and disable this?

 

[winnielinnie]

Are you on Core or SCALE?

For TrueNAS Core: System -> Boot -> ACTIONS -> Stats/Settings

You can change its interval. I'm not sure if setting this to "0" will completely disable it. The tooltip is not clear.

 

[Whattteva]

Ah thanks. That setting is kinda' buried. I am on CORE. I suppose I'm going to set it to 0 for now and see if that works.

EDIT: 0 is not possible it seems. It barfs back an error message saying 0 is not a valid number of days.

 

[Whattteva]

I suppose, for now, I'm just going to set it to max 32-bit unsigned integer (65535). If anyone has a better suggestion, I'm all ears.

 

[Whattteva]

Another interesting thing I notice is SCALE doesn't announce boot pool scrubbing under alerts, but stats/settings suggests that it still performs scrubs on the boot pool silently. Does anyone know if this is a bug or actually intended functionality? If it is intended, why the difference in behavior with CORE?

I noticed this on my experimental SCALE VM. I have no real data here. Just basically trying out the UI and TrueCharts apps.

 

[joeschmuck]

I think I read somewhere in the past that when you're virtualizing TrueNAS, it's advisable to disable scrubs for the boot pool since it's on a virtual disk,

Hum... I've been virtualizing TrueNAS for something like a decade and been performing a SCRUB on the boot pool weekly without any ill effect. I'm using ESXi 7.0 right now but I've used 5.5 and later. I update periodically as ESXi updates occur.

Now I could see changing the periodicity for the bootpool if it were on a USB Flash Drive, but that is a different thing and not a virtualized TrueNAS.

 

[Whattteva]

Hum... I've been virtualizing TrueNAS for something like a decade and been performing a SCRUB on the boot pool weekly without any ill effect. I'm using ESXi 7.0 right now but I've used 5.5 and later. I update periodically as ESXi updates occur.

Now I could see changing the periodicity for the bootpool if it were on a USB Flash Drive, but that is a different thing and not a virtualized TrueNAS.

Well, I suppose letting it scrub isn't any harm, but as the boot pool is already hosted on a ZFS file system, I just don't see any point in scrubbing it again in the VM.

 

[joeschmuck]

Are you saying that your VM boot drive for TrueNAS is being hosted on a ZFS file system? Not that the bootpool itself is a ZFS pool within the virtual machine. If yes, I'm curious what hypervisor you are using. I'm asking to educate myself. I just freed up an older ESXi 6.0 server (I was going through some old computers and didn't know what was on them so I just bootstrapped one, it has ESXi and on that Sophos (my old firewall), a FreeNAS backup server (I forgot how nice the older GUI was), and Ubuntu). So I have a computer (well many), that I could spin up a ZFS based hypervisor.

 

[Whattteva]

Are you saying that your VM boot drive for TrueNAS is being hosted on a ZFS file system? Not that the bootpool itself is a ZFS pool within the virtual machine. If yes, I'm curious what hypervisor you are using. I'm asking to educate myself. I just freed up an older ESXi 6.0 server (I was going through some old computers and didn't know what was on them so I just bootstrapped one, it has ESXi and on that Sophos (my old firewall), a FreeNAS backup server (I forgot how nice the older GUI was), and Ubuntu). So I have a computer (well many), that I could spin up a ZFS based hypervisor.

Yeah, the boot pool itself is hosted on a ZFS file system (Proxmox). So ideally, I'd like the boot pool to be formatted in UFS, but the install doesn't make that possible. I could probably run it on a vanilla FreeBSD machine, but I rather like the TrueNAS CORE UI.

I'm aware that it's not recommended and ESXi is what is recommended, but I already have a baremetal TrueNAS CORE machine with identical pool, so I'm not worried about any data losses and mainly trying to see if this is viable for a production-level machine in the long term.

 

[Volts]

I read somewhere in the past that when you're virtualizing TrueNAS, it's advisable to disable scrubs for the boot pool since it's on a virtual disk,

The more-conservative TrueNAS and ZFS communities would not give that advice.

Scrubs verify data checksums from the perspective of that ZFS pool. That responsibility can't be shifted to a virtualized storage provider. (It doesn't matter if the virtual provider is also ZFS.)

The outer virtualized storage provider must be responsible for interacting with the hardware, because the inner ZFS can't.
But the outer virtualized storage layer can't verify the inner ZFS data checksums, that must be done from the inner ZFS perspective.

Leave the boot-pool scrub enabled! It takes a minute to complete, and it's a frequent early warning of system problems.

 

[Whattteva]

The more-conservative TrueNAS and ZFS communities would not give that advice.

Scrubs verify data checksums from the perspective of that ZFS pool. That responsibility can't be shifted to a virtualized storage provider. (It doesn't matter if the virtual provider is also ZFS.)

The outer virtualized storage provider must be responsible for interacting with the hardware, because the inner ZFS can't.
But the outer virtualized storage layer can't verify the inner ZFS data checksums, that must be done from the inner ZFS perspective.

Leave the boot-pool scrub enabled! It takes a minute to complete, and it's a frequent early warning of system problems.

Are you sure about this? I thought ZFS mostly operates at block level so this is irrelevant. It's what makes zfs send/recv snapshots infinitely better than rsync when it comes to backing up large VM disk images.

I've also restored VM images simply by just using the underlying hypervisor ZFS snapshot of the whole dataset. The state of the actual VM disks or even what it's formatted as, for that matter, is completely irrelevant.

In any case, I don't really care about the boot pool going bad cause it's a no brainer to restore with a backed up config. Heck, I don't even care about the config cause I can recreate everything from scratch (just very tedious process).

Plus, as I said earlier, if anything happens, I just rollback to an earlier snapshot from the parent hypervisor.

EDIT:
I found the reference where I read it. Seems to originate from actual iXSystems blog (first-party source), though the link no longer works unfortunately. However, the OP summarized the blog post.

Installation in vmware best practices clarification

I want to install freenas in vmware. I also need a windows installation on the same machine so I tried to install freenas as primary OS but the windows installation had very poor performance. I found this article (http://www.freenas.org/blog/yes-you-can-virtualize-freenas/) but i have 2...

www.truenas.com

 

[Volts]

It's like putting writing letters on paper, putting them in envelopes and giving them to a mailman.
The mailman can say he delivered all of the envelopes he received.
But the mailman can't know if he received all of the envelopes he should have.
And the mailman can't know if all the pages were inside each envelopes.
And the mailman can't know if the message on each page was correct.
Only the recipient can validate that all messages are present and complete and correct.

The state of the actual VM disks or even what it's formatted as, for that matter, is completely irrelevant.

Right, and maybe that's a useful way to think about it. The virtual storage layer doesn't care about the contents at all, in the same way that an HDD doesn't care. An HDD doesn't know what filesystem is in use, or if it's healthy or corrupt, or if a file was written correctly. HDDs and virtual storage providers are both content agnostic.

HDDs are pretty reliable, but they occasionally have errors. And sometimes other system bugs exist too! That's why ZFS scrub is useful.

Virtual storage is also pretty reliable, and also occasionally has errors. And sometimes other system bugs exist - there are a lot more components involved. That's why ZFS scrub is useful.

That article is available in the Wayback Machine:
https://web.archive.org/web/20180215194736/www.freenas.org/blog/yes-you-can-virtualize-freenas/

A ZFS scrub reads data and verifies hashes. It's not going to rewrite or damage a single-device, copies=1 pool, but it will alert you to system issues.

The scrub-of-doom idea sounds similar to the old non-ECC scrub-of-doom fearmongering. I don't ... I just don't ...

In any case, I don't really care about the boot pool going bad cause it's a no brainer to restore with a backed up config. Heck, I don't even care about the config cause I can recreate everything from scratch (just very tedious process).

Fair enough, I agree! It's a great mindset - protect data, but configure fresh servers.

 

[Joe Z]

That article is available in the Wayback Machine:
https://web.archive.org/web/20180215194736/www.freenas.org/blog/yes-you-can-virtualize-freenas/

I found this thread after reading similar information here.

Virtualized TrueNAS
Finally, the ultimate TrueNAS hardware question is whether to use actual hardware or choose a virtualization solution. At the heart of the TrueNAS design is OpenZFS. The design from day one works with physical storage devices. It is aware of their strengths and compensates for their weaknesses.

TrueNAS developers virtualize TrueNAS every day as part of their work and it is intended only for use as a development environment.

While possible to deploy TrueNAS in a virtual enviroment this is not recommended for regular deployment of TrueNAS whenever storing production or crtical data. Virtualizing TrueNAS and using virtual disks for your zpool is fine for ad hoc proof-of-concept, but it is not a supported configuration and it might result in data corruption.
When the need arises to virtualize TrueNAS (for ad hoc proof-of-concept):
Pass hardware disks or the entire storage controller to the TrueNAS VM if possible (requires VT-d/AMD-Vi support).
Disable automatic scrub pools on virtualized storage such as VMFS, and never scrub a pool while also running storage repair tasks on another layer.
Use a least three vdevs to provide adequate metadata redundancy, even with a striped pool.
Provide one or more 8 GB or larger boot devices.
Provide the TrueNAS VM with adequate RAM per its usual requirements.
Consider jumbo frame networking if all devices support it.
Understand that the guest tools in FreeBSD might lack features found in other guest operating systems.
Enable MAC address spoofing on virtual interfaces and enable promiscuous mode to use VNET jail and plugins.

Thanks for this. I'm also using Proxmox and I virtualized my boot device but passed through the drives to the vm. I'll leave the boot pool scrub at the default.
Further reading today I realize I should pass through the controller instead of the drives. The host can run smart tests, but TrueNAS cannot. Besides that, everything works. I'll start some research today on a dedicated controller device to pass the drives through. I need to add support for additional drives anyway for raidz. It's just mirrored now.

 

[HoneyBadger]

Further reading today I realize I should pass through the controller instead of the drives. The host can run smart tests, but TrueNAS cannot. Besides that, everything works. I'll start some research today on a dedicated controller device to pass the drives through. I need to add support for additional drives anyway for raidz. It's just mirrored now.

I'd recommend an LSI SAS2308-based controller if you're going to run CORE. The latest kernel updates to FreeBSD 13.x (which CORE runs) have exposed some weird edge case issues around timeouts communicating with a passthrough SAS2008 controller.

This will need to be an additional (second) controller vs. the one you're using to boot and run ProxMox - but if you're using onboard/integrated SATA for that, you can continue there.

For controller passthrough information on ProxMox, check here:

PCI Passthrough - Proxmox VE

pve.proxmox.com

 

[ChrisRJ]

I'd recommend an LSI SAS2308-based controller if you're going to run CORE. The latest kernel updates to FreeBSD 13.x (which CORE runs) have exposed some weird edge case issues around timeouts communicating with a passthrough SAS2008 controller.

Is this something that should be added to @jgreco 's resource about virtualization?

 

[jgreco]

I'd recommend an LSI SAS2308-based controller if you're going to run CORE. The latest kernel updates to FreeBSD 13.x (which CORE runs) have exposed some weird edge case issues around timeouts communicating with a passthrough SAS2008 controller.

Are we referring to the doorbell handshake issue? Because I can assure you that this has been around for a lot longer than that.

 

[HoneyBadger]

Are we referring to the doorbell handshake issue? Because I can assure you that this has been around for a lot longer than that.

That's the one - but the recent changes to upstream FreeBSD 13.0 seem to have created a whole lot more frequent and more consistent cases where the SAS2008 is failing in a virtualized setup. I know we've got a few threads on here to reference but it seems like users are reporting it working after going to a SAS2308, regardless of TrueNAS and ESXi version.

 

[jgreco]

I think if I was going to say anything at all about it, it would be just to take a pass on the 6Gbps controllers. They're also problematic in ESXi due to the ESXi LSI driver deprecation issues and can cause some weird issues at boot time, install time, etc., and can be challenging for newbies to get provisioned properly for PCIe passthrough.

 

[Whattteva]

I don't have a SAS2008, but I've been using the SAS2308 with zero issues on Proxmox.