Drives writing every 5 seconds despite moving system dataset to SSD boot pool

[huwf]

My current setup has 2 HDD's in a pool and an SSD as my boot pool. Current position of my home server in the house means that the constant writes to disk are an annoyance. Simplest solution would be to replace the HDD's with SSD's but budget currently doesn't allow for this. Reading the many posts on this forum about similar issues I decided to move the system dataset to the boot pool but this hasn't done anything to resolve the constant writes. I've confirmed nothing else is reading/writing to Truenas by disconnecting from the network.

Any ideas on how I can further diagnose what is making the HDD pool write every few seconds?

 

[Samuel Tai]

Did you reboot after moving the system dataset?

 

[huwf]

Did you reboot after moving the system dataset?

Yes rebooted after moving. I am running TrueNAS as a VM through Proxmox (set up in the way outlined in resources), would I need to also reboot the Proxmox machine? Seems unnecessary?

 

[Samuel Tai]

How are you passing the drive through from ProxMox to TrueNAS?

 

[huwf]

How are you passing the drive through from ProxMox to TrueNAS?

The boot drive is a virtual drive which is on an SSD. The HDD's are direct passthrough's (no dedicated SATA controller) done with these instructions.

 

[Samuel Tai]

This passthrough methodology isn't reliable with bhyve, the hypervisor in TrueNAS. It's impossible to cleanly separate writes between the guest and the host, and you will eventually lose data. As it is, it's impossible to tell if it's the host or the guest doing the continuing writes.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[huwf]

This passthrough methodology isn't reliable with bhyve, the hypervisor in TrueNAS. It's impossible to cleanly separate writes between the guest and the host, and you will eventually lose data. As it is, it's impossible to tell if it's the host or the guest doing the continuing writes.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Yes, I’m currently working on an upgrade for my machine so that I can do PCIe pass through instead as recommended.

The HDD’s are not used at all by the host, would that still mean that writes might be coming from them?

 

[Samuel Tai]

The HDD’s are not used at all by the host, would that still mean that writes might be coming from them?

It's impossible to say. Does ProxMox have some kind of disk health check? Is it actively trying to flush the disk buffer?

 

[huwf]

It's impossible to say. Does ProxMox have some kind of disk health check? Is it actively trying to flush the disk buffer?

Hmmmmm, it has access to view size and available space remaining. As for health checks not so sure. How could I check if it is trying to flush the buffer? Thank you for your help so far it has been very informative

 

[Samuel Tai]

Hmmmmm, it has access to view size and available space remaining. As for health checks not so sure. How could I check if it is trying to flush the buffer?

Sorry, I don't know. However, having both guest and host have simultaneous full access to the same disk is not a recipe for long-term success.

 

[huwf]

Sorry, I don't know. However, having both guest and host have simultaneous full access to the same disk is not a recipe for long-term success.

Thanks for all your help. Will go and mull this over

 

[huwf]

Sorry, I don't know. However, having both guest and host have simultaneous full access to the same disk is not a recipe for long-term success.

Hi again,

Looping back round to this. I have now set up the drives in the correct way by passing through the integrated SATA controller. The issue still persists with the files being written to every 5 seconds. I can confirm that Proxmox cannot even see the drives so the writes are definitely coming from TrueNAS. I've also tripled checked that the system dataset is pointed to the boot-pool which has an SSD

 

[huwf]

Leaving this here for anyone who might have this issue in future.

I tracked down the issue to be related to the k3s service which writes logs out every 5 seconds to the pool that ix-applications is on. You can turn this off with systemctl.

NOTE: This will mean you can’t use applications which will be a deal breaker for many.

 

[Redcoat]

@huwf So this is Scale...

This thread https://www.truenas.com/community/threads/k3s-server-uses-10-cpu-for-no-reason.91326/#post-688942 have anything useful for you on the apps front (I know zero about Scale)?

 

[huwf]

@huwf So this is Scale...

This thread https://www.truenas.com/community/threads/k3s-server-uses-10-cpu-for-no-reason.91326/#post-688942 have anything useful for you on the apps front (I know zero about Scale)?

Yes should have specified it’s Scale sorry.

Nothing in particular apart from the systemctl command. I wasn’t having CPU issues with running one app. I did find after my comment that docker also does periodic writes as well despite no apps running. Makes sense to need the logging while apps are running but why when they aren’t?

 

[ragametal]

I'm experiencing the same issue but on TrueNAS core.
Is there a similar service that i could just disable to silence the disks?