BOOT-Pool continuos writing [HELP]

[ThEnGI]

Good evening,
I am aware that the HW configuration is not the best and requires redundancy, but at the moment it is fine.
I notice continuous use of the disk reserved for the BOOT-Pool even if the "system dataset" is housed in the "FAST" pool.
Knowing that many people use USB sticks as boot pools, I don't expect all this data!
If there was little data it wouldn't be a problem but as you can see from the image it's just over 1GB/h. The poor SSD does not have a high TBW (due Low capacity) and at this rate will need to be replaced in less than a year.
Is this behavior considered normal? If not, where can I go to find the cause?
I haven't changed any particular configuration, and Decuplication is disabled

I attach two explanatory images

Spec:
Truenas SCALE 23.10.0.1
CPU I3 10105T
MB TUF Gaming B460M
RAM 16GB (8x8GB) DDR4 NO-ECC
Boot drive 1 SSD 128GB
Storage 1 NVME 2TB + 1 IRONWOLF PRO 12TB
PSU 550W Gold

Pool Storage : 1vDev 1 HDD
Pool Fast: 1vDev 1 NVME

I leave you with the post from the Italian section of Truenas as a reference (debug attempt, thanks to Davvo)

I apologize for the English and for any errors, I remain available for any questions
ThEnGi

 

[winnielinnie]

Does SCALE come with iotop installed by default?

 

[ThEnGI]

No iotop
For completeness, I also have about the same amount of writes on the Nvme disk (system dataset is here,I'm guessing it's the log). but being 20 times larger the problem does not arise,when I activate the dockers (now turned off) those 250kb/s in writing on the nvme are not influential.

 

[winnielinnie]

Apologies that I cannot follow the Italian discussion.

But did you rule out aggressive swapping to disk?

SCALE should ship with htop by default, and it can be configured to present information about I/O, memory, swap, and processes using swap.

You can add the columns "M_SWAP" and "IO_WRITE_RATE". Then you can sort by those values from highest to lowest to find the culprit.

 

[ThEnGI]

Of course, I didn't pretend that we could follow the discussion in Italian
I had assumed it was a swap problem, not knowing how truenas works I thought it was using the boot disk.
htop confirms that it is not swap
Hoping you configured it correctly doesn't show what it is

 

[winnielinnie]

Can you hit "TAB" to check on the I/O tab?

Something's not right.

Might this perhaps be a bug in the reporting page of the GUI? Does your case's "disk activity LED" flicker nonstop?

Nevermind. Your SSD is clearly being slammed by wayyyyyyyyyy too many writes. Nearly 500 GB total written already? For a boot device without a System Dataset...

 

[winnielinnie]

htop confirms that it is not swap

This doesn't show on your screenshot. You cropped the image.

 

[ThEnGI]

I can confirm it for you, the swap value is 0.
I'll try to send you the other images this evening.
For the moment I have turned off the nas.

 

[ThEnGI]

Yesterday I ran the command not as root. I'm inexperienced
As you can see zero swap, but lots of writing from "K3S".
Am I reading correctly? is it Kubernet?
I had already turned off all Docker, qbittorent remained active (Upload only).
Even with everything turned off, no change
I had also tried to "unset pool", but the writings remained.(App pool is the Nvme)
No led activity

 

[sfatula]

In Cobia, system logs are now written to the boot pool, might it be this?

See: https://www.truenas.com/docs/scale/23.10/gettingstarted/scalereleasenotes/

 

[ThEnGI]

If that is the case, can they be deactivated or partialized (example: only serious errors) ?
I understand that in many situations (mission critical) it is not an advisable solution, but changing SSD every year on the Home NAS is not exactly the best.
Even using SSD with high TBW (therefore high capacity) to write 10Gb of system plus logs on it is a waste!

 

[winnielinnie]

My goodness SCALE is quite "chatty".

10-15% nonstop CPU usage?

Constant writes to the bool-pool at 1 GB/hr?

And you're not using any third-party scripts or custom installed packages through "apt"?

 

[ThEnGI]

What you see is in an instant because the writings go up and down I caught the worst moment.
I'm 100% sure I didn't install anything via console, i'm using multi_report.sh (via cron job 7AM)
I leave the last hours of the CPU screen, on the dashboard average 2%-5 %

EDIT:I also have a script that cleans files older than x days in a given folder, but it's 3 lines: echo, find, echo.
It has always worked without problems (cron job 6AM)

 

[winnielinnie]

Does SCALE have an option to specify the system logs to be saved to the System Dataset? And this System Dataset will reside on your 2-TiB NVMe instead?

 

[ThEnGI]

The system datasets are already on the NVME, but I couldn't find any settings regarding the logs

 

[winnielinnie]

The system datasets are already on the NVME, but I couldn't find any settings regarding the logs

In Cobia, system logs are now written to the boot pool, might it be this?

Now I'm starting to get it. I didn't know what the changelog was in reference to, but I think I do now.

A recent change removed the ability to use the System Dataset to hold your logs? Now, no matter what, the logs are written to the boot-pool?

EDIT: Now I remember.

Constant writes to boot device despite moving system dataset

My current setup is using a HPE Proliant Microserver Gen 10+ with 2 16 TB HDDs in raid0 (I know it has no redundancy, but it is just a small homeserver to start with), 16 GB ECC memory, 1 TB NVME SSD to test things like L2ARC/SLOG and now moving the system dataset to it so the boot pool doesn't...

www.truenas.com

 

[ThEnGI]

the only thing that comes to mind is accessibility if the pools crash. Or if a pool is encrypted, so the logs are always clear.
But why not let me choose where to put them?
And then 1GB/h of log?

 

[winnielinnie]

But why not let me choose where to put them?
And then 1Gb/h of log?

I linked to another discussion. (See my edited post.) I don't agree with the decision to remove the ability to choose where to write your syslog.

Sorry to say, but looks like this just how it's going to be for SCALE. I really hope it doesn't make its way to Core.

 

[Whattteva]

I'm kinda' late to the game here, but just wanted to point this out.

Knowing that many people use USB sticks as boot pools, I don't expect all this data!

I don't think there are many people that do this at all. There used to be a lot of us back during FreeNAS 8.3.0 days, but those days are long gone. I think ixSystems has stopped recommending USB sticks as boot pools ince version 9.0 really.

We might do this for emergencies just to get the system back up and running again if we have no other option, but it's definitely not advisable for a long term solution.

 

[winnielinnie]

Granted, this doesn't really justify, in my opinion, outright removing the choice of where to store your syslog.

• Using USB sticks as your boot-pool may be a bad idea, but some people still do it (or they don't want to sacrifice one or two ports on their motherboard just for a boot-pool that rarely sees I/O).
• Smaller and lower-quality SSDs have lower write endurance.
• You can no longer store your syslog to an encrypted dataset (which may contain identifying / sensitive information).
• We had this choice before, but now the choice is being removed. (It's becoming too "Apple-ly" where more and more flexibility is being removed).

I suppose those of us who are still on Core are lucky in two ways:

1. We still have this choice in the settings.
2. We don't have K3s that loves constantly writing data.