SOLVED TrueNAS unknown disk access

[Lucas Rey]

Dear community, I'm asking help since I'm getting crazy. From last few weeks, my TrueNAS perform an unknow access to the disk that I don't understand where it comes from.

This is the specs:
TrueNAS is virtualized on Proxmox, VM has 16 Cores and 64GB RAM, running on Intel 9900K.
Previously pool: Raid-Z2 4x4TB ==> Disks were passed directly to TrueNAS VM
New pool: VDevs Mirror 4x12TB ==> The whole SATA controller is now passed to TrueNAS VM

Both pools have the disk access issue anyway. Problem is that new disks are Iron Wolf Pro, that compared to old WD Red I owned, are much more noisy, and my home server is close to me from all the day (working at home).

Now, come to the issue, about every 10 seconds I can hear the disks doing something (Read?Write?). Trrr, Trrr..... Trrr, Trrr.... Trrr, Trrr..... in an infinite loop! This is very annoying . Sometimes I'm forced to turn off the TrueNAS VM to stop this activity. As I wrote this is started from last weeks/months, and changing disks do not solve the issue.

There could be something that access the disk and this is what I did:
1) Disconnected the vnet interface to be sure no one can access the pool from external ==> No Way
2) Install a fresh copy of TrueNAS and just import the pool. I didn't touched anything else on settings ==> And BAM! Access to the disk is gone. I can still hear it sometime but very rarely, and definitely not every 10 seconds.

Now, coming back to the original TrueNAS installation, what could be the reason of such continuosly disks access? I already know you guys don't have the "crystal ball", but could someone help me to point to a right direction?

Thank you
Lucas

Anyway, these are my pools, all healthy

 

[chuck32]

Now, coming back to the original TrueNAS installation, what could be the reason of such continuosly disks access? I already know you guys don't have the "crystal ball", but could someone help me to point to a right direction?

Just a quick guess, your noisy setup uses your HDD pool for the System dataset.
When you install fresh and import afterwards the pool does not contain the system dataset.

I'd try to move the system dataset to your boot pool (assuming this is on a SSD/quieter HDD).

 

[Lucas Rey]

Just a quick guess, your noisy setup uses your HDD pool for the System dataset.
When you install fresh and import afterwards the pool does not contain the system dataset.

I'd try to move the system dataset to your boot pool (assuming this is on a SSD/quieter HDD).

You're a f*****g genious
I moved the System dataset to the boot pool (vdisk on quite and fast nvme) and the hard disk noise is gone! Thank you very much, you saved me from several headaches!
Just curious on what the system dataset read/write every 10 seconds and when this behaviour was introduced in TrueNAS. But it's not my primary think, I'm happy now I can work without the hard disks noise!

 

[joeschmuck]

You can do a Google search on TrueNAS system dataset and you will find some good information on it but it does update often these days. The old days this was minimized due to using a Flash drive as the boot device, now a SSD (non-Flash) is recommended due to durability.

 

[Lucas Rey]

I'd try to move the system dataset to your boot pool (assuming this is on a SSD/quieter HDD).

DAMN! I got again the hard disk noise every 5 seconds. This happen after some days I upgraded the TrueNAS Core to Scale Bluefin release.

It seems the system is still on boot pool (the nvme one)

Code:

# zfs list -t filesystem | grep -F ".system"
boot-pool/.system                                           1.38G  11.0G     1.35G  legacy
boot-pool/.system/configs-8a3eaa4a5a3647668792532b9af005d5  2.19M  11.0G     2.19M  legacy
boot-pool/.system/cores                                       96K  1024M       96K  legacy
boot-pool/.system/ctdb_shared_vol                             96K  11.0G       96K  legacy
boot-pool/.system/glusterd                                    96K  11.0G       96K  legacy
boot-pool/.system/rrd-8a3eaa4a5a3647668792532b9af005d5      32.0M  11.0G     32.0M  legacy
boot-pool/.system/samba4                                     680K  11.0G      460K  legacy
boot-pool/.system/services                                    96K  11.0G       96K  legacy
boot-pool/.system/syslog-8a3eaa4a5a3647668792532b9af005d5    100K  11.0G      100K  legacy
boot-pool/.system/webui                                       96K  11.0G       96K  legacy

And from my reasearch this is the offending parameter:

Code:

# cat /sys/module/zfs/parameters/zfs_txg_timeout
5

If I setup this value to e.g. 3600 the noise disappear. I already know that when changing this number it may result in a higher probability of lost data in the event of a power outage, but really I can't live with this irritating noise as my home server is close to me on my work desk.

I don't know why now only after some days from upgrade, anyway, is there anything I can do? Other than increase such value and cross fingers.
Thank you

 

[chuck32]

You can check in the GUI under datasets which pool is used for what. Post a screenshot.

Any Apps / VMs on the HDD pool?

 

[joeschmuck]

This happen after some days I upgraded the TrueNAS Core to Scale Bluefin release.

Have you tried to roll back the boot environment? If you upgraded your ZFS flags, you will not be able to do that.

 

[Davvo]

The old days this was minimized due to using a Flash drive as the boot device, now a SSD (non-Flash) is recommended due to durability.

For the records, I have been using a USB MLC stick as my boot drive for about one and a half year for testing purposes. The system's dataset is on a different (SSD) pool used for Jails and VMs. I have not been experiencing any issues so far.​

 

[joeschmuck]

For the records, I have been using a USB MLC stick as my boot drive for about one and a half year for testing purposes. The system's dataset is on a different (SSD) pool used for Jails and VMs. I have not been experiencing any issues so far.​

That is pretty good, I really wish more people could say that. I have a few drives that would stand up to the abuse as well, they were not cheap devices but they are very good. Now days the cost of a SSD over a high quality USB Flash drive could be upside down. Amazon sells it for $32USD, or a Patriot 120GB SSD for $15USD. But if a person didn't have a spare SATA port, the USB drive you have listed looks to be a solid bet.

Thanks for sharing, proof that not all USB flash drives are created equal.

 

[winnielinnie]

DAMN! I got again the hard disk noise every 5 seconds. This happen after some days I upgraded the TrueNAS Core to Scale Bluefin release.

Are you using any "Apps" by chance?

 

[Lucas Rey]

You can check in the GUI under datasets which pool is used for what. Post a screenshot.

Any Apps / VMs on the HDD pool?

I have several datasets used only via SMB, absolutely no app installed.

Have you tried to roll back the boot environment? If you upgraded your ZFS flags, you will not be able to do that.

Unfortunately I updated the flags right yesterday

Are you using any "Apps" by chance?

Nope, in my TrueNAS installation I also have data, and movies who will be shared via SMB using PLEX


EDIT: Well, that's strange, I restored the original value:
echo 5 >/sys/module/zfs/parameters/zfs_txg_timeout
And now the noise is gone, this will drive me crazy.

 

[chuck32]

Just wanted to confirm, in this overview you will see which pool is used for applications, system dataset etc.

Unfortunately I updated the flags right yesterday

For future reference: don't, unless you know you need the new flags.

And now the noise is gone, this will drive me crazy.

I'd chalk it up to the major upgrade from core to scale. Did you upgrade or did you do a clean install?

 

[Lucas Rey]

Did you upgrade or did you do a clean install?

Upgrade from the latest core version

 

[Davvo]

Reporting/SMART data pooling?

 

[Lucas Rey]

Reporting/SMART data pooling?

Do you mean this?

 

[Davvo]

No. I mean that the drive access you are getting might be related to the drives being pooled for data ie temperature.

 

[Lucas Rey]

No. I mean that the drive access you are getting might be related to the drives being pooled for data ie temperature.

How can I check it?

 

[Davvo]

No idea.