Constant I/O, no Disk temps on totally new install

[zook007]

Wanted to add to other thread, but thread was closed... figured maybe this may help ?

Yesterday installed TrueNAS Scale (latest version) on brand new clean disks. System installed onto NVMe, and pool created on 4 HDDs. Also created 2 users, but that is all I did beyond the installation. GUI showed disk temps right after install, but after shutdown and restart disk temps are not available anymore. I have no shares created, no containers, all services turned off.

Also, and most worrisome, is constant disk I/O to pool disks, and I do mean constant, see plot below, that actually goes back for a full 24 hrs. There is absolutely no data on the pool disks beyond teh creation of a data-set, but have not uploaded any files to it. There are no other devices on the network overnight (turned off the computer accessing the GUI), so the server is not responding to any net traffic. There are also no READS, it is all WRITES. There is no outgoing network traffic out of the NAS, judging by the activity light on the RJ-45 jack on the switch.

Read another post of someone's NVMe cards wearing out within weeks, perhaps due to this constant I/O ?

Anyways, for now I will keep the NAS turned off, as I do not want to cause un-needed wear on my storage devices, or perhaps should switch to Core ?

 

[zook007]

...I should add, the disk activity is the same for all 4 HDDs, two are shown in the pic above, but the other two are identical. I have a RAIDz.

 

[winnielinnie]

It's your System Dataset + logging.

You can move your system dataset to your boot-pool instead.

Look under your System settings.

 

[zook007]

Thank you for the reply @winnielinnie . Got it, though if I do that then will there be constant writes to my NVMe drive that houses the system OS ? I am concenrned as this coudl wear out my NVMe quickly ?

 

[winnielinnie]

Got it, though if I do that then will there be constant writes to my NVMe drive that houses the system OS ? I am concenrned as this coudl wear out my NVMe quickly ?

Pick your poison.

Though these writes are small, kilobytes to megabytes at a time. While it slightly adds to the overall writes for your boot drives, it's not like it's going to make you reach the supposed TBW lifetime that much sooner. And your boot drives are the least important of your drives anyways.

 

[zook007]

Thanks again @winnielinnie . I am completely new to TrueNAS, so sorry if what I am asking is dumb... Can I stop the logging, or atleast change the intervals it needs to write ? every 5 seconds seems excessive for my server, as it can sit there for days without me accessing it. Would actually be nice (for my situation) if I could stop all that writing to disk altogether... ?

Would TrueNAS Core be different if I do not use ZFS ?

 

[winnielinnie]

There are some options throughout to make the log less "chatty".

But the System Dataset will always be a source of constant writing.

TrueNAS, Core or SCALE, are pure ZFS solutions. There is no TrueNAS without ZFS.

However, now that I think about it, you might be witnessing writes to the System Dataset + logging + K3s/ix-application, since you are on SCALE.

 

[zook007]

Thank you for the reply @winnielinnie . I do not want to sound "contrarian" but I would think that if there is nothing going on with requests to the server, seems no point in constantly writing to disk -- nothing is changing, so no point i writing the same stuff over and over ? ...except perhaps disk temperatures, but taking a temp every 30 minutes and storing into memory until buffer full, would not require a disk write until many many hours later.

Really my worry is that my M.2 NVMe system disk will wear out too soon, due to the constant writes -- there is a post about this happening to someone here, where his RAID system disk on NVMe's wore out within weeks....

However, now that I think about it, you might witnessing writes to the System Dataset + logging + K3s/ix-application, since you are on SCALE.

Given what you say above, would Core be less disk write intensive ?

 

[ChrisRJ]

Really my worry is that my M.2 NVMe system disk will wear out too soon, due to the constant writes -- there is a post about this happening to someone here, where his RAID system disk on NVMe's wore out within weeks....

The normal system logging is absolutely a non-issue. All servers have been doing this for many decades and it is necessary for proper system operation.

What post are you referring to?

What does your hardware look like?

Given what you say above, would Core be less disk write intensive ?

Not really. Details may vary a bit here and there, but again this is not an issue for any decent (incl. consumer) SSD.

 

[doze5]

zook007​

Are you talking about HDD activity every ~5 second? There is many topic about that on this and other forums. No one seems to know how to solve this issue. Constant sound from my Toshiba N300 was very, very loud especially at night. I had to do something. My solution from hardware point of view is still under construction at my workshop, but there is also a software solution for 5 sec activity. The problem is from flushing of ZFS TXGs.

1.You have to use a script to change it.
echo 500 >/sys/module/zfs/parameters/zfs_txg_timeout

With that you will get the sound every 500 seconds - you can change that for 30 sec or what you want.

In TrueNAS-SCALE-22.02.4 you have to go to System Settings -->Advanced --> Init/Shutdown Scripts

I don't know what is the path on the TrueNAS SCALE 22.12, because apps wouldn't work for me on TrueNAS SCALE 22.12, so I had to install TrueNAS SCALE 22.02.4.

2. Move your System Dataset Pool to SSD (boot-pool) - few other sounds.

 

[jgreco]

My solution from hardware point of view is still under construction at my workshop, but there is also a software solution for 5 sec activity. The problem is from flushing of ZFS TXGs.

This is very not recommended and may mess with other things in unexpected ways.

 

[Patrick M. Hausen]

Why are people so obsesses with keeping their disk drives from doing disk drive things?

 

[doze5]

Why are people so obsesses with keeping their disk drives from doing disk drive things?

What is more important for you? Sleep or disk drive things? And yes, my server is in another room but sound from HDD was ~48dB. That was driving me and my family crazy.

This is very not recommended and may mess with other things in unexpected ways.

Could you be more specific?

 

[jgreco]

Could you be more specific?

Not really. It wasn't intended to be used in this manner and it messes with the write throttle algorithm. Because we don't encourage people to do this, I haven't collected a list of ways in which this screws with you. We normally just recommend that people who want quiet get a pair of cheap SSD's and put their system pool on that; it's cheap, straightforward, and expected to work just fine. Messing with the delay is like covering all your vents with duct tape because you found the noise coming from the machine to be "too loud"; sure, it'll fix it, but it might have unanticipated side effects.

 

[HoneyBadger]

Could you be more specific?

A 500s timeout on your transaction group means that writes too small to trip your "commit to disk" threshold won't get committed until up to 8 minutes and 20 seconds after you save them.

In an async write scenario (default for most SMB workloads) this means that your system could potentially have up to 320MB of data lost in a power outage or hardware failure if it's not written.

 

[WI_Hedgehog]

no Disk temps,​

Constant I/O

I'm getting that RAID controller feeling...

 

[doze5]

We normally just recommend that people who want quiet get a pair of cheap SSD's and put their system pool on that; it's cheap, straightforward, and expected to work just fine.

I have SSD and my system is on it. Also moving System Dataset Pool to SSD it was first thing I did, but that didn't help. No effect on 5 sec HDD activity at all because the problem is different thing. That is why there is so many topic on this forum and other forums about 5 seconds HDD activity and no solution, so I had to find it on my own. I am very, very happy with the result - it works for a change. Nevertheless I will build a hardware solution in few weeks, Toshiba N300 is loud and in the future I will have more of them, so hardware solution is a must.

HDD noise every 5 seconds (already moved system pool)

Like the tittle said, my disks are having some sort of activity every 5 seconds or so, which is notable by their LED as well as a clunking noise. I know that this is usually because TrueNAS writes it's logs on the system dataset which is installed on the first data pool. But this behavior...

www.truenas.com

hdd activity every five seconds or so

Hello, My newly operational freeNAS build (intel Avoton with six HGST Deskstar NAS sata disks for the data pool) is accessing the data pool/volume about every five seconds...at least I hear the hdd's clicking for about two seconds in about five second intervals. Found a few other posts on...

www.truenas.com

Find what is accessing the HDD

Hello, I am trying to figure what process(es) is (are) accessig my HDD every 5 minutes. Is there a way to detect what is doing it? I have added a SSD (formated in ZFS) as a second pool and put the system pool in it. I was expecting the HDD to sleep automatically but it doesn't work. I have...

www.truenas.com

 

[WI_Hedgehog]

@zook007 has a constant write pattern of 100KB/sec.

This is different from the seek/write @doze5 is not happy with--and on that, it is Polish to "solve" a software situation with a hardware workaround, that I understand. However, doing so creates something called "side-effects" which are loathed because it breaks the normal "cause-effect" cycle and introduces random anomalies. The outcome is the system generally becomes "unstable."

 

[doze5]

WI_Hedgehog​

I know my English isn't very good, but you should understand me. I have solved a software problem with a software solution - script. I am a noob in therms of software, so yes I'm very happy. Maybe zook007 problem is different, maybe not. Pattern is very similar.

In therms of HDD noise, it is a hardware thing - I am talking about normal work. When you use HDD for a Jellyfin server there is sound from hardware - hard disk, and Toshiba N300 is a loud one. If you know how to quiet it down without a hardware solution I am waiting for your "not Polish" and a software solution for normal HDD noise. I think you will get a Nobel Prize or something - that would be straight up magic. Maybe some kind of force filed in python or C++?

Also tell to the automotive industry that they shouldn't quiet down and isolate insides of cars, that is bad and "a Polish" solution for a software problem ... They should quiet down engine and tyres sounds with a software solution. Dude please, don't be salty.

 

[WI_Hedgehog]

@doze5 : Teraz rozumiem! Now I understand! The 5-second HDD issue is the first issue you have reoriented with software.

The "hardware solution" you speak of is to resolve a second issue unrelated to the first, but rather related to Jellyfin, something you had not before clearly defined.

Now I understand why you feel I have not showered in fresh water after swimming in the sea, and am appropriately seasoned. Even if that were so, I would not code in Python.