Continous background disk writes

[bal0an]

I am trying to understand why my freenas 9.3 continously writes to the 5 disks in the RAIDZ array of my freenas instance:

The writing takes place at ~150kB/s. The break between 21:00 and 21:15 was an update of freenas to the latest version.
I've already tried to shutdown the plex plugin or rsyncd both which had no effect.
System dataset Syslog and Reporting Database is both configured off.

Any idea what is writing to the disks?
Regards, Andreas

 

[ITGuy1024]

I'd still think it's activity from .system dataset.

 

[cyberjock]

Try reading this ticket...

https://bugs.freenas.org/issues/4882

If we can ascertain what the problem is exactly that would be nice. You aren't the first to have this problem (and probably won't be the last). Unfortunately if you reboot the problem will magically go away, so we'll be clueless again. ;)

 

[DrKK]

No. I have this problem persistently. Constant disk activity at about this I/O rate that he specified.

 

[bal0an]

I spend some time today trying to identify the reason:

• top -m io was inconclusive
• stopping collectd didn't make a difference.

1. I checked and unchecked System dataset|Syslog and observed an error message: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory
so I created the log directory
[root@freenas] /var# mkdir /var/log/samba4
and unchecked System dataset|Syslog again. After a reboot the background writes were gone. Now I could see log files in the samba4 directory.
2. Trying to reproduce I checked System dataset|Syslog. What I could observe was a recycling of the log directory:
drwxr-xr-x 2 root wheel 160 Jan 12 16:53 log/
drwxr-xr-x 5 root wheel 800 Jan 12 16:12 log.20150112161525/
and some background writes again.
3. Unchecked System dataset|Syslog again. No more background writes but again the samba4 log directory missing. I had hoped the missing samba4 directory would cause endless writes on the system data pool.
Jan 12 16:21:08 freenas syslog-ng[9991]: syslog-ng starting up; version='3.5.6'
Jan 12 16:32:02 freenas smbd[10718]: STATUS=daemon 'smbd' finished starting up and ready to serve connectionsUnable to open new log file '/var/log/samba4/log.smbd': No such file or directory
Jan 12 16:32:02 freenas smbd[10718]: [2015/01/12 16:32:02.476970, 0] ../lib/util/debug.c:595(reopen_logs_internal)
Jan 12 16:32:02 freenas smbd[10718]: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory
Jan 12 16:32:02 freenas smbd[10718]: [2015/01/12 16:32:02.477207, 0] ../lib/util/debug.c:595(reopen_logs_internal)
Jan 12 16:32:02 freenas smbd[10718]: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory
Jan 12 16:32:02 freenas smbd[10718]: [2015/01/12 16:32:02.477505, 0] ../lib/util/debug.c:595(reopen_logs_internal)
Jan 12 16:32:02 freenas smbd[10718]: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory
Jan 12 16:32:02 freenas smbd[10718]: [2015/01/12 16:32:02.531418, 0] ../lib/util/debug.c:595(reopen_logs_internal)
Jan 12 16:32:02 freenas smbd[10718]: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory
Jan 12 16:32:02 freenas smbd[10718]: [2015/01/12 16:32:02.548134, 0] ../lib/util/debug.c:595(reopen_logs_internal)
Jan 12 16:32:02 freenas smbd[10718]: Unable to open new log file '/var/log/samba4/log.smbd': No such file or directory

My server shuts down during the night. Will keep you posted whether the background writes come back after a restart.

 

[bal0an]

The continuous writing returned. Tonight it stopped when I turned off plex.

Some previous occasions the writes would only stop after shutting down the jail as well.
That seems pretty reproducable.

 

[Knowltey]

Hmm, I have this as well, I had assumed since the writes were small that it was just something with how 9.3 operated.

 

[bal0an]

I changed plex to be off by default. So when I turn on plex at (1) I can observe continous writes to start and when I turn it off at (2) they stop.

This only happens since I upgraded plex to the latest version during my freenas upgrade from 9.2* to 9.3. Any idea how to configure freenas to turn the continous writing off?

 

[Knowltey]

Hmm, seems to be PLEX for you at least. However I don't use PLEX, or any of the plugins for that matter and I have constant writes as well at around a similar size as you as well.

 

[cyberjock]

This only happens since I upgraded plex to the latest version during my freenas upgrade from 9.2* to 9.3. Any idea how to configure freenas to turn the continous writing off?

That would be a question to ask the Plex forum. Basically it could be almost anything in the plugin/jail, so you should probably do some troubleshooting to rule out Plex itself. But if it is Plex, their forum would be the place to go for questions and answers.

 

[Fraoch]

I often see this behaviour on 9.3. It seems to change whenever the server is rebooted. It's calmed down a bit today, but this is still a lot of writes for a server doing basically nothing.

No plugins here at all.

 

[Fraoch]

On reboot after an OS update, same situation as the OP and worse than my last post.

 

[kyp]

Are there any new developments on this topic because it seems a bit unusual?

 

[Fraoch]

It never really changed, even now. The spikes have gotten higher. I wonder if this is because my Linux machine has the drive mounted permanently (through NFS)? It mounts it on boot and keeps it attached through fstab. There isn't constant access but perhaps the Linux machine "pings" it continuously.

I would have thought that would be reads though, not writes.

 

[depasseg]

Disable the share and plugins and see if the activity goes away. I have a 12 drive backup pool, with no shares and is only used for ZFS replication. It's pretty much at 0 when replication isn't running.

Although, those spikes look a lot like the pools that have my system datasets.

.system:

My other .system:

Dataset only used for Replication:

 

[Fraoch]

Disable the share and plugins and see if the activity goes away. I have a 12 drive backup pool, with no shares and is only used for ZFS replication. It's pretty much at 0 when replication isn't running.

There's been no change. By the way, I applied an update here. But you can see a small read where the share gets unmounted.

Although, those spikes look a lot like the pools that have my system datasets.

Aha, I think you're on to something here! Yes, my system dataset is indeed on my share pool, not my boot pool.

It's been a while since I've really worked with FreeNAS so I'm rusty on this, but should I move my system dataset to my boot pool? There must have been a reason I kept it on my main pool, I'm wondering what the recommendation was at the time (and now). Perhaps I was trying to avoid writes on my boot pool, which consists of two cheap SSDs, mirrored. They have more write endurance than USB sticks for sure, but it would still be best to avoid continuous writes. However they'd be cheaper to replace than the mechanical drives.

Should I move the system dataset to my boot pool?

Thanks for the advice, by the way, and thanks to kyp for resurrecting this.

 

[depasseg]

If you have boot SSD's, I'd move the system dataset to that.

 

[Fraoch]

Well that solved it pretty conclusively. Thanks!

Boot pool:

Storage pool:

Makes sense when you think about it, the syslog and graphs are being continuously written to wherever the system dataset is located.

 

[kyp]

Very good, I'm using a 16GB SATADOM for boot - so I think it should also be fine to move system dataset there.
What typically is the disk usage / size of the system dataset?

According to the documentation, "The system dataset stores debugging core files and Samba4 metadata such as the user/group cache and share level permissions".
So what is the impact of doing a future fresh install of FreeNAS?

I'm not so worried about debugging, system logs or reporting information being wiped but what about Samba4 metadata and share level permissions?
Is this something that would be re-created when importing the ZFS pool?
Or is the CIFS share destroyed and has to be manually re-created in this scenario?

 

[depasseg]

If you lose the boot device, you will lose your Cifs configuration. So you will want to make sure you have your config backed up. You can also mirror the boot drive to protect against a device failure.