Weird write ops. on storage

[blckhm]

Hi everyone,

we're using freenas for serving static content such as jpg, gif etc. to our members on NFS share.

But we have weird operation behaviors on our disk drives.

unexpected write requests occurs on these drives.

"top -m io" shows us only READ ops. such as we expect

Code:

last pid: 74082;  load averages:  0.23,  0.28,  0.29   up 28+15:17:25  17:08:49
64 processes:  1 running, 63 sleeping
CPU:  0.1% user,  0.0% nice,  1.5% system,  0.2% interrupt, 98.2% idle
Mem: 17M Active, 814M Inact, 478M Laundry, 112G Wired, 3753M Free
ARC: 88G Total, 69G MFU, 11G MRU, 4619K Anon, 1858M Header, 6194M Other
     62G Compressed, 81G Uncompressed, 1.30:1 Ratio
Swap: 10G Total, 10G Free

  PID USERNAME       VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
2410 root            570      1    124      8      0    132 100.00% nfsd
98821 root            774      2      0      0      0      0   0.00% collectd
2922 root            262      1      0      0      0      0   0.00% rrdcached
2614 root              3      0      0      0      0      0   0.00% python3.6
1762 root             17      0      0      0      0      0   0.00% vmtoolsd
  252 root             17      1      0      0      0      0   0.00% python3.6
74059 root              3      0      0      0      0      0   0.00% top
2610 root              2      2      0      0      0      0   0.00% snmpd
2768 root              2      0      0      0      0      0   0.00% dtrace
2766 root              3      0      0      0      0      0   0.00% dtrace
99304 root              4      0      0      0      0      0   0.00% python3.6
2767 root              1      0      0      0      0      0   0.00% dtrace
57073 www               5      0      0      0      0      0   0.00% nginx
2675 root              2      0      0      0      0      0   0.00% ntpd

When we checked who's connected via nfs on storage with "netstat -an | grep {nfs_server_ip}.2049"
only 7 servers established such as we expect.

Code:

Proto Recv-Q Send-Q Local Address           Foreign Address          (state)
tcp4       0    124 nfs_server_ip.2049      nfs_client_ip1.731       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip2.787       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip3.640       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip4.813       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip5.804       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip6.792       ESTABLISHED
tcp4       0      0 nfs_server_ip.2049      nfs_client_ip7.863       ESTABLISHED

Finally, we checked nfs client stats on these 7 servers with "nfsstat -c", everything looks fine, a few (1-2) write operation occurred in 10 seconds.



But disk ops. charts shows us weird write operations on all of these disk drives.

Here is the one;
View attachment 30616


We could not find any reasonable meaning for these write operations. Is there something we missed?

How can we examine further ?

Kind Regards

 

[Chris Moore]

Why are you looking at this? Is there some malfunction that you are trying to diagnose?
Storage pool disks also hold the system dataset. There's going to be writing to that even if the server is idle.

 

[blckhm]

Why are you looking at this? Is there some malfunction that you are trying to diagnose?
Storage pool disks also hold the system dataset. There's going to be writing to that even if the server is idle.

Because we made some workload tests on different pools, and we could not find like this write footprints on these drives. So we started to think that, what are we missing?

On the other hand, freenas maybe updates last access times of files, or hit counts of files for caching stuff but we are curious about that.

 

[Chris Moore]

we made some workload tests on different pools

FreeNAS creates the system dataset in the first storage pool created when the system is built. Any other pool will not have the additional storage pool access to account for the system dataset.

On the other hand, freenas maybe updates last access times of files

This is very likely. It can be disabled if it is causing performance problems.

Is the system performing below expectation?

 

[blckhm]

FreeNAS creates the system dataset in the first storage pool created when the system is built. Any other pool will not have the additional storage pool access to account for the system dataset.

This is very likely. It can be disabled if it is causing performance problems.

Is the system performing below expectation?

This pool is the second pool that we created. So this pool does not contain any system dataset as you said.

There was not any performance problem at all, but these disks are always under write operations and disks are always busy.