The Secret Life of TrueNAS

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Hi
I have a question that's tricky to classify, perhaps because of it's very general nature. Here goes. From time to time, I notice my NAS drives are chundering away to themselves - lots of furious activity, judging from the sounds of heads skittering to and fro (sounds like underwater gargling actually!). When this is happening, I have no tasks being performed, no backups running, no plugins active. Additionally, no scheduled tasks happening either. No cron, no replication, no snapshots, no scrubs. And so what on earth is the NAS up to with itself? Why all this furious activity? Mostly read I note, although not a complete absence of write. And this is not the system disk either - that's a ssd, so silent. This is full scale riotious hard disk access, from time to time when unattended.

What on earth is the NAS up to?

How can I find out what it's doing? I'm no CLI person and I have minimal knowledge of FreeBSD and ZFS. I tried running 'top'; that didn't tell me much. Some unrecognised acronyms plus a few pythons (not enough for a circus) and all seemingly very small in both memory and cpu useage. So - does anyone have an idea how I can go about gaining an insight into what on earth it's doing in these energetic episodes - which can last for over an hour by the way? Thanks!
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
While it's happening you could run fstat and/or htop to try to figure it out.
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,448
Offhand I can think of a few things. There are processes in the middleware to monitor drives, SMART data, collectd (which stores performance data, etc)
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
While it's happening you could run fstat and/or htop to try to figure it out.
Thanks Sretalla. Although it's not happening at the moment, I tried both these commands just to see what comes back. Fstat in particular threw yards of data at me, none of which I can understand. I can say that most (what are they, processes?) get many many entries each.

Perhaps I can post a sample from fstat the next time it happens, and maybe some kind soul can give me some pointers on how to read it's results?!!
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Offhand I can think of a few things. There are processes in the middleware to monitor drives, SMART data, collectd (which stores performance data, etc)
Actually I set up Grafana which gives lots of eye-candy encapsulated feedback, but in this case it can't help, beyond visually confirming the drives really are read/writing a lot - so it's not an auditory hallucination at least. However while it documents it's happening, it's beyond it's capabilities to tell me what's behind it; what's driving all that intense & sustained activity.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Perhaps I can post a sample from fstat the next time it happens, and maybe some kind soul can give me some pointers on how to read it's results?!!
Sure. Go for it.
 

NumberSix

Contributor
Joined
Apr 9, 2021
Messages
188
Sure. Go for it.


Well it's a lot like a phenomena I know well whereby upon visiting a Doctor with a worrying complaint, all symptoms vanish - just - while the Doctor is looking for them, then they return later. In this case, the hard disks haven't gone into frenzied activity for the last 36 hours or so while I've been waiting to catch them at it and run fstab. That said, they are now doing bursts of activity in a pattern of On for one second, Off for 5 seconds, and then repeat. Occasionally silent for a minute. I'd call it 'deranged metronome' mode.

I ran fstat while this was happening. There are no scheduled tasks of any kind, by the way, and for completeness, I switched off plugins Syncthing and Plex, but left Grafana on, hoping it might help with diagnosing this mysterious activity.

root python3.8 1513 3 /dev 5 crw-r--r-- random r
root cron 1443 text / 133297 -r-xr-xr-x 45760 r
root cron 1443 wd /var 113 drwxr-x--- 64 r
root cron 1443 root / 34 drwxr-xr-x 27 r
root cron 1443 0 /dev 45 crw-rw-rw- null rw
root cron 1443 1 /dev 45 crw-rw-rw- null rw
root cron 1443 2 /dev 45 crw-rw-rw- null rw
root cron 1443 3 /var 84 drwxr-xr-x 2368 r
root cron 1443 4 /var 700 -rw------- 4 w
avahi avahi-daemon 1408 text / 114416 -rwxr-xr-x 118968 r
avahi avahi-daemon 1408 wd / 34 drwxr-xr-x 27 r
avahi avahi-daemon 1408 root / 34 drwxr-xr-x 27 r
avahi avahi-daemon 1408 0 /dev 45 crw-rw-rw- null r
avahi avahi-daemon 1408 1 /dev 45 crw-rw-rw- null w
avahi avahi-daemon 1408 2 /dev 45 crw-rw-rw- null w
avahi avahi-daemon 1408 3* local dgram fffff800117fbc00 <-> fffff800117fb500
avahi avahi-daemon 1408 5* pipe fffff800068812f8 <-> fffff80006881460 0 rw
avahi avahi-daemon 1408 6* pipe fffff80006881460 <-> fffff800068812f8 0 rw
avahi avahi-daemon 1408 7* pipe fffff800068932f8 <-> fffff80006893460 0 rw
avahi avahi-daemon 1408 8* pipe fffff80006893460 <-> fffff800068932f8 0 rw
avahi avahi-daemon 1408 9* local stream fffff800117fbd00
avahi avahi-daemon 1408 10
avahi avahi-daemon 1408 11 /etc 120 drwxr-xr-x 448 r
avahi avahi-daemon 1408 12 /etc 123 drwxr-xr-x 384 r
avahi avahi-daemon 1408 13* internet dgram udp fffff8008a0323d0
avahi avahi-daemon 1408 14* internet dgram udp fffff8008a0321e8
avahi avahi-daemon 1408 15* route raw 0 0
messageb dbus-daemon 1384 text / 8315 -rwxr-xr-x 205184 r
messageb dbus-daemon 1384 wd / 34 drwxr-xr-x 27 r
messageb dbus-daemon 1384 root / 34 drwxr-xr-x 27 r
messageb dbus-daemon 1384 0 /dev 45 crw-rw-rw- null rw
messageb dbus-daemon 1384 1 /dev 45 crw-rw-rw- null rw
messageb dbus-daemon 1384 2 /dev 45 crw-rw-rw- null rw
messageb dbus-daemon 1384 3* local stream fffff80011b63a00
messageb dbus-daemon 1384 4
messageb dbus-daemon 1384 5 / 149055 drwxr-xr-x 2 r
messageb dbus-daemon 1384 6 /etc 437 drwxr-xr-x 128 r
messageb dbus-daemon 1384 7 / 115338 drwxr-xr-x 3 r
messageb dbus-daemon 1384 8* local stream fffff800117fbe00 <-> fffff80011811d00
messageb dbus-daemon 1384 9* local stream fffff80011811d00 <-> fffff800117fbe00
www nginx 1379 text / 114588 -r-xr-xr-x 1021880 r
www nginx 1379 wd / 34 drwxr-xr-x 27 r
www nginx 1379 root / 34 drwxr-xr-x 27 r
www nginx 1379 0 /dev 45 crw-rw-rw- null rw
www nginx 1379 1 /dev 45 crw-rw-rw- null rw
www nginx 1379 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 147 -rw------- 342 w
www nginx 1379 3* internet stream tcp fffff8009c17c3d0
www nginx 1379 4 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 147 -rw------- 342 w
www nginx 1379 5* internet stream tcp fffff8009c17b000
www nginx 1379 6* internet6 stream tcp fffff8009c17b3d0
www nginx 1379 7* internet stream tcp fffff8009c17b7a0
www nginx 1379 8* internet6 stream tcp fffff8009c17bb70
www nginx 1379 9* local stream fffff80011b63b00 <-> fffff80011b63c00
www nginx 1379 10
www nginx 1379 11* internet stream tcp fffff8009c17d3d0
www nginx 1379 14* local dgram fffff80011d7b400 <-> fffff800117fb300
www nginx 1379 15* internet stream tcp fffff800b91627a0
www nginx 1379 18* internet stream tcp fffff80202b8d000
root nginx 1378 text / 114588 -r-xr-xr-x 1021880 r
root nginx 1378 wd / 34 drwxr-xr-x 27 r
root nginx 1378 root / 34 drwxr-xr-x 27 r
root nginx 1378 0 /dev 45 crw-rw-rw- null rw
root nginx 1378 1 /dev 45 crw-rw-rw- null rw
root nginx 1378 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 147 -rw------- 342 w
root nginx 1378 3* local stream fffff80011b63c00 <-> fffff80011b63b00
root nginx 1378 4 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 147 -rw------- 342 w
root nginx 1378 5* internet stream tcp fffff8009c17b000
root nginx 1378 6* internet6 stream tcp fffff8009c17b3d0
root nginx 1378 7* internet stream tcp fffff8009c17b7a0
root nginx 1378 8* internet6 stream tcp fffff8009c17bb70
root nginx 1378 9* local stream fffff80011b63b00 <-> fffff80011b63c00
root winbindd 1376 text / 114736 -rwxr-xr-x 1146208 r
root winbindd 1376 wd / 34 drwxr-xr-x 27 r
root winbindd 1376 root / 34 drwxr-xr-x 27 r
root winbindd 1376 0 /dev 45 crw-rw-rw- null rw
root winbindd 1376 1 /dev 45 crw-rw-rw- null rw
root winbindd 1376 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 150 -rw-r--r-- 4352 w
root winbindd 1376 3 /dev 45 crw-rw-rw- null rw
root winbindd 1376 4 /var/db/system/samba4 136 -rw------- 430080 rw
root winbindd 1376 5 /var 682 -rw-r--r-- 20 rw
root winbindd 1376 6* local dgram fffff80011da0000
root winbindd 1376 7 /var 602 -rw-r----- 12288 rw
root winbindd 1376 8* pipe fffff800068928e8 <-> fffff80006892a500 rw
root winbindd 1376 9 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root winbindd 1376 10 /var/db/system/samba4 130 -rw------- 528384 rw
root winbindd 1376 11 /var/db/system/samba4 134 -rw------- 4792 rw
root winbindd 1376 12 /dev 45 crw-rw-rw- null rw
root winbindd 1376 13 /var 666 -rw-r--r-- 5 w
root winbindd 1376 14
root winbindd 1376 15* pipe fffff80006880be0 <-> fffff80006880d480 rw
root winbindd 1376 16 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 150 -rw-r--r-- 4352 w
root winbindd 1376 17* pipe fffff80006892a50 <-> fffff800068928e80 rw
root winbindd 1376 19 /var 668 -rw------- 696 rw
root winbindd 1376 20 /var/db/system/samba4 277 -rw------- 32768 rw
root winbindd 1376 21* local stream fffff800117fe100
root winbindd 1376 22* local stream fffff80011d81c00
root winbindd 1376 25 /var 616 -rw-r--r-- 135168 rw
root winbindd 1376 26* local stream fffff800117fb400 <-> fffff800117fb800
root rrdcached 1368 text / 9586 -rwxr-xr-x 79448 r
root rrdcached 1368 wd /var 40 drwxrwxrwt 2496 r
root rrdcached 1368 root / 34 drwxr-xr-x 27 r
root rrdcached 1368 0 /dev 45 crw-rw-rw- null rw
root rrdcached 1368 1 /dev 45 crw-rw-rw- null rw
root rrdcached 1368 2 /dev 45 crw-rw-rw- null rw
root rrdcached 1368 3* local stream fffff801eb93f300 <-> fffff801eb92fd00
root rrdcached 1368 4* local stream fffff800119e9a00
root winbindd 1367 text / 114736 -rwxr-xr-x 1146208 r
root winbindd 1367 wd / 34 drwxr-xr-x 27 r
root winbindd 1367 root / 34 drwxr-xr-x 27 r
root winbindd 1367 0 /dev 45 crw-rw-rw- null rw
root winbindd 1367 1 /dev 45 crw-rw-rw- null rw
root winbindd 1367 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 153 -rw-r--r-- 4216 w
root winbindd 1367 3 /dev 45 crw-rw-rw- null rw
root winbindd 1367 4 /var/db/system/samba4 136 -rw------- 430080 rw
root winbindd 1367 5 /var 681 -rw-r--r-- 21 rw
root winbindd 1367 6* local dgram fffff800117fba00
root winbindd 1367 7 /var 602 -rw-r----- 12288 rw
root winbindd 1367 8* pipe fffff8000ae705f0 <-> fffff8000ae707580 rw
root winbindd 1367 9 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root winbindd 1367 10 /var/db/system/samba4 130 -rw------- 528384 rw
root winbindd 1367 11 /var/db/system/samba4 134 -rw------- 4792 rw
root winbindd 1367 12 /dev 45 crw-rw-rw- null rw
root winbindd 1367 13 /var 666 -rw-r--r-- 5 w
root winbindd 1367 14
root winbindd 1367 15* pipe fffff80006880be0 <-> fffff80006880d480 rw
root winbindd 1367 16 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 153 -rw-r--r-- 4216 w
root winbindd 1367 17* pipe fffff8000ae70758 <-> fffff8000ae705f00 rw
root winbindd 1367 18 /var/db/system/samba4 133 -rw-r--r-- 421888 rw
root winbindd 1367 19 /var 668 -rw------- 696 rw
root winbindd 1367 20 /var/db/system/samba4 277 -rw------- 32768 rw
root winbindd 1367 21* local stream fffff800117fe100
root winbindd 1367 22* local stream fffff80011d81c00
root winbindd 1367 23 /var/db/system/samba4 129 -rw------- 425984 rw
root winbindd 1367 24 /var/db/system/samba4 135 -rw------- 421888 rw
root winbindd 1367 25 /var 616 -rw-r--r-- 135168 rw
root winbindd 1367 26* local stream fffff800117fb900 <-> fffff80011d7b800
root winbindd 1367 27 /var/db/system/samba4 128 -rw------- 421888 rw
root smbd 1365 text / 114676 -rwxr-xr-x 88368 r
root smbd 1365 wd / 34 drwxr-xr-x 27 r
root smbd 1365 root / 34 drwxr-xr-x 27 r
root smbd 1365 0 /dev 45 crw-rw-rw- null rw
root smbd 1365 1 /dev 45 crw-rw-rw- null rw
root smbd 1365 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1365 3 /dev 45 crw-rw-rw- null rw
root smbd 1365 4 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root smbd 1365 5 /var 678 -rw-r--r-- 20 rw
root smbd 1365 6* local dgram fffff800119e8200
root smbd 1365 7 /var 602 -rw-r----- 12288 rw
root smbd 1365 8 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1365 9 /var/db/system/samba4 136 -rw------- 430080 rw
root smbd 1365 10 /var/db/system/samba4 130 -rw------- 528384 rw
root smbd 1365 11 /var 656 -rw-r--r-- 4792 rw
root smbd 1365 12 /dev 45 crw-rw-rw- null rw
root smbd 1365 13 /var 657 -rw-r--r-- 5 w
root smbd 1365 14
root smbd 1365 15* pipe fffff801af33f2f8 <-> fffff801af33f4600 rw
root smbd 1365 16 /var 1169 -rw-r--r-- 12288 rw
root smbd 1365 19 /var 659 -rw------- 20480 rw
root smbd 1365 20 /var 660 -rw------- 4792 rw
root smbd 1365 21 /var 661 -rw------- 40960 rw
root smbd 1365 22 /var 662 -rw------- 20480 rw
root smbd 1365 23 /var 663 -rw-r--r-- 122120 rw
root smbd 1365 24 /var 664 -rw-r--r-- 835584 rw
root smbd 1365 25 /var 672 -rw-r--r-- 962560 rw
root smbd 1365 26 /var 676 -rw-r--r-- 77824 rw
root smbd 1364 text / 114676 -rwxr-xr-x 88368 r
root smbd 1364 wd / 34 drwxr-xr-x 27 r
root smbd 1364 root / 34 drwxr-xr-x 27 r
root smbd 1364 0 /dev 45 crw-rw-rw- null rw
root smbd 1364 1 /dev 45 crw-rw-rw- null rw
root smbd 1364 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1364 3 /dev 45 crw-rw-rw- null rw
root smbd 1364 4* pipe fffff800068935f0 <-> fffff800068937580 rw
root smbd 1364 5 /var 677 -rw-r--r-- 21 rw
root smbd 1364 6* local dgram fffff80011d8a900
root smbd 1364 7 /var 602 -rw-r----- 12288 rw
root smbd 1364 8 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root smbd 1364 9 /var/db/system/samba4 136 -rw------- 430080 rw
root smbd 1364 10 /var/db/system/samba4 130 -rw------- 528384 rw
root smbd 1364 11 /var 656 -rw-r--r-- 4792 rw
root smbd 1364 12 /dev 45 crw-rw-rw- null rw
root smbd 1364 13 /var 657 -rw-r--r-- 5 w
root smbd 1364 14
root smbd 1364 15* pipe fffff801af33f2f8 <-> fffff801af33f4600 rw
root smbd 1364 16 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1364 17* pipe fffff80006893758 <-> fffff800068935f00 rw
root smbd 1364 19 /var 659 -rw------- 20480 rw
root smbd 1364 20 /var 660 -rw------- 4792 rw
root smbd 1364 21 /var 661 -rw------- 40960 rw
root smbd 1364 22 /var 662 -rw------- 20480 rw
root smbd 1364 23 /var 663 -rw-r--r-- 122120 rw
root smbd 1364 24 /var 664 -rw-r--r-- 835584 rw
root smbd 1364 25 /var 672 -rw-r--r-- 962560 rw
root smbd 1364 26 /var 676 -rw-r--r-- 77824 rw
daemon python3.8 1343 text / 9550 -r-xr-xr-x 5368 r
daemon python3.8 1343 wd /var 70 dr-xr-xr-x 0 r
daemon python3.8 1343 root /var 70 dr-xr-xr-x 0 r
daemon python3.8 1343 jail /var 70 dr-xr-xr-x 0 r
daemon python3.8 1343 0 /dev 45 crw-rw-rw- null rw
daemon python3.8 1343 1* pipe fffff800b918f168 <-> fffff800b918f0000 rw
daemon python3.8 1343 2* pipe fffff800b918f168 <-> fffff800b918f0000 rw
daemon python3.8 1343 3 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 143 -rw-r--r-- 0 w
daemon python3.8 1343 4 /var/db/system/samba4 130 -rw------- 528384 rw
daemon python3.8 1343 5* local stream fffff800117fb200 <-> fffff800117fb100
daemon python3.8 1343 6* local stream fffff800117fb000 <-> fffff800117ffe00
daemon python3.8 1343 7
daemon python3.8 1343 8* internet dgram udp fffff800119fa3d0
daemon python3.8 1343 9* internet dgram udp fffff800119fa5b8
daemon python3.8 1343 10* internet stream tcp fffff800b915eb70
root daemon 1342 text / 133307 -r-xr-xr-x 15880 r
root daemon 1342 wd / 34 drwxr-xr-x 27 r
root daemon 1342 root / 34 drwxr-xr-x 27 r
root daemon 1342 0 /dev 45 crw-rw-rw- null rw
root daemon 1342 1 /dev 45 crw-rw-rw- null rw
root daemon 1342 2 /dev 45 crw-rw-rw- null rw
root daemon 1342 3* local dgram fffff80011811200 <-> fffff800117fb500
root daemon 1342 4 /var 98 drwxr-xr-x 1408 r
root daemon 1342 5 /var 675 -rw------- 4 w
root daemon 1342 6* pipe fffff800b918f000 <-> fffff800b918f1680 rw
root winbindd 1326 text / 114736 -rwxr-xr-x 1146208 r
root winbindd 1326 wd / 34 drwxr-xr-x 27 r
root winbindd 1326 root / 34 drwxr-xr-x 27 r
root winbindd 1326 0 /dev 45 crw-rw-rw- null rw
root winbindd 1326 1 /dev 45 crw-rw-rw- null rw
root winbindd 1326 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 151 -rw-r--r-- 2584 w
root winbindd 1326 3 /dev 45 crw-rw-rw- null rw
root winbindd 1326 4 /var/db/system/samba4 136 -rw------- 430080 rw
root winbindd 1326 5 /var 671 -rw-r--r-- 21 rw
root winbindd 1326 6* local dgram fffff80011d81600
root winbindd 1326 7 /var 602 -rw-r----- 12288 rw
root winbindd 1326 8* pipe fffff800b9190000 <-> fffff800b91901680 rw
root winbindd 1326 9 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root winbindd 1326 10 /var/db/system/samba4 130 -rw------- 528384 rw
root winbindd 1326 11 /var/db/system/samba4 134 -rw------- 4792 rw
root winbindd 1326 12 /dev 45 crw-rw-rw- null rw
root winbindd 1326 13 /var 666 -rw-r--r-- 5 w
root winbindd 1326 14
root winbindd 1326 15* pipe fffff80006880be0 <-> fffff80006880d480 rw
root winbindd 1326 16 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 151 -rw-r--r-- 2584 w
root winbindd 1326 17* pipe fffff800b9190168 <-> fffff800b91900000 rw
root winbindd 1326 18 /var 616 -rw-r--r-- 135168 rw
root winbindd 1326 19 /var 668 -rw------- 696 rw
root winbindd 1326 20 /var/db/system/samba4 277 -rw------- 32768 rw
root winbindd 1326 21* local stream fffff800117fe100
root winbindd 1326 22* local stream fffff80011d81c00
root winbindd 1326 23* local stream fffff800117fb700 <-> fffff80011d81400
root winbindd 1326 25 /var/db/system/samba4 135 -rw------- 421888 rw
root winbindd 1326 26 /var/db/system/samba4 129 -rw------- 425984 rw
root winbindd 1326 27 /var/db/system/samba4 128 -rw------- 421888 rw
root winbindd 1321 text / 114736 -rwxr-xr-x 1146208 r
root winbindd 1321 wd / 34 drwxr-xr-x 27 r
root winbindd 1321 root / 34 drwxr-xr-x 27 r
root winbindd 1321 0 /dev 45 crw-rw-rw- null rw
root winbindd 1321 1 /dev 45 crw-rw-rw- null rw
root winbindd 1321 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 152 -rw-r--r-- 35197 w
root winbindd 1321 3 /dev 45 crw-rw-rw- null rw
root winbindd 1321 4 /var/db/system/samba4 136 -rw------- 430080 rw
root winbindd 1321 5 /var 667 -rw-r--r-- 21 rw
root winbindd 1321 6* local dgram fffff80011d81300
root winbindd 1321 7 /var 602 -rw-r----- 12288 rw
root winbindd 1321 8 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root winbindd 1321 9 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 152 -rw-r--r-- 35197 w
root winbindd 1321 10 /var/db/system/samba4 130 -rw------- 528384 rw
root winbindd 1321 11 /var/db/system/samba4 134 -rw------- 4792 rw
root winbindd 1321 12 /dev 45 crw-rw-rw- null rw
root winbindd 1321 13 /var 666 -rw-r--r-- 5 w
root winbindd 1321 14
root winbindd 1321 15* pipe fffff80006880be0 <-> fffff80006880d480 rw
root winbindd 1321 16* pipe fffff80006880d48 <-> fffff80006880be00 rw
root winbindd 1321 17* pipe fffff8000ae70be0 <-> fffff8000ae70d480 rw
root winbindd 1321 18* pipe fffff8000ae70d48 <-> fffff8000ae70be00 rw
root winbindd 1321 19 /var 668 -rw------- 696 rw
root winbindd 1321 20 /var/db/system/samba4 277 -rw------- 32768 rw
root winbindd 1321 21* local stream fffff800117fe100
root winbindd 1321 22* local stream fffff80011d81c00
root winbindd 1321 24* local stream fffff80011d81400 <-> fffff800117fb700
root winbindd 1321 25 /var 616 -rw-r--r-- 135168 rw
root winbindd 1321 27* local stream fffff80011d7b800 <-> fffff800117fb900
root winbindd 1321 28* local stream fffff800117fb800 <-> fffff800117fb400
root smbd 1313 text / 114676 -rwxr-xr-x 88368 r
root smbd 1313 wd / 34 drwxr-xr-x 27 r
root smbd 1313 root / 34 drwxr-xr-x 27 r
root smbd 1313 0 /dev 45 crw-rw-rw- null rw
root smbd 1313 1 /dev 45 crw-rw-rw- null rw
root smbd 1313 2 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1313 3 /dev 45 crw-rw-rw- null rw
root smbd 1313 4 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 388 -rw-r--r-- 426695 w
root smbd 1313 5 /var 658 -rw-r--r-- 20 rw
root smbd 1313 6* local dgram fffff80011da0100
root smbd 1313 7 /var 602 -rw-r----- 12288 rw
root smbd 1313 8 /var/db/system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3 387 -rw-r--r-- 2076498 w
root smbd 1313 9 /var/db/system/samba4 136 -rw------- 430080 rw
root smbd 1313 10 /var/db/system/samba4 130 -rw------- 528384 rw
root smbd 1313 11 /var 656 -rw-r--r-- 4792 rw
root smbd 1313 12 /dev 45 crw-rw-rw- null rw
root smbd 1313 13 /var 657 -rw-r--r-- 5 w
root smbd 1313 14
root smbd 1313 15* pipe fffff801af33f2f8 <-> fffff801af33f4600 rw
root smbd 1313 16* pipe fffff801af33f460 <-> fffff801af33f2f80 rw
root smbd 1313 17* pipe fffff801af33f000 <-> fffff801af33f1680 rw
root smbd 1313 18* pipe fffff801af33f168 <-> fffff801af33f0000 rw
root smbd 1313 19 /var 659 -rw------- 20480 rw
root smbd 1313 20 /var 660 -rw------- 4792 rw
root smbd 1313 21 /var 661 -rw------- 40960 rw
root smbd 1313 22 /var 662 -rw------- 20480 rw
root smbd 1313 23 /var 663 -rw-r--r-- 122120 rw
root smbd 1313 24 /var 664 -rw-r--r-- 835584 rw
root smbd 1313 25 /var 672 -rw-r--r-- 962560 rw
root smbd 1313 26 /var 676 -rw-r--r-- 77824 rw
root smbd 1313 27 /var/db/system/samba4 131 -rw------- 421888 rw
root smbd 1313 28 /var 616 -rw-r--r-- 135168 rw
root smbd 1313 29* local stream fffff800117fbb00
root smbd 1313 30 /var/db/system/samba4 129 -rw------- 425984 rw
root smbd 1313 31 /var/db/system/samba4 128 -rw------- 421888 rw
root smbd 1313 32 /var/db/system/samba4 135 -rw------- 421888 rw
root smbd 1313 33 /var 683 -rw------- 32768 rw
root smbd 1313 34* internet6 stream tcp fffff800adf113d0
root smbd 1313 35* internet6 stream tcp fffff800adf11000
root smbd 1313 36* internet stream tcp fffff800adf15b70
root smbd 1313 37* internet stream tcp fffff800adf157a0
root smbd 1313 38 /var 1169 -rw-r--r-- 12288 rw
ntpd ntpd 1277 text / 133506 -r-xr-xr-x 861632 r
ntpd ntpd 1277 wd / 34 drwxr-xr-x 27 r
ntpd ntpd 1277 root / 34 drwxr-xr-x 27 r
ntpd ntpd 1277 0 /dev 45 crw-rw-rw- null r
ntpd ntpd 1277 1 /dev 45 crw-rw-rw- null r
ntpd ntpd 1277 2 /dev 45 crw-rw-rw- null r
ntpd ntpd 1277 3* local dgram fffff80011d8a800 <-> fffff800117fb300
ntpd ntpd 1277 20* internet6 dgram udp fffff800b994c988
ntpd ntpd 1277 21* internet dgram udp fffff800b994cb70
ntpd ntpd 1277 22* internet dgram udp fffff8008a0317a0
ntpd ntpd 1277 23* internet6 dgram udp fffff800b994c1e8
ntpd ntpd 1277 24* internet6 dgram udp fffff800b9946988
ntpd ntpd 1277 25* internet dgram udp fffff800b99465b8
ntpd ntpd 1277 26* route raw 0 0
root watchdogd 1189 text / 133680 -r-xr-xr-x 15656 r
root watchdogd 1189 wd / 34 drwxr-xr-x 27 r
root watchdogd 1189 root / 34 drwxr-xr-x 27 r
root watchdogd 1189 0 /dev 45 crw-rw-rw- null rw
root watchdogd 1189 1 /dev 45 crw-rw-rw- null rw
root watchdogd 1189 2 /dev 45 crw-rw-rw- null rw
root watchdogd 1189 3* local dgram fffff80011d81200 <-> fffff800117fb500
root watchdogd 1189 4 /dev 36 crw------- fido rw
root watchdogd 1189 5 /var 84 drwxr-xr-x 2368 r
root watchdogd 1189 6 /var 646 -rw------- 4 w
root syslog-ng 1080 text / 114692 -rwxr-xr-x 14176 r
root syslog-ng 1080 wd /var 7 drwxr-xr-x 1280 r
root syslog-ng 1080 root / 34 drwxr-xr-x 27 r
root syslog-ng 1080 0 /dev 45 crw-rw-rw- null r
root syslog-ng 1080 1 /dev 45 crw-rw-rw- null w
root syslog-ng 1080 2 /dev 45 crw-rw-rw- null w
root syslog-ng 1080 3
root syslog-ng 1080 4* pipe fffff801af950a50 <-> fffff801af9508e80 rw
root syslog-ng 1080 6* pipe fffff801af950000 <-> fffff801af9501680 rw


[Some redacted here to meet the <30000 character post limit]



root devd 294 text / 6595 -r-xr-xr-x 934392 r
root devd 294 wd / 34 drwxr-xr-x 27 r
root devd 294 root / 34 drwxr-xr-x 27 r
root devd 294 0 /dev 45 crw-rw-rw- null rw
root devd 294 1 /dev 45 crw-rw-rw- null rw
root devd 294 2 /dev 45 crw-rw-rw- null rw
root devd 294 3 /dev 7 crw------- devctl r
root devd 294 4* local stream fffff800119e7000
root devd 294 5* local seqpak fffff800119e8e00
root devd 294 6 /var 84 drwxr-xr-x 2368 r
root devd 294 7 /var 119 -rw------- 3 w
root devd 294 8* local seqpak fffff80011d88100 -> fffff80011d88200
root devd 294 9* local stream fffff80011811500 -> fffff80011811600
root init 1 text / 3258 -r-xr-xr-x 1194072 r
root init 1 wd / 34 drwxr-xr-x 27 r
root init 1 root / 34 drwxr-xr-x 27 r
root kernel 0 wd / 34 drwxr-xr-x 27 r
root kernel 0 root / 34 drwxr-xr-x 27 r
root@truenas[~]#
root@truenas[~]#
root@truenas[~]#

Note, I had to chop LOTS AND LOTS (maybe as much as half) out of the fstab result because there's a limit (of 30000) to the number of characters I can put in a post it seems.

So can I ask, what does fstat show here - and in the 'teach a man to fish' tradition, how do you know what it's saying? Morever, does it give a real answer to the question "why are the disks never resting?".

Thank you!

Edit:
I decided to try the old 'have you tried turning it off and on again' technique. Actually I rebooted, rather than turning off. The issue went away entirely. Here's some Grafana output either side of the reboot:
screenshot.2.jpg


Although I'm grateful the demented metronome has stopped, the question remains as ever - why was it happening? Exacerbated if anything by it's variability. I've seen this sort of thing before. TrueNAS is perfect for a while, then, maybe in the middle of the night, it's dementing away again. I'd really like to get to the bottom of this and fix it permanantly if I can, so that resets are not necessary to the long term stability of the system. Thanks again!
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
So can I ask, what does fstat show here - and in the 'teach a man to fish' tradition, how do you know what it's saying? Morever, does it give a real answer to the question "why are the disks never resting?".
Fstat does not show activity but simply all open file handles. The task would then be to scan them and weed out processes that can definitely not be the cause to end with a few remaining that warrant further investigation ...
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
So can I ask, what does fstat show here - and in the 'teach a man to fish' tradition, how do you know what it's saying? Morever, does it give a real answer to the question "why are the disks never resting?".
So the aim was to use a combination of htop (to find busy processes) and then use fstat to find which files the busiest processes had open, perhaps shedding some light on the situation.

I'll just drop you the link to the manpage for fstat (rather than trying to type out some kind of custom explanation of what it does (https://www.unix.com/man-page/FreeBSD/1/fstat/), but in essence you could use the -p or -u to restrict the list to the busy user or process and get some understanding of what files are being accesses, then do some research into why that might be.
 
Top