ARC completely missing on one of two systems?

[Patrick M. Hausen]

Hi all,

I now have two systems running this identical software:

Code:

12.0-MASTER-202006040424
FreeBSD freenas-pmh.local 12.1-STABLE FreeBSD 12.1-STABLE 34b610ce0f2(freenas/12-stable) TRUENAS  amd64

System 1, top:

Code:

last pid:  8870;  load averages:  0.91,  1.15,  1.06                                                        up 1+01:32:09  16:39:05
149 processes: 1 running, 147 sleeping, 1 zombie
CPU:  1.1% user,  0.0% nice,  4.2% system,  0.1% interrupt, 94.7% idle
Mem: 6503M Active, 9815M Inact, 10G Laundry, 33G Wired, 3480M Free
ARC: 27G Total, 7061M MFU, 19G MRU, 34M Anon, 1467M Header, 1296M Other
     24G Compressed, 31G Uncompressed, 1.26:1 Ratio
Swap: 16G Total, 600M Used, 15G Free, 3% Inuse

System 2, top:

Code:

last pid:  5819;  load averages:  0.49,  0.41,  0.27                                                        up 0+00:13:39  16:44:06
37 processes:  1 running, 36 sleeping
CPU:  1.9% user,  0.0% nice,  1.9% system,  0.0% interrupt, 96.2% idle
Mem: 994M Active, 91M Inact, 924M Wired, 14G Free
Swap: 16G Total, 16G Free

Note the complete absence of ARC stats. Likewise the second system reports 0 values via collectd --> Influx --> Grafana. (Which is where I noticed it first). First system looks good.
The hit ratio in Grafana is broken for both, but that's a secondary problem.

To file a proper issue in JIRA I would need some help. What lower level information (sysctl?) to look for on both systems?

Kind regards,
Patrick

 

[Patrick M. Hausen]

Well it looks like the stats are missing because the ARC is missing:

Code:

root@freenas2-pmh[~]# sysctl kstat.zfs.misc.arcstats.size
kstat.zfs.misc.arcstats.size: 0

Any idea what could lead a system to behave like this? Even with just 16G of RAM but no services running besides smartd and sshd there should be plenty of room for cache ...

Just slightly less than a G in "wired" state hints that there is indeed no ARC at work ...

 

[Yorick]

I’d file this with a debug and output from arc_summary.py (or was it arc-summary.py?)

 

[Patrick M. Hausen]

https://twitter.com/allanjude/status/1268990772684128256?s=21

 

[Ericloewe]

Probably worth a bug report, just to make sure it doesn't get forgotten.

 

[Patrick M. Hausen]

See the Twitter discussion, please. It's fixed in upstream.

 

[Ericloewe]

I did, I mean that the ticket may be important to avoid the fix being left behind for a while because it didn't make it into the next pull from upstream or whatever.

 

[Patrick M. Hausen]

https://jira.ixsystems.com/browse/NAS-106421

 

[freqlabs]

This is fixed now.