SOLVED Most reports are blank, TrueNAS-SCALE-22.12.1

[SmurfHunter]

Hello all, I believe this is the right place to post, please correct me if I'm wrong.

I have spent the last few days ill and working on a few projects I had on the back burner - one of them being a NAS setup. I have four Hitachi 3TB SAS drives on a Dell H310 flashed into IT mode. My motherboard is an Asus board running 2x4GB DDR4 and an i5-7500. If the motherboard model is relevant I can find it, but I do not immediately have the model in front of me. The model of the Hitachi drives, which are 7200 RPM, are HMRSK3000GBAS07K. Attached is an example smartctl -a output - I'm getting functional outputs for every drive. Note that I'm running a full smart test on them right now, which is accurately reflected in the output.

I have seen people with this issue with both core and scale, and have tried a few things. What I find most odd about this issue is it came about following an sg_format of the disks from 520 to 512 byte sectors - they were reporting accurately before.

I have rebooted the sector, repeatedly.

I deleted the pool and remade it, no luck.

I went to reporting > disks > settings > changed the points and length of the graph to clear results, still not reporting.

I feel it's also noteworthy that my NVMe drive is reporting temperatures over smart, but is NOT reporting them into reporting - that comes up null too, as well as all of ZFS's reporting info and all the drive's read and writes. CPU and memory temperature and usage are reporting.

Does anyone have suggestions here? Any logs I can give that might indicate what's going on here? Any assistance is appreciated - as you can probably tell by the temperatures, I'm fighting my PC case to get adequate cooling, and temperature monitoring is something I'd love to have, along with ZFS cache reporting. Thanks for any help offered :)

 

[Samuel Tai]

You could try to reset Reporting altogether via midclt call reporting.clear.

 

[SmurfHunter]

Running that command outputs the word "Null." I've let it sit for about five minutes with no update to reporting. I'll keep an eye on it and report back after the smart long tests if it's still not working.

 

[Samuel Tai]

Yes, that's the proper output. It wipes out all the saved data, so Reporting can rebuild it starting fresh.

 

[SmurfHunter]

Looks like none of the report tabs are populating at all now. Rebooting did not resolve that. It's all null now

 

[Samuel Tai]

Are your reporting processes running? What's the output of pgrep -fl collectd and pgrep -fl rrdcached?

 

[SmurfHunter]

4404 collectd
2915 rrdcached
Smart service is also running it seems.

 

[Samuel Tai]

Under /var/db/system, do you have multiple rrd-* directories?

 

[SmurfHunter]

Looks like just a single rrd-* directory.

 

[Samuel Tai]

This is brute force, but try stopping collectd and rrdcached via service collectd stop and service rrdcached stop. Then cd /var/db/system/rrd-8f179c8648fc4419af075a5cf26c19f8/localhost. There should be folders for all the reporting categories. Delete them all via rm -rf *. (Make sure you're in the localhost folder before you try the delete.) Then restart the collectd and rrdcached processes via service collectd start and service rrdcached start to have the Reporting rebuild all the folders.

 

[winnielinnie]

(Make sure you're in the localhost folder before you try the delete.)

Make a recursive snapshot of your root datasets for all pools.

 

[SmurfHunter]

This is cursed. Stopped services, pulled a whole copy of the rrd folder, wiped the rrd localhost folder and started the processes, no dice. Rebooted, no dice. Cleared out the whole rrd folder for the heck of it, still no dice. Put back the journal and truenas.local folders, still no luck. Everything is reading null.

 

[Samuel Tai]

Did the folders repopulate, and are there *.rrd files in each folder?

 

[SmurfHunter]

Localhost I left deleted and it repopulated. Screenshot of all the folders that are inside the folder.

 

[Samuel Tai]

OK, within each of the folders there should be files with .rrd extensions; this is the actual graph data. If there are, then the Reporting graphs should eventually repopulate. If there aren't, then there's a different problem going on.

 

[SmurfHunter]

.rrd files are present in a couple folders I checked. Graphs still not repopulating. Tried a recursive chmod 744 and 777 in case it's a permissions issue, graphs still not populating. How long should the graphs take to repopulate?

 

[Samuel Tai]

Permissions should be 644 on the .rrd files. You may need to wait until all folders have populated with .rrd files.

 

[SmurfHunter]

Thank you for the help so far! I'll check all the folders tomorrow and check to see if reporting has restored and report back.

 

[SmurfHunter]

Hey guys, just a quick update. I have checked all the folders and they have at least a few .rrd files, yet reporting remains null. I'll be able to work further on this tonight when I get home. Is there anything else to check for this issue? Given the age of the NAS, I'm genuinely considering backing the config up and doing a clean install to get reporting back. I only have a freshly dragged terabyte over that I can just re-copy, so I don't even need the config.

 

[SmurfHunter]

Reporting clear!