Gravwell SYSLOG Queries

[NickF]

<div class="bbWrapper">Hi Everyone,<br /> <br /> I am working in my work life with a company called Gravwell (Free for Home Use, <a href="https://www.gravwell.io/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://www.gravwell.io/</a>). I have no affiliation with them other than being a customer of theirs. I asked their help to create a &quot;Kit&quot; to pull things out of the logs and make meaningful dashboards, as I have alot of crummy servers running TrueNAS (and FreeNAS still lol) for video surveillance applications. This may be a great tool in conjunction with <a href="https://www.truenas.com/community/members/101/" class="username" data-xf-init="member-tooltip" data-user-id="101" data-username="@joeschmuck">@joeschmuck</a>'s script. <b>Am I missing anything critical here?</b><br /> <br /> Some examples from my dataset:<br /> <script class="js-extraPhrases" type="application/json"> { "lightbox_close": "Close", "lightbox_next": "Next", "lightbox_previous": "Previous", "lightbox_error": "The requested content cannot be loaded. Please try again later.", "lightbox_start_slideshow": "Start slideshow", "lightbox_stop_slideshow": "Stop slideshow", "lightbox_full_screen": "Full screen", "lightbox_thumbnails": "Thumbnails", "lightbox_download": "Download", "lightbox_share": "Share", "lightbox_zoom": "Zoom", "lightbox_new_window": "New window", "lightbox_toggle_sidebar": "Toggle sidebar" } </script> <div class="bbImageWrapper js-lbImage" title="1676072063828.png" data-src="https://www.truenas.com/community/attachments/1676072063828-png.63450/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676072063828-png.63450/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676072063828.png" title="1676072063828.png" width="1920" height="828" loading="lazy" /> </div> <br /> <br /> <div class="bbImageWrapper js-lbImage" title="1676072172934.png" data-src="https://www.truenas.com/community/attachments/1676072172934-png.63451/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676072172934-png.63451/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676072172934.png" title="1676072172934.png" width="1559" height="866" loading="lazy" /> </div> <br /> <div class="bbImageWrapper js-lbImage" title="1676072234322.png" data-src="https://www.truenas.com/community/attachments/1676072234322-png.63452/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676072234322-png.63452/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676072234322.png" title="1676072234322.png" width="1883" height="818" loading="lazy" /> </div> <br /> <br /> <div class="bbImageWrapper js-lbImage" title="1676074878485.png" data-src="https://www.truenas.com/community/attachments/1676074878485-png.63453/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676074878485-png.63453/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676074878485.png" title="1676074878485.png" width="1372" height="592" loading="lazy" /> </div> <br /> <div class="bbImageWrapper js-lbImage" title="1676074929891.png" data-src="https://www.truenas.com/community/attachments/1676074929891-png.63454/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676074929891-png.63454/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676074929891.png" title="1676074929891.png" width="1920" height="845" loading="lazy" /> </div></div>

 

[joeschmuck]

<div class="bbWrapper">Nick,<br /> One of the things I wanted to do but it's outside my wheelhouse was to create a dashboard or GUI that runs on TrueNAS and provides the data that the Multi-Report script provides and the adjustable alarm setpoints. I have not examined the TrueNAS logs so I'm not sure what else can be displayed. If I understand you correctly, the data provided above is sourced from the TrueNAS logs. I have to assume you have read the log data so I guess the question is, what data is relevant to you as the person maintaining a set of servers that is int eh log data and what are you missing?<br /> <br /> As for the temperature data, the Multi-Report script also creates a CSV file of all the relevant data each time the script is run. You can also run the script just to collect statistical data (no email output) which is what I do, once an hour. This CSV can then be opened in a Spreadsheet application and the data can be examined, plotted, etc. Take a look at the first line of data, this contains the column titles. It works fine when doing some data analysis. But you could use this data to create a dashboard if you desire. I'm not sure if this helps you or not.<br /> <br /> Cheers!</div>

 

[NickF]

<div class="bbWrapper"><blockquote data-attributes="member: 101" data-quote="joeschmuck" data-source="post: 743427" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=743427" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-743427">joeschmuck said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Nick,<br /> One of the things I wanted to do but it's outside my wheelhouse was to create a dashboard or GUI that runs on TrueNAS and provides the data that the Multi-Report script provides and the adjustable alarm setpoints. I have not examined the TrueNAS logs so I'm not sure what else can be displayed. If I understand you correctly, the data provided above is sourced from the TrueNAS logs. I have to assume you have read the log data so I guess the question is, what data is relevant to you as the person maintaining a set of servers that is int eh log data and what are you missing?<br /> <br /> As for the temperature data, the Multi-Report script also creates a CSV file of all the relevant data each time the script is run. You can also run the script just to collect statistical data (no email output) which is what I do, once an hour. This CSV can then be opened in a Spreadsheet application and the data can be examined, plotted, etc. Take a look at the first line of data, this contains the column titles. It works fine when doing some data analysis. But you could use this data to create a dashboard if you desire. I'm not sure if this helps you or not.<br /> <br /> Cheers! </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Hi Joe,<br /> <br /> The logs, even by default, give us insight into things that you haven't been tracking historically.<br /> <br /> Here's an example, this drive, according to SMART is fine from multireport:<br /> <div class="bbImageWrapper js-lbImage" title="1676082596005.png" data-src="https://www.truenas.com/community/attachments/1676082596005-png.63458/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676082596005-png.63458/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676082596005.png" title="1676082596005.png" width="1265" height="357" loading="lazy" /> </div> <br /> <br /> <br /> But that same drive spit out 27 CAM errors over the past 30 days, and may be failing.<br /> <div class="bbImageWrapper js-lbImage" title="1676082661987.png" data-src="https://www.truenas.com/community/attachments/1676082661987-png.63459/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676082661987-png.63459/" data-url="" class="bbImage" data-zoom-target="1" style="width: 956px" alt="1676082661987.png" title="1676082661987.png" width="1920" height="783" loading="lazy" /> </div> <br /> <br /> <br /> Where as this other system (similar spec) shows everything is fine, and there have been no CAM errors.<br /> <div class="bbImageWrapper js-lbImage" title="1676082845006.png" data-src="https://www.truenas.com/community/attachments/1676082845006-png.63460/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676082845006-png.63460/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676082845006.png" title="1676082845006.png" width="1198" height="402" loading="lazy" /> </div> <br /> <br /> But I am getting several ctl_datamove aborted errors, which means that ISCSI is having a problem and their may be a networking issue or the backing zpool for that LUN is not able to keep up with the writes being committed.<br /> <br /> <div class="bbImageWrapper js-lbImage" title="1676082984803.png" data-src="https://www.truenas.com/community/attachments/1676082984803-png.63461/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676082984803-png.63461/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676082984803.png" title="1676082984803.png" width="1386" height="527" loading="lazy" /> </div> <br /> <br /> <br /> I'm sure there are a whole bunch more useful things we can do with the logs, I have a pretty big dataset. But I'm just not sure what else to look for, other than those few examples and whatever data SMART is sending to Syslog (which is not everything, and not for every drive, I'm not sure what triggers SMART to send something to syslog).<br /> <br /> Then I have this one, where you picked up 4 read errors in multireport, but all the drives show as healthy, I want to try and figure out in the logs how to catch this:<br /> <div class="bbImageWrapper js-lbImage" title="1676083303336.png" data-src="https://www.truenas.com/community/attachments/1676083303336-png.63462/" data-lb-sidebar-href="" data-lb-caption-extra-html="" data-single-image="1"> <img src="https://www.truenas.com/community/attachments/1676083303336-png.63462/" data-url="" class="bbImage" data-zoom-target="1" style="" alt="1676083303336.png" title="1676083303336.png" width="1186" height="520" loading="lazy" /> </div> <br /> <br /> Just looking for feedback for some common errors that folks have run into. I like having the flexibility to visualize this type of data, and also compare it in conjunction with other data sources I have.</div>

 

[joeschmuck]

<div class="bbWrapper"><blockquote data-attributes="member: 40967" data-quote="NickF" data-source="post: 743437" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=743437" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-743437">NickF said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> But that same drive spit out 27 CAM errors over the past 30 days, and may be failing. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>What is the error message just before the CAM errors? Read, Write, Abort, Reset, etc.<br /> <br /> A CCB error does not automatically mean the hard drive is bad from what little I understand about CCB messages. But to be clear, I do not know much at all about CAM Control Bus errors. I do not use ISCSI so my experience is just what I look into and when I played with ISCSI about 9 years ago for a few months. But if this is a factor which it could be, then maybe it should be included in the Multi-Report script, eventually, after I understand more.<br /> <br /> <blockquote data-attributes="member: 40967" data-quote="NickF" data-source="post: 743437" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=743437" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-743437">NickF said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Then I have this one, where you picked up 4 read errors in multireport, but all the drives show as healthy </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>That is a function of <code class="bbCodeInline">zpool status</code>, not anything fancy from me. I would expect TrueNAS to report an unhealthy pool.</div>