Alert During Scrub, pretty please

[frosty]

Wow, that is a 5 month old thread you pulled on. But if someone wants this feature, add it to the Bug Report Features and if enough people want it, it will eventually get included, maybe.

But it would be nice to just add a feature like that, maybe create a listing of items for which a popup would be nice to have and make those as an option which the user can select. Call it "System Busy Popups" and it can cover a variety of things instead of just a scrub. I wouldn't imagine the list would be very long.

When searching, I found this telling someone else (who asked the same question) to post it in "Bug Report Features".. but i couldn't find that. So hopefully this is the right place lol

Please please add an alert for when Scrubs are taking place <3
--------------------
Every few months (just long enough for me to forget) I'll find myself dealing with shitty performance. All of a sudden 1080p videos start to chug from plex. VM spin ups slow down, network transfers barely break 30MB/s, and a usage spike appears on the freenas box under the name of "python3.6".
(Or at least i think that's what it shows up as, I forgot to actually add it to my notes.. again.. because "i'll remember this time, no way i forget this hell again" lol)

Since this is just a home lab, I ignore it, at first, assuming it's just an unlucky group of machines that decided to start updating or something. After it doesn't go away I spend the next several hours/days rebooting VMs/hypervisors, digging through logs in VMs/hypervisors, then swapping out RAM and re-seating CPU's to flush out any gremlins.. but nothing helps, so i scream out "WHY!?" to the server gods.. they never answer

I eventually misclick on the top dataset and just.. facepalm... I remember that these

aren't the only buttons. And that these

exist. I click status knowing it's going to tell me "scrub in progress", and yes, that's what it was
--------------------
I love freenas btw, you guys are amazing! I suggest it to everyone I know

 

[Ericloewe]

I don't know which version you're running, but there already is an alert for this in 11.1 and later. I get an email every time a scrub starts and ends.

 

[frosty]

Sorry, I'm on 11.1-U6, i meant in the actual little alert box though (included the system info in the pic just so you'd know for sure which build i was on)
I don't have email alerts on, but honestly I probably wouldn't even think to check my email for something like that. I ignore email unless i specifically waiting for something.

 

[Jailer]

I check my email a heck of a lot more often than I check the FreeNAS UI.

You're asking for a feature that already exists (scrub alerts). This is not a FreeNAS issue, it sounds to me like you have a poor scrub schedule set up. Change the time that the scrub takes place and you won't have this issue.

 

[Ericloewe]

Alerts are more configurable in 11.2.

That said, not having email alerts setup is a very bad idea, as they are the primary way of warning you of urgent issues.

 

[frosty]

I check my email a heck of a lot more often than I check the FreeNAS UI.

You're asking for a feature that already exists (scrub alerts). This is not a FreeNAS issue, it sounds to me like you have a poor scrub schedule set up. Change the time that the scrub takes place and you won't have this issue.

Good for you? I check my freenas UI more than my email. But even if i didn't, the chances of missing the email are MUCH higher than the chances of missing a glowing orange icon on every screen 0.o

By "already exist" you mean the email alert right? Because that isn't what i'm asking for at all..

The schedule is as good as its going to get, hence why it only really pops up every few months. But with the size of my pool it pretty much always takes ~a day to complete anyway. And seeing as i use my stuff virtually every day, should i schedule it for the next timewarpday lol?

Alerts are more configurable in 11.2.

That said, not having email alerts setup is a very bad idea, as they are the primary way of warning you of urgent issues.

By "more configurable" does that mean it's possible to have it do that?

I hate email, its 95% crap I don't care about lol. Which means in order to get that 0.01% chance that the email is important, i have to sift through mountains of shit. It's just not worth it. Especially since i'm in the ui all the time messing with stuff anyway.

 

[Ericloewe]

The manual should tell you what your options are in 11.2.

 

[Chris Moore]

Sounds like you need to get a Gmail account that is just for the monitoring of devices like FreeNAS that send important messages out on email in an attempt to get your attention.
The alert is for errors. The scrub is not an error. There is no reason why a scrub should show up in the error alerts.
What is the configuration of the hardware? I do a scrub and it takes about eight hours. You should be able to schedule it for a time when you are not using the NAS.

 

[danb35]

its 95% crap I don't care about

Sounds like you need better spam filtering. Gmail's works pretty well if you don't want to mess with running your own mail server.

in order to get that 0.01% chance that the email is important

Sounds like you've got 4.99% unaccounted for.

i'm in the ui all the time messing with stuff anyway

Why? Once you get the server set up, there really shouldn't be any need to be in the UI with any frequency.

In any event, many mobile carriers also support email-to-SMS, so FreeNAS's email notifications can come by text message.

 

[frosty]

The manual should tell you what your options are in 11.2.

Fair enough, rtfm it is lol
-----------------------------------------------------------------------

Sounds like you need to get a Gmail account that is just for the monitoring of devices like FreeNAS that send important messages out on email in an attempt to get your attention.
The alert is for errors. The scrub is not an error. There is no reason why a scrub should show up in the error alerts.
What is the configuration of the hardware? I do a scrub and it takes about eight hours. You should be able to schedule it for a time when you are not using the NAS.

I have thought about getting a separate gmail just for server messages before, and i may end up doing that. But I still think it would be a good idea for it to be prevalent in the UI somewhere when you log in. Because once you start hunting for what is causing an issue, i bet most people open up a terminal or a ui before they open up their email and start scrolling back.

Maybe i have a different definition of "errors" but i get alerts (in the UI) for all sorts of things that aren't errors (in my mind). Like updates and capacity set breaches. To me alerts are just very important things you need to be aware of. As for "no reason why a scrub should show up in alerts", there is a significant impact on performance.. that feels like a pretty good reason.

I use the box for vm's as well as storage, and i tinker with stuff basically all day every day. (unemployed loser, so almost literally all day every day) So there really isn't a better time for me to have it scrub
Hardware Config
CPU: 2x Xeon E5-2670
RAM: 24g (had more but it died)
Hard Drives: 12x WD-Red (6x3tb z2 [x2])
MoBo: Supermicro X9DRI-F-O
HBA: 2x SAS9211-8I
Case: Norco RPC-4224

-----------------------------------------------------------------------

Sounds like you need better spam filtering. Gmail's works pretty well if you don't want to mess with running your own mail server.

Sounds like you've got 4.99% unaccounted for.

Why? Once you get the server set up, there really shouldn't be any need to be in the UI with any frequency.

In any event, many mobile carriers also support email-to-SMS, so FreeNAS's email notifications can come by text message.

Yes, gmail's spam filter works pretty damn well. But it includes all the random useless notices sent out from services i use. Like "ooo look a steam sale!" because those are rare >.> Lol

Every month i would say about 5% of emails are actually important, freenas would be sending me like.. one or two a month at most (hopefully lol). So the 0.01% was freenas's hold of the 5%. I thought that was clear from context, but i can see how you could read it the other way since i didn't include "and from freenas" in the "that the email is important" part.

As for why i'm in the ui a lot.. I like tinkering and looking at the reports and compression ratios (breaking 2x is very satisfying). I'll set up a bunch of stuff one way, start using it, realize it's stupid and tear it all down and start again. Especially with naming schemes and dataset structures. Or i'll decide to implement security rules, add users, etc. Lots of stuff really

Now the SMS thing.. that could be very helpful! And much more useful to me than some of the other responses which are basically just "I don't need X, because I do Y. You should just do Y instead, even though Y isn't a solution to your problem!" which is always a fun response lol

 

[danb35]

You should just do Y instead, even though Y isn't a solution to your problem!

"You should adjust your problem to fit my solution" is a common answer in IT. On the SMS thing, again, check with your carrier. I'm with AT&T, and it's <phonenumber>@txt.att.net.

 

[frosty]

"You should adjust your problem to fit my solution" is a common answer in IT. On the SMS thing, again, check with your carrier. I'm with AT&T, and it's <phonenumber>@txt.att.net.

No i know, it's just always a little funny to me lol

and thanks again

 

[Chris Moore]

Hard Drives: 12x WD-Red (6x3tb z2 [x2])
MoBo: Supermicro X9DRI-F-O
HBA: 2x SAS9211-8I

I have a similar setup with twelve 4TB drives in two RAIDz2 vdevs, my scrub finishes in about five hours. I have it set to run at 2 in the morning, and if I am still up when it kicks off, I know it is time for bed, but it should not be causing your pool a significant slowdown.
Unless there is a lot more activity on your pool with other work. It makes me think there might be some other problem with the system.

I use the box for vm's as well as storage

What kind and quantity of VMs are you doing? Do they have a lot of disk activity?

Do you have monitoring scripts setup to keep track of your disk health?

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[frosty]

I have a similar setup with twelve 4TB drives in two RAIDz2 vdevs, my scrub finishes in about five hours. I have it set to run at 2 in the morning, and if I am still up when it kicks off, I know it is time for bed, but it should not be causing your pool a significant slowdown.
Unless there is a lot more activity on your pool with other work. It makes me think there might be some other problem with the system.

What kind and quantity of VMs are you doing? Do they have a lot of disk activity?

Do you have monitoring scripts setup to keep track of your disk health?

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

my pool is hovering around 80% full, which probably doesn't help lol. But i definitely don't have the cash to upgrade the current disks. And several of the crappy norco backplanes decided to shatter when i was working the molex cables in. So adding drives, instead of replacing, isn't really a possibility either since the 3 surviving backplanes are filled.

At any given time i probably have ~12 vms running. None with substantial disk io requirements, as far as i know. At least half of them are just sitting there being targets, clients just waving at a server, or hanging out until i need to test something version/os specific.

The most intensive one is probably the plex server, serving 1 client at most.

I have SMART scans scheduled, but that's it. I'll checkout that github

 

[Chris Moore]

my pool is hovering around 80% full, which probably doesn't help

Yes. The more data in the pool, the longer it will take to scrub. I manage one at work that contains around 180TB of data and takes about five days to complete a scrub.
I understand if money is tight, but you might want to take a look at these and see if you think they could be used to fix your system up:
https://www.ebay.com/itm/Norco-RPC-4020-Backplane-old-style-sata/264112023788

Welcome to the forum.

 

[Apollo]

@frosty, Could be memory starved.
Is your swap being used at all?

 

[Chris Moore]

True, with the VMs you certainly need more memory.

 

[frosty]

Yes. The more data in the pool, the longer it will take to scrub. I manage one at work that contains around 180TB of data and takes about five days to complete a scrub.
I understand if money is tight, but you might want to take a look at these and see if you think they could be used to fix your system up:
https://www.ebay.com/itm/Norco-RPC-4020-Backplane-old-style-sata/264112023788

Welcome to the forum.

omg.. Thank you!!

When they broke i reached out to norco about getting replacements and they said they didn't sell them, so i just kinda "oh well"d it. I figured i would eventually pull everything apart so i could take them out and try to frankenstein on another molex connector. I didn't even think to check ebay! ><

@frosty, Could be memory starved.
Is your swap being used at all?

Freeee as a bird

True, with the VMs you certainly need more memory.

I had 36g before the great RAM purge of 2018.. I had 4 sticks die >< (including my spare stick)

Most the VMs are using <512mb, the only one using more than a gig is plex; who gets 2g
(i tried giving it 8g a while back thinking it would cache the entire video it was playing, but it didn't seem to even try to use it and it didn't impact scrubbing --of video-- performance)

But i do plan on adding more memory asap
----------------------------------------------------------
And thanks all, for the warm welcome

 

[Apollo]

Netdata provides realtime metrics and with it, you can monitor HDD performance, mainly Busy state which should tell you if the HDD is the cause of the slow scrub. When value exceed 80-90%, your drive is most likely you issue, only if the throughput is below it's nominal transfer rate.
If you can monitor the evolution of the scrub from the time it was started, you will see the throughput will be mostly flat (maxed out) at the beginning and decays when reaching the inner side of the platters toward the end of the scrub when your drive are quite full.