middlewared 100%

[Liriel]

I have been having this problem for well over a year now. Periodically (I have not been able to discern a specific frequency, but at least a few times per month), I find middlewared consuming 100% CPU. Once this starts it will continue until I restart the service. I have not noticed the behaviour correlated to any other activity and I have not noticed any unusual entries in middlewared.log. However, because it happens infrequently I don't always notice it when it first starts.

So two related questions:
1. Any ideas on what is causing this?
2. What type of monitoring or reports would be useful for me to provide next time it happens, so that you can better answer question #1?

Hardware info:

• Motherboard make and model: Supermicro X10SAE-O
• CPU make and model: Intel Xeon E3-1220 v3
• RAM quantity: 32GB ECC
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
• Storage:
  • 6x5TB RAIDZ2 Toshiba X300 (HDWE150)
  • 6X8TB RAIDZ2 Shucked WD Elements (WD80EZAZ-11TDBA0)
  • Mirrored 2x3TB Toshiba DT01ACA300
  • Mirrored 2x32 Sandisk Ultra Fit (boot pool)
• Hard disk controllers: LSI 9240-8i
• Network cards: None

 

[Spearfoot]

I have been having this problem for well over a year now. Periodically (I have not been able to discern a specific frequency, but at least a few times per month), I find middlewared consuming 100% CPU. Once this starts it will continue until I restart the service. I have not noticed the behaviour correlated to any other activity and I have not noticed any unusual entries in middlewared.log. However, because it happens infrequently I don't always notice it when it first starts.

So two related questions:
1. Any ideas on what is causing this?
2. What type of monitoring or reports would be useful for me to provide next time it happens, so that you can better answer question #1?

Hardware info:

• Motherboard make and model: Supermicro X10SAE-O
• CPU make and model: Intel Xeon E3-1220 v3
• RAM quantity: 32GB ECC
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
• Storage:
  • 6x5TB RAIDZ2 Toshiba X300 (HDWE150)
  • 6X8TB RAIDZ2 Shucked WD Elements (WD80EZAZ-11TDBA0)
  • Mirrored 2x3TB Toshiba DT01ACA300
  • Mirrored 2x32 Sandisk Ultra Fit (boot pool)
• Hard disk controllers: LSI 9240-8i
• Network cards: None

Which version of FreeNAS are you running?

 

[Liriel]

Which version of FreeNAS are you running?

Oh god, not sure how I forgot to mention that! I am running 11.3-U4
This has been happening at least since 11.3 first came out...can't remember if it was present too in 11.2.

 

[Spearfoot]

Oh god, not sure how I forgot to mention that! I am running 11.3-U4

We all flub the dub sometimes... and you certainly gave all of the other relevant information!

This has been happening at least since 11.3 first came out...can't remember if it was present too in 11.2.

A quick forum search doesn't yield much... Hmmm...

How long have you been booting from those particular Sandisk Ultra Fit USB flash drives? And do you store the system dataset on the boot pool? Flash drives tend to just wear out in the best of cases, and they wear out even more quickly if you store the system dataset on them.

So... maybe you just need to replace them and re-install FreeNAS -- after saving off your configuration file, of course!

Another slightly more expensive boot alternative would be to use a SATA DOM drive -- your motherboard supports these.

And of course a small, inexpensive SSD is always a good choice for the boot device.

That's all I can think of... perhaps another forum member will drop by and help out.

Good luck!

 

[Liriel]

Thanks @Spearfoot I have been running this USB drive for about 2 years and the system dataset is not on the boot pool

I could definitely explore getting a new boot drive configuration. First step will be actually reading what the latest recommendations are. Last I checked that it WAS mirrored usb drives, although admittedly that was a long time ago lol

Scrubs come out clean so far, so middlewared spiking would be a weird symptom. Nonetheless, probably a good excuse to replace those usb drives.