Freenas 9.3-CURRENT keeps crashing, not sure what's happening.

[ryanallan]

It seems my freenas box likes to stop running these days.
Here's a screenshot of the terminal.
The evening of May 11, it was doing something, I'm not sure what.
Then at midnight it seemed to shutdown.
Tonight, May 12, at 22:29:24 I discovered it wasn't running and had to power cycle.
Can anyone shed some light on what happened?

 

[hugovsky]

bad PSU maybe? have you tried to test your hardware some way?

 

[Bidule0hm]

Please list your hardware so we can help.

 

[ryanallan]

Updated my signature with current hardware.

 

[ryanallan]

bad PSU maybe? have you tried to test your hardware some way?

I'm not sure if it's hardware.
Looking at the logs, freenas appears to be attempting to update but cant find the web address of the update and fails. It tries this 4 times and then stops.
Last night it seemed to be successful in the update but when finished, it didn't appear to boot. It's like it updated and then shutdown.
I am not familiar enough with the verbiage used in the terminal to know for sure, but this is what I think is happening.

 

[titan_rw]

Keep in mind not everyone sees signatures. Mobile users generally don't. And some web users turn off signatures to unclutter the forums.

 

[ryanallan]

Keep in mind not everyone sees signatures. Mobile users generally don't. And some web users turn off signatures to unclutter the forums.

Ah, OK. Did not know that.

FreeNAS Build 9.3 CURRENT | ASRock c2550d4i | Intel Avoton C2550
16GB ECC | picoPSU-160-XT | 4 x 4TB WD RED | ZFS-2 | :)

 

[Bidule0hm]

PicoPSU... well, I think we found the problem... @jgreco will love this thread :P

Around midnight there's some scripts that are executed by the OS. Of course they use more ressources than a idling system. More ressources = more power needed. More power on this ridiculous PSU = brownout/dropout.

I wonder how you can even boot with this PSU I'd love to see an oscilloscope capture of the 12 V rail at power-up...

Edit: my god, $50 for a 160 W crappy PSU...

Seriously, get a SeaSonic G-360 for $60 ;)

 

[ryanallan]

It's been running fine for about 12 months now, strange that suddenly it wouldn't be powerful enough.
Regardless, I suppose picoPSU won't even be taken seriously.

During boot, the system consumes around 80W. After settling down, the system draws a steady 43W. Documentation says the HDD’s consume about 15W each during boot, and 5W each once running. This leaves the CPU /motherboard/RAM at around 20W.

 

[Ericloewe]

It's not really about nominal power. It's about quality.

I have serious doubts regarding all those picoPSUs I see advertised.

 

[Bidule0hm]

You're probably at the edge, caps are older by one year so their ESR is higher, that's probably why it worked (barely) before and not anymore.

It's not that I don't take the PicoPSU not seriously, but it's just not made for this usage and from all I've seen it's not very high quality.

Try to launch a scrub and why not stress a bit more the CPU at the same time to see what happens <-- WARNING: if the PSU is the problem this can potentially damage some of the hardware and/or corrupt the data, however as you trust the PSU you shouldn't have any problem.

 

[mjws00]

Heh. I'll take the under on the PSU bet. It is ~50% load at boot, peaking at 80w. They are talking 200w peak 160w continuous. ;) Only 4 drives and a 2550, meh. We even have a nice little wattmeter to view. What odds are you guys giving?

Isolate software vs. hardware. Stick vanilla BSD on there and see how it runs. Or a linux livecd. Even 9.2.1.9 might give you clues. I'd also be running a memtest, and definitely taxing cpu and drives for brown-out. Check temps and airflow as well. Basically you are back to burn-in stage. An update or script crash seems plausible as well.

Edit: I also see a lego case and temp issues encountered previously... might be a clue there. Also hope you are not on the Marvell ports.

 

[Bidule0hm]

This is measured on the mains side and the caps inside the PSU greatly smooth the current peak, and with a (dubious) wattmeter who adds some integration on top of that.

I'm pretty sure if you look at the 12 V rail during start-up you'll see that it goes out of specs. And probably the same if you access all the drives simultaneously, you'll have dips in the voltage and those dips can crash the system. If you're on the edge the dips are just a bit too small to cause crashs but once in a while (or more likely, when you load the system a bit more) a dip is big enough to crash it.

Of course it's not 100 % sure it's the problem but this PSU is not reliable enough to be used on a server anyway. And on the long term bad quality power will cause wear on the others components (especially the drives). Also, I bet this PSU doesn't meet the ATX spec and will dropout if used with an offline UPS during the commutation.

Given the fact you have the G-360 for an extra $10 I don't know why you'd choose the PicoPSU to begin with :p

 

[ryanallan]

This is measured on the mains side and the caps inside the PSU greatly smooth the current peak, and with a (dubious) wattmeter who adds some integration on top of that.

I'm pretty sure if you look at the 12 V rail during start-up you'll see that it goes out of specs. And probably the same if you access all the drives simultaneously, you'll have dips in the voltage and those dips can crash the system. If you're on the edge the dips are just a bit too small to cause crashs but once in a while (or more likely, when you load the system a bit more) a dip is big enough to crash it.

Of course it's not 100 % sure it's the problem but this PSU is not reliable enough to be used on a server anyway. And on the long term bad quality power will cause wear on the others components (especially the drives). Also, I bet this PSU doesn't meet the ATX spec and will dropout if used with an offline UPS during the commutation.

Given the fact you have the G-360 for an extra $10 I don't know why you'd chose the PicoPSU to begin with :p

I appreciate your comments. I have a spare 450W, so I'll give that a try.
The picoPSU was chosen as I needed something external for my build.
You have me scared of this thing now. I'll be speeding home, running stop lights, just to make sure it hasn't started anything on fire during the day.

Heh. I'll take the under on the PSU bet. It is ~50% load at boot, peaking at 80w. They are talking 200w peak 160w continuous. ;) Only 4 drives and a 2550, meh. We even have a nice little wattmeter to view. What odds are you guys giving?

Isolate software vs. hardware. Stick vanilla BSD on there and see how it runs. Or a linux livecd. Even 9.2.1.9 might give you clues. I'd also be running a memtest, and definitely taxing cpu and drives for brown-out. Check temps and airflow as well. Basically you are back to burn-in stage. An update or script crash seems plausible as well.

Edit: I also see a lego case and temp issues encountered previously... might be a clue there. Also hope you are not on the Marvell ports.

haha, ya, lego!
Heat issues have been addressed, I think.
SMRTCTL says the drives idle around 30C. I haven't checked during a scrub.
I am using the 4 Intel sata ports, yes.

 

[Bidule0hm]

Good idea, that way you can be 100 % sure to know if it's the PSU or not, and work from here ;)

It won't catch fire behind your back (but, maybe, if it's overloaded... who knows :p) but I won't use it for a server though. A good use case would be for a car PC (I believe it's why this PSU was invented first) or a small HTPC for example :)

 

[mjws00]

I wouldn't buy one either. But there are a lot of assumptions being made about this PSU, and NO first hand experience. These DC-DC PSU's are pretty common and bulletproof amongst the micro and embedded crowds. They are often run at much higher loads than seen here, imho. This is a BRICK specified to maintain necessary watts with DC-DC convertors that can hold pretty tight voltages. ATX spec is 5% and this thing should be able to hold that at 50% load in its sleep. It is certainly possible that it can't... but unlikely.

On the measuring side. Yep the gear is simple and on the mains side, but it will have accuracy far greater than needed here.

Truthfully, I defer to ya on the electronics side all day @Bidule0hm. But I have seen a ton of DC-DC gear and built enough power supplies to know that 160w is a walk in the park to regulate @ 12V.

;) Plus someone has to offer something besides. ewwww picoPSU.

 

[Bidule0hm]

Yep, usually it's not a big deal but here the problem is that given the size of the PSU they can't put enough filtering to smooth correctly the output (and SMPS are very noisy...), this is still in specs (probably at least, I don't know what's the ATX ripple max value) but this noise is not a good thing. I've also no doubt the regulation isn't too bad but is probably out of specs at start-up, and even if it's in specs, a SeaSonic PSU has a very tight regulation (the X series is 1 % IIRC, don't know about the G series but it should be very good too) which means less stress on the others components. The problem with this PSU isn't that there's a big design problem (AFAIK) but that it's rather a stack of small details that can lead to crashs or at least to more wear.

I would be than happy to do some tests on one of those PSUs but I just can't spend $50 just to do testing; maybe if I ask for a sample... :D :p

For the wattmeter the problem isn't the accuracy but it's that you don't see the real peak value because of cap smoothing and meter integration.

 

[ryanallan]

Bidule0hm, are you actually in France?

Last night I installed my 450W PS. I'll see how it goes over the next couple of days.

I also noticed that the STABLE branch of 9.3 is about 1 month newer than the CURRENT branch. Tonight, I'll probably switch branches, perhaps my issue was a bug...

 

[Bidule0hm]

Yes, why?

Just need to wait now :P

 

[ryanallan]

If you were closer I'd loan you my pico ;)