Pools offline after shutdown / power on

[icemule1]

I've been using TrueNAS for a few months now and it seems like when I shut it down and power it back on, it's like flipping a coin whether or not the array will come back online without me having to manually import them with CLI commands (don't seem to be able to do it through the GUI). This has happened several times now and I've posted about it previously, but nobody seems to be able to provide a reason on why this keeps happening. Does anyone have any insight? I wanted to stick it out with TrueNAS but I'm getting tired of wondering if my array will be ok every time I power the server on.

Here are some CLI commands I ran for troubleshooting purposes while the array was still down:

Code:

root@truenas[~]# zpool import
   pool: Scratch
     id: 4402457943842566774
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        Scratch                                 ONLINE
          c2eab11b-8f47-400b-a945-242e0641fc3c  ONLINE

   pool: WD Ultrastar 12x6
     id: 16881426510993768706
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        WD Ultrastar 12x6                         ONLINE
          raidz2-0                                ONLINE
            fc918cdf-6c9e-4cfb-89eb-f48a2b46d8ef  ONLINE
            02543dd6-0f35-4efd-899b-3502642e512d  ONLINE
            2dabdf9d-1802-4a6b-abc5-785e0d37ba09  ONLINE
            86c62170-f85e-4978-933b-b3738c0746ad  ONLINE
            d1fefe70-75b3-4fdf-b06f-0bf78364a320  ONLINE
            384fb5d2-1392-4eeb-b965-3eea9ef9c87d  ONLINE

Code:

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:20 with 0 errors on Tue Apr 11 03:45:24 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdc3      ONLINE       0     0     0

errors: No known data errors

 

[Samuel Tai]

How are your array disks connected?

 

[icemule1]

How are your array disks connected?

My 6-disk array is connected via SATA to the motherboard.
My 1-disk array is connected via SATA to a PCI-E SATA card.

When this issue happens, both arrays go offline.

 

[Patrick M. Hausen]

Power supply a bit marginal, possibly?

 

[icemule1]

Power supply a bit marginal, possibly?

It's a 450W Corsair that I only used for a year before putting it in this TrueNAS server. I connected a Kill-A-Watt to the server and it doesn't come anywhere close to 450W.

 

[Samuel Tai]

It's most likely a combination of your power supply and the SATA expansion card.

• Consumer power supplies tend to allocate the bulk of their amperage to the 12V rail, and starve their 5V and 3.3V rails, which SATA drives need. There's a reason server power supplies exist.
• As for the SATA expansion card, see https://www.truenas.com/community/r...t-multipliers-and-cheap-sata-controllers.177/

 

[icemule1]

It's most likely a combination of your power supply and the SATA expansion card.

• Consumer power supplies tend to allocate the bulk of their amperage to the 12V rail, and starve their 5V and 3.3V rails, which SATA drives need. There's a reason server power supplies exist.
• As for the SATA expansion card, see https://www.truenas.com/community/r...t-multipliers-and-cheap-sata-controllers.177/

I checked the specs of the power supply and it has 110W available for the 5V and 3.3V rails, that is sufficient, right?

The SATA card is only being used for the single-drive array which I'm not too concerned about (no important data on it). If the SATA card were the problem, why would it cause the 6 drive array, which has all of its drives connected directly to the motherboard, to also go offline?

 

[Samuel Tai]

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

 

[Samuel Tai]

You're also not thinking of your system as a system. The motherboard and PCI bus power distribution are also factors.

 

[artlessknave]

dear lord. 450W? nooooo!
I wouldn't run what you listed on anything less than 500W or 600W. I usually just go with 750W so i have lots of breathing room.
there is generally no such thing as a PSU that is too big; the worst you might get is marginally less efficiency.
there very much is such thing as a PSU that is too small, and that is how I would rate yours.
the first thing I thought of while reading the first post was "PSU is bad/too small"

additionally, you would be best served by replacing that ewaste SATA card with a solid SAS card.

 

[icemule1]

dear lord. 450W? nooooo!
I wouldn't run what you listed on anything less than 500W or 600W. I usually just go with 750W so i have lots of breathing room.
there is generally no such thing as a PSU that is too big; the worst you might get is marginally less efficiency.
there very much is such thing as a PSU that is too small, and that is how I would rate yours.
the first thing I thought of while reading the first post was "PSU is bad/too small"

additionally, you would be best served by replacing that ewaste SATA card with a solid SAS card.

I don't really need the SATA card, I just installed it for a test pool I was working with. Will remove, problem solved.

As for the power supply, you mentioned 500W or 600W minimum, but per Samuel Tai's post earlier, more important than overall maximum wattage is the limit on the 3.3V and 5V rails. Apparently the 110W that my power supply provides doesn't cut it, so what's the minimum I'll need there, assuming 6 hard drives? The entire server doesn't pull more than 150W from the wall at any given time.

 

[artlessknave]

the PSUs likely to have more Wattage are the PSU's with higher a wattage rating....

as such, you need a higher watt PSU. they generally increase all the watts. most of the specs for desktop power supplies include GPU rails.
in the desktop space, the PSUs are pretty much all going to be wasting power potential for the 12V rail. a server PSU i would expect to be more balanced, but they are generally part of a server chassis.

a corsair CX 750 that I found the spec pic for says 130W for 3.3/5V rails.

a few extra points:
remember that on power on, the system will usually use more power than it does idle. 150W idle.
PSUs tend to put out less power as they get older. if you were runing it at the top of its spec, as it ages that will not longer be enough juice.
typically, you want to aim for 10-30% overpower, so you have some room for this loss, and expansion, as well as small changes in system parts (like, nobody wants to replace a drive to find out it uses 2W more and your PSU dies)

also, it depends on *which* corsair. the cheaper corsair power supplies (CX/CS iirc) should only be used in desktops, not something you want of server reliability.


as I don't wanna deal with not enough power, i simply always get a PSU i know will have enough power.

 

[Davvo]

Did you touch the drives' power propieties or some eco mode/c states settings?

Regarding the PSU it would be nice to know the model of the drives so that we can give you more than vague guidelines.
The steps to find the numbers are in the resource previously linked though: it's useful, please read it.

 

[jgreco]

It's a 450W Corsair that I only used for a year before putting it in this TrueNAS server. I connected a Kill-A-Watt to the server and it doesn't come anywhere close to 450W.

That's not anywhere close to how you determine what size PSU is required.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can find. DO NOT DO THIS. Your NAS lives or dies by...

www.truenas.com

 

[samarium]

Indicative power requirements: https://outervision.com/power-supply-calculator

Although a recent run thru for me suggested:

Load Wattage: 569 W
Recommended UPS rating: 1100 VA
Recommended PSU Wattage: 619 W
+3.3V +5V +12V
19.6 A 38.4 A 30.1 A
3.3+5 = 257 W 12 = 362 W

Given 620W I was considering 750W to 850W. A quick look around did not find anything around 3.3V/5V rails in the consumer space, even a Seasonic, most PSUs I checked limit 3.3V and 5V rails to 20A. Given IPMI VGA, I don't neeed the 12V headroom, but it seems I need more 3.3/5V.

 

[artlessknave]

I don't see anything in your setup that would require all that much 3.3/5v. the spinners will use 5v and 12v
I would think a 750w should be plenty. an 850 would give you some nice extra.
depends a little on what is available and prices. if you see an 850w on sale for close to a 750w? just get that.

I have test systems with ancient, horrible power efficiency (dual e5640) that i have run with 8 drives on 650-750W without any obvious problems.

the problem is just that 450W is way to low. even 600 is far more reasonable.

 

[Samuel Tai]

Actually, the SATA power connector uses all 3 rails. Most spinners will work with just the 5V and 12V rails, but I don't know if this is universal.

What You Need to Know About SATA 15-Pin Power Connector Pinouts

This is the complete pinout for a SATA 15-pin power connector. It's one of two current standard power connectors designed for peripheral devices.

www.lifewire.com

 

[artlessknave]

Actually, the SATA power connector uses all 3 rails.

to my knowledge, while the drives have the 3.3 available, they mostly use the 5 and 12.
not really my area of expertise, though. part of why i just overprovision so i never have to deal with it.

 

[jgreco]

Indicative power requirements: https://outervision.com/power-supply-calculator

Please don't use these insipid calculators. They are almost always not designed to accommodate the special needs of a device with a larger than usual number of hard drives.

to my knowledge, while the drives have the 3.3 available, they mostly use the 5 and 12.

This is correct. There was a time when the SATA spec envisioned that the 3.3 might be useful, perhaps for laptops, but this never panned out in practice.

 

[jgreco]

To reiterate, please use proper techniques to size your PSU:

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can find. DO NOT DO THIS. Your NAS lives or dies by...

www.truenas.com