Hardware Guide - NVMe and Fusion Pool Power Loss Protection

[Michael Schefczyk]

Dear All,

The hardware guide does address the requirement for power loss protection for SLOG devices very clearly.

As prices for larger NVMe devices with five year warranties are dropping, NVMe pools would become more economical now compared to the recent past. In addition, TrueNAS 12 (core) will provide the capabilities for fusion pools which can include NVMe devices in parallel with HDDs. There, NVMe devices would store small files and metadata, if I understand correctly.

Am I correct in assuming that the need for power loss protection is the same for all NVMe pools and fusion pools as for SLOG devices? In that case, users who will not afford large NVMe devices with power loss protection should stick to HDDs and not pursue the gain in speed, correct? If this is correct, should it be made made more clear in the hardware guide?

Regards,

Michael Schefczyk

 

[Ericloewe]

For data and metadata vdevs, data loss protection is a nice to have, mostly. The real risk is if the disk is overwriting internal metadata. Crucial SSDs used to have minimal power loss protection sufficient only to safeguard metadata. I don't know what strategies other vendors use. It is conceivable that you could cause data loss/corruption on an SSD, affecting data other than what's being written. Overall, the risk is probably small.

For SLOG, having no power loss protection is very likely to lead to data loss of whatever was being written. Since SLOG is in use instead of sync=never, the presumption is that such a scenario is not acceptable, hence PLP is a de-facto requirement.

Keep in mind that ZFS atomically transitions from one valid state to another, so unless the backing storage actively corrupts data beyond ZFS' ability to recover, data not in flight is safe.

 

[Jamberry]

Sorry to revive this old post, but could you be a little bit more specific @Ericloewe on how Fusion Pools work?

I think I understand how SLOG works and because of that, I get why a SLOG does not need a UPS, but PLP. Otherwise there is a risk of loosing the entire pool.

https://www.truenas.com/docs/core/storage/pools/fusionpool/ only states

When using SSDs with an internal cache, add uninterruptible power supply (UPS) to the system to help minimize the risk from power loss.

Which leads me to believe that a Fusion Pool behave different from SLOG, because I only need to protect it with a UPS?
What happens to the Fusion Pool if the system has an unexpected shutdown?

As far as I understood PLP != PLP.
Cheap PLP only protects you from data being overwritten. The Crucial you mentioned and Kingston DC1000B offer this.
Expensive PLP offers real "in flight" PLP. This is what we wan't for SLOG.

Do fusion pools need:
- only a UPS?
- PLP?
- "in flight" PLP
to be 100% save?

 

[DigitalMinimalist]

May I ask an additional question.

Are there any further configuration options for fusion pools?
I created one in my testing phase with a RAIDZ2 HDD pool and a pair of mirrored (PLP) 1TB NVMEs and I would like to utilize the NVME speed as best as possible, e.g. small files beside metadata

 

[Jamberry]

Maybe this is a little bit above my pay grade but I will try anyway. Take it with big grain of salt:

> Are there any further configuration options for fusion pools?

No. Fusion pools store only the metadata. That way small files are faster written and read.
What you ask for is using L2ARC to speed up reads or SLOG to speed up sync writes.
Using them as a some kind of write cache is not possible. SLOG would not be great because missing PLP.
L2ARC could be an option. You could partition them and use some of it for L2ARC instead of fusion pool, BUT I don't know if it is even possible to shrink a fusion pool and you are probably better of by adding (way faster) RAM.

 

[DigitalMinimalist]

Thx

I was looking for a setting like „file size threshold to store on special vDev“
On the other hand the max NVME utilization is 75% - not sure if I will end up with 10% utilization or if TrueNAS will utilize the 75% efficiently

I wasn’t planning on L2ARC, as I will have 64GB RAM - no SLOG either as I do not have many sync writes (I think).

Special vDev sounded like the best option to speed up my single storage pool

 

[Jamberry]

Thx

I was looking for a setting like „file size threshold to store on special vDev“
On the other hand the max NVME utilization is 75% - not sure if I will end up with 10% utilization or if TrueNAS will utilize the 75% efficiently

I wasn’t planning on L2ARC, as I will have 64GB RAM - no SLOG either as I do not have many sync writes (I think).

Special vDev sounded like the best option to speed up my single storage pool

This is getting off topic, maybe an admin can move this? Anyway, I think the first thing is to think about is what exactly you wanna speed up. Read, write? SMB or iSCSI? Describe a scenario you wanna speed up. Otherwise nobody can give you any recommendations.

 

[DigitalMinimalist]

This is getting off topic, maybe an admin can move this? Anyway, I think the first thing is to think about is what exactly you wanna speed up. Read, write? SMB or iSCSI? Describe a scenario you wanna speed up. Otherwise nobody can give you any recommendations.

You are right. I will create a new thread and follow here

 

[Ericloewe]

I think there’s something of a misconception here. If you can afford to lose a few seconds of writes, you don’t need an SLOG, just use sync=none.
If you can’t afford that and the performance of the on-pool ZIL isn’t good enough, then you need a proper SLOG device, that’s fast enough and can deal with power loss.

All that is completely orthogonal to Special vdevs (metadata-mostly vdevs, also marketed as fusion pools). There’s nothing very different about these vdevs, they’re just set up to receive metadata and/or small data writes, via the ZFS allocation classes feature. This is not exclusive and ZFS can fall back on the rest of the pool, e.g. if the vdev is full. It also, obviously, has no impact on data written before the vdev is added.
As such, all the same usual criteria apply.

 

[Ericloewe]

I was looking for a setting like „file size threshold to store on special vDev“

That does not exist, as the pool layer of ZFS does not deal with files, but merely with blocks. I think there’s a configuration option for what block sizes to send to the special vdev, which does a lot of what you’re looking to do. It might be an OS-level option, though.

 

[spuky]

That does not exist, as the pool layer of ZFS does not deal with files, but merely with blocks. I think there’s a configuration option for what block sizes to send to the special vdev, which does a lot of what you’re looking to do. It might be an OS-level option, though.

I guess he was Looking for "Metadata (Special) Small Block Size" which decides what smal blocks only get written to the metadata special vdev. This is an option that must be set on the Dataset level...

Edit: must be smaler than your blocksize else every block would end up on the metadata special vdev..

 

[DigitalMinimalist]

I think there’s a configuration option for what block sizes to send to the special vdev, which does a lot of what you’re looking to do. It might be an OS-level option, though.

That was exactly my question - thanks :)

 

[Jamberry]

I think there’s something of a misconception here. If you can afford to lose a few seconds of writes, you don’t need an SLOG, just use sync=none.
If you can’t afford that and the performance of the on-pool ZIL isn’t good enough, then you need a proper SLOG device, that’s fast enough and can deal with power loss.

All that is completely orthogonal to Special vdevs (metadata-mostly vdevs, also marketed as fusion pools). There’s nothing very different about these vdevs, they’re just set up to receive metadata and/or small data writes, via the ZFS allocation classes feature. This is not exclusive and ZFS can fall back on the rest of the pool, e.g. if the vdev is full. It also, obviously, has no impact on data written before the vdev is added.
As such, all the same usual criteria apply.

Not sure if understand what you are saying.
Lets assume that I can't lose a few seconds of writes.

What level of protection (and why) do fusion pools need to be save?
- only a UPS?
- PLP?
- "in flight" PLP?

The manual makes it seem like only a UPS is sufficient enough. My guess is, that this does not prevent you from loosing metadata writes in case of a unexpected shutdown. I would argue that exactly like SLOG, you need to have "in flight" PLP to prevent this. But maybe that guess is based on a wrong understanding of how fusion pools work.

One misunderstanding of me could be, that fusion pools never hold sync writes to begin with and because of that there is no real risk by not having PLP on fusion pools.

 

[spuky]

Fusion Pools do not help with sync writes... it just helps with latency in accessing metadata and smal blocks if configured. Read performance.

For Sync writes you are still needing Slog with PLP / or live with poor performance (of course a sync write of a smal block to ssd could still be faster than to spinning rust)

 

[Jamberry]

Fusion Pools do not help with sync writes... it just helps with latency in accessing metadata and smal blocks if configured. Read performance

So if I don‘t have a SLOG, fusion pools does not speed up write performance of metadata?

 

[Ericloewe]

Again, SLOG and special vdevs solve very orthogonal problems. There are situations where you may need one, the other, or both.

 

[Jamberry]

Ok, let me rephrase it.
Do fusion pools speed up write writing lots of metadata?
Because @spuky response give me the idea that it only affects read performance, which I don't believe to be true.

 

[Ericloewe]

Do fusion pools speed up write writing lots of metadata?

To some extent, yes. The effect is probably going to be more visible on reads, typically.

 

[spuky]

(of course a sync write of a smal block to ssd could still be faster than to spinning rust)

@Jamberry you need to read the whole post ;-)

 

[Jamberry]

Thank you @Ericloewe and sorry to you @spuky, I was on mobile

The only thing I still do not get is, why a USP is sufficient enough and I don't need a "in flight" PLP protected fusion pool.
In my imagination, metadata gets to the SSD, the SSD says "yeah sure I got this" while the data is still only in cache, the system has an unexpected shutdown and puuf, the metadata in cache is lost an not written to the fusion pool.