New checksum default? (When did this happen?)

[winnielinnie]

Currently running TrueNAS Core 12.0-U6.

I just noticed that there is a Checksum option when creating a new dataset, and it defaults to SHA512.

All my datasets (created prior to 12.0-U6, and likely on 12.0-RELEASE) are using On for their Checksum option. (Confirmed with GUI and with console.) I'm assuming this is Fletcher4.

What does this mean? Is SHA512 faster than Fletcher4? More reliable? Is it overkill to be used on standards ZFS datasets, when the previous default (Fletcher4) is "good enough"? Why has SHA512 been made the new default? (Is this an iXsystems' decision or upstream OpenZFS?)

It appears that if you use SHA512, you might introduce compatibility issues when attempting to read the data on an older system / ZFS version:

requires pool feature org.illumos:sha512

salted sha512 currently not supported for any filesystem on the boot pools

Checksums and Their Use in ZFS — OpenZFS documentation

openzfs.github.io

In the meantime, I'm going to continue to override the default SHA512 with On, until there's a clear explanation for this new default.

 

[winnielinnie]

Okay, so it did in fact happen with the release / update of 12.0-U6.

From the release notes:

• OpenZFS dedup performance improvements by defaulting to SHA-512 algorithm

Emphasis mine.

Does this imply that SHA512 is only faster when using deduplication, or that it's always faster even when no deduplication is being used?

If it's only the former, then why is it being pushed as a default for all datasets? I'd argue that the majority of users are not using deduplication. Why not just leave the default to On for non-dedup'ed datasets, and default to SHA512 when ZFS Deduplication is selected?

 

[Samuel Tai]

@winnielinnie, check out the linked Jira ticket from the release notes, https://jira.ixsystems.com/browse/NAS-110523.

Currently, when dedup is enabled on a pool, the crypto safe checksum algorithm selected by default is SHA254.

Performance testing has found that SHA512 performs better on our 64 bit platforms.

Please make our TrueNAS default for this value SHA512

CUrrent recommendation for dedup customers will be to manually change the algorithm on the pool at the time dedup is enable on the pool or any dataset/volume within the pool:

zfs set checksum=sha512 <pool name>

If you're not using dedup, then this setting has no effect.

 

[winnielinnie]

@winnielinnie, check out the linked Jira ticket from the release notes, https://jira.ixsystems.com/browse/NAS-110523.

I'm not using, nor have I ever used ZFS Deduplication on any dataset.

The explanation is not clear:

"Performance testing has found that SHA512 performs better on our 64 bit platforms."

Overall performance on 64bit platforms? Or performance of datasets with deduplication? (A key word mentioned just prior to this claim.)

Are they comparing performance of SHA512 vs SHA256... or vs Fletcher4? Is it concerning dedup or non-dedup or both? (The only information I could find is that Fletcher4 out-performs SHA512, however Fletcher4 is not sufficient if Dedup is enabled.)

If you're not using dedup, then this setting has no effect.

If you see my screenshot, this is the default selection for Checksum, and I'm not using dedup.

 

[winnielinnie]

I think they made a mistake when implementing this feature. You get this dialog popup when enabling Dedup. (Just to reiterate: You only get this popup if you intentionally enable Dedup.)

I think the developers meant to only change the default selected Checksum if the user enables Dedup.

I filed a bug report: https://jira.ixsystems.com/browse/NAS-113018

 

[winnielinnie]

I updated the bug report with an additional required change: The wording of the dialog popup warning is incorrect and ambiguous


---


▼ ▼ ▼ ...Snippet from the bug report... ▼ ▼ ▼

On top of that, the dialog popup is incorrect, and must be re-worded to reflect a more accurate warning.

It reads:

Code:

"Default Checksum Warning"
"The default 'Checksum' value for ZFS pools used to be SHA256. Our testing has shown that SHA512 performs better.
So, we have changed the default value from SHA256 to SHA512. You can change it back in 'Advanced Options'."

This is incorrect / ambiguous / misleading for multiple reasons:

#1) Checksum is applied on a per-dataset level, not the entire pool.

#2) The default checksum for Dedup datasets was SHA256. The default for non-Dedup datasets always was (and still should be) On (i.e, Fletcher4). The popup is ambiguous and does not clarify this very important detail (of Dedup vs non-Dedup).

#3) The popup only appears when the user enables ZFS Deduplication, yet it makes no mention of Dedup vs non-Dedup in the warning. (As I explain in point #2 above.)

 

[morganL]

Currently running TrueNAS Core 12.0-U6.

I just noticed that there is a Checksum option when creating a new dataset, and it defaults to SHA512.

All my datasets (created prior to 12.0-U6, and likely on 12.0-RELEASE) are using On for their Checksum option. (Confirmed with GUI and with console.) I'm assuming this is Fletcher4.

What does this mean? Is SHA512 faster than Fletcher4? More reliable? Is it overkill to be used on standards ZFS datasets, when the previous default (Fletcher4) is "good enough"? Why has SHA512 been made the new default? (Is this an iXsystems' decision or upstream OpenZFS?)

It appears that if you use SHA512, you might introduce compatibility issues when attempting to read the data on an older system / ZFS version:

Checksums and Their Use in ZFS — OpenZFS documentation

openzfs.github.io

In the meantime, I'm going to continue to override the default SHA512 with On, until there's a clear explanation for this new default.

Thanks for documenting and reporting.

We've confirmed that this was an unintended mistake. It still passed all datapath tests, but we did not have a check on the default for non-ddup datasets.

SHA-512 is generally very performant and runs better on 64 bit processors than SHA-256... so most users will not see any differences. Overriding with fletcher for non-ddup datasets is still OK.

We're looking at providing an interim update to correct this issue. We'll be able to confirm next week. SHA-512 willl still be the default for ddup datasets.

 

[ChrisRJ]

I am a bit confused what I should read from this thread regarding the QA process. Could someone possible comment here? Thanks!

 

[winnielinnie]

Thanks for documenting and reporting.

We've confirmed that this was an unintended mistake. It still passed all datapath tests, but we did not have a check on the default for non-ddup datasets.

@morganL, thank you for chiming in.

I want to recap that I have multiple concerns from this, which I've explained in the bug report as well as in this thread. I'll list them out as bullet-points:

• The original Improvement Suggestion (NAS-110523) mentions "Dedup" and "Dedup customers", which implies it only affects new datasets in which ZFS Deduplication is intentionally enabled (or inherited.) There's even a remark on how it is now recommended to manually select SHA512 for new Dedup datasets (prior to this new default Improvement being implemented in U6.) It's obvious that this new default can wait (not urgent), since TrueNAS/FreeNAS systems have been using SHA256 for Dedup datasets; only missing out on the performance improvements from SHA512 on 64bit architecture.
• Yet updating from U5.1 -> U6 introduces this new default (anything U2+ is considered ready for "mission critical" use-cases.) This wasn't a matter of upgrading from 12.x -> 13.x.
• It was implemented incorrectly, affecting the defaults for all new datasets, including the more ubiquitous non-Dedup datasets. This can lead to a performance penalty from Fletcher4 -> SHA512 on new non-Dedup datasets. (I'm no expert, but if I was the gambling type, I'd bet my house that there exists more non-Dedup datasets than Dedup.)
• The popup warning dialog is inaccurate and ambiguous. It is triggered to show when the user enables ZFS Deduplication, yet the warning / popup makes no mention of Deduplication whatsoever. (Must also note that the dialog mentions the "Checksum" applied on the ZFS pool, when in fact "Checksum" is a dataset property.)

We're looking at providing an interim update to correct this issue. We'll be able to confirm next week. SHA-512 willl still be the default for ddup datasets.

I'm relieved to know this has been pushed back to TrueNAS CORE 13.x, and until then a recommendation to all your customers to start using SHA512 on their Dedup datasets makes sense. I based it from NAS-110523, which has been changed to "Fix Version/s: 13.0". Are you saying it hasn't been pushed back to 13.x, then? Can we expect U6.1 or U7 to undo this in the meantime?

Like @ChrisRJ, I'm nervous about the QA process.

 

[morganL]

@morganL, thank you for chiming in.

I want to recap that I have multiple concerns from this, which I've explained in the bug report as well as in this thread. I'll list them out as bullet-points:

• The original Improvement Suggestion (NAS-110523) mentions "Dedup" and "Dedup customers", which implies it only affects new datasets in which ZFS Deduplication is intentionally enabled (or inherited.) There's even a remark on how it is now recommended to manually select SHA512 for new Dedup datasets (prior to this new default Improvement being implemented in U6.) It's obvious that this new default can wait (not urgent), since TrueNAS/FreeNAS systems have been using SHA256 for Dedup datasets; only missing out on the performance improvements from SHA512 on 64bit architecture.
• Yet updating from U5.1 -> U6 introduces this new default (anything U2+ is considered ready for "mission critical" use-cases.) This wasn't a matter of upgrading from 12.x -> 13.x.
• It was implemented incorrectly, affecting the defaults for all new datasets, including the more ubiquitous non-Dedup datasets. This can lead to a performance penalty from Fletcher4 -> SHA512 on new non-Dedup datasets. (I'm no expert, but if I was the gambling type, I'd bet my house that there exists more non-Dedup datasets than Dedup.)
• The popup warning dialog is inaccurate and ambiguous. (It is triggered to show when the user enables ZFS Deduplication, yet the warning / popup makes no mention of Deduplication whatsoever.)

I'm relieved to know this has been pushed back to TrueNAS CORE 13.x, and until then a recommendation to all your customers to start using SHA512 on their Dedup datasets makes sense. I based it from NAS-110523, which has been changed to "Fix Version/s: 13.0". Are you saying it hasn't been pushed back to 13.x, then? Can we expect U6.1 or U7 to undo this in the meantime?

Like @ChrisRJ, I'm nervous about the QA process.

If the change was only made for dedup datasets, everything would be fine.... the dialog would be OK even if a little ambiguous.
The automated QA process did not catch this...primarily because the system passed all tests - both dedupe and non-dedupe. The impact is minor (except for porting the pool back to older software). No data is harmed. The Community QA process did work.. thank you.
We are reviewing how to make the changes in 12.0-U7 or earlier. we can't decide until we have worked out the fix.

 

[cela96]

Hello, after updating to TrueNAS-12.0-U6.1 the checksum option completely disappeared from dedup and non dedup dataset.
Am I doing something wrong?
Thanks

 

[Samuel Tai]

Hello, after updating to TrueNAS-12.0-U6.1 the checksum option completely disappeared from dedup and non dedup dataset.
Am I doing something wrong?
Thanks

View attachment 50716 View attachment 50715

Please read the Release Notes for 12.0-U6.1. In particular, see https://jira.ixsystems.com/browse/NAS-113018. The default for non-dedup datasets stays at Fletcher4. For dedup datasets, it's now SHA512.

 

[cela96]

Please read the Release Notes for 12.0-U6.1. In particular, see https://jira.ixsystems.com/browse/NAS-113018. The default for non-dedup datasets stays at Fletcher4. For dedup datasets, it's now SHA512.

Yes, however, before the update I could change the checksum algorithm from the GUI, now the option has been removed completely.
I still have some non dedup datasets created with the previous version in SHA512 and I would like to be able to change the setting without using CLI.

Code:

zfs get checksum | grep sha512

This command returned to me dedup and non dedup dataset, probably it has been fixed for new datasets, however for existing dataset i cannot change the checksum algorithm with TrueNAS-12.0-U6.1 GUI thing which I previously could do.

Basically what I'm trying to say is that the checksum selection is completely gone from GUI.

Thanks

 

[Samuel Tai]

Yes, this appears to be a deliberate design choice to remove the ability to change the checksum algorithm from the GUI. You'll have to resort to the CLI to change this.

 

[cela96]

Yes, this appears to be a deliberate design choice to remove the ability to change the checksum algorithm from the GUI. You'll have to resort to the CLI to change this.

Ok, thanks
I did further tests and I think there is still something wrong, I created 2 new datasets in a new pool and despite being dedup ON the checksum remained on "on" instead of "SHA512"

"TrueNAS-12.0-U6.1-checksum-test" is the name of the new pool created 10 minutes ago

 

[Samuel Tai]

Go ahead and submit a bug report. It's entirely possible SHA512 was disabled as an option accidentally in this release.

 

[winnielinnie]

This command returned to me dedup and non dedup dataset, probably it has been fixed for new datasets, however for existing dataset i cannot change the checksum algorithm with TrueNAS-12.0-U6.1 GUI thing which I previously could do.

Like @Samuel Tai mentioned, you can use the CLI to change them back to "on". In the meantime, until they fix/restore the GUI option.

You can manually change the non-dedup (that were previously created) from "sha512" to "on". And you can manually change the dedup from "on" (which is sha256) to "sha512".

For non-dedup:
zfs set checksum=on poolname/nondedup-ds

For dedup:
zfs set checksum=sha512 poolname/dedup-ds

 

[Ravefiend]

So just to confirm on TrueNAS-12.0-U6, newly created datasets having ZFS Deduplication turned Off and on which new data was copied onto while Checksum was set to SHA512, do not need to be recreated for optimal performance?

 

[winnielinnie]

In your case, yes.

Simple rule of thumb:

No Dedup used on a dataset? Set checksum=on (which uses the default Fletcher4 checksum)

Currently using Dedup on a dataset? Set checksum=sha512 (which uses the SHA512t256 checksum; truncated form of SHA512 for ZFS)

Once the GUI option is reimplemented/fixed (12.0-U7? 13.0-RELEASE?), then you can set it from the GUI instead of the command-line.

---

FURTHER DISCUSSION:

If upstream OpenZFS ever changes the default Checksum for Dedup datasets to SHA512t256, then leaving it at the default or setting checksum=on will suffice. However, as it stands now, Dedup datasets set to checksum=on default to SHA256.

Non-dedup have been and always should remain checksum=on (Fletcher4).

 

[Ravefiend]

In your case, yes.

Simple rule of thumb:

No Dedup used on a dataset? Set checksum=on (which uses the default Fletcher4 checksum)

Thanks for confirming that!