- Joined
- Mar 25, 2021
- Messages
- 204
With much love for the communities, iXsystems and Klara have developed and donated the Fast Dedup capability for OpenZFS. The Open Source software has been made available to the OpenZFS community for review, further testing and integration into a future release of OpenZFS. The Fast Dedup capability will also be available in future releases of TrueNAS with it landing in the nightly images of TrueNAS SCALE in March 2024.
Fast Dedup is a major overhaul of the original OpenZFS deduplication capability. One of the primary issues with traditional deduplication with ZFS has been the need to keep the deduplication hash tables in memory at all times to avoid massive performance penalties. This existing functionality led to performance challenges when scaling up capacity and usability issues during operation.
With the introduction of Fast Dedup, there have been several major innovations including:
Led by Allan Jude of Klara, the development team included engineers from Klara and iXsystems. The development was completed in less than a year thanks to the strong OpenZFS expertise of the team. The project was primarily sponsored by iXsystems for use in TrueNAS, the industry’s most used storage platform.
“Klara is very proud of the speed and functional quality of the Fast Dedup development.” said Allan Jude. “We look forward to completing the integration and testing work needed for this to be a standard OpenZFS capability, with no extra cost.”
By contributing the software now, Klara and iXsystems are working to ensure it will have the quality needed to be included in OpenZFS 2.3. The contributed software has gone through preliminary validation and we look forward to wider community testing. Making it available to the OpenZFS community will allow enhanced review and testing with a broader range of use cases and capacities.
“The OpenZFS Community is very excited to receive this major contribution of Fast Dedup software,” said Matt Ahrens. “Fast Dedup has been a major user desire for many years and we are pleased to see that Klara and iXsystems have developed the feature in-line with our general requirements. We look forward to testing and integrating the new software.”
OpenZFS 2.3 is expected to take most of the next nine months to mature and reach release quality and status. These Fast Dedup features will be integrated with TrueNAS SCALE “Electric Eel” along with other expected OpenZFS improvements.
Test software, not to be used for production, is expected in Q2 for TrueNAS SCALE. This will allow validation of functionality and performance prior to a production release.
A formal TrueNAS release with Fast Dedup included is planned for the second half of 2024. This will be in both TrueNAS SCALE Community Edition and TrueNAS Enterprise appliances, including the TrueNAS F-Series, M-Series, and R-Series.
“Fast Dedup has been a longstanding desired feature of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the performance.” said Kris Moore, SVP of Engineering. “These attributes will significantly improve the economics of OpenZFS storage relative to Cloud storage and proprietary storage, and our team could not be more ecstatic to see this come to fruition.”
For more information or to volunteer for the ongoing testing and validation processes, please contact one of the primary developers.
iXsystems is the company behind TrueNAS. You can make an impact in the community by getting to know TrueNAS SCALE now, developing on nightly builds in March, and testing closer to RELEASE later this year. If we can also help your company with TrueNAS Enterprise appliances, book a time to chat with a product specialist.
Klara is a leading developer of OpenZFS and FreeBSD software. Interested in Klara developing software for your OpenZFS project? Contact Klara to learn more about their offerings.
The post Fast Dedup is a Valentines Gift to the OpenZFS and TrueNAS Communities appeared first on iXsystems, Inc. - Enterprise Storage & Servers.
Continue reading...
Fast Dedup is a major overhaul of the original OpenZFS deduplication capability. One of the primary issues with traditional deduplication with ZFS has been the need to keep the deduplication hash tables in memory at all times to avoid massive performance penalties. This existing functionality led to performance challenges when scaling up capacity and usability issues during operation.
With the introduction of Fast Dedup, there have been several major innovations including:
- The size of metadata is now dynamically sized to fit in either RAM or dedicated flash devices to avoid hitting the performance penalty wall.
- The metadata structure has been completely re-engineered to enable efficient updates using a log append process, greatly improving performance for large updates such as deletions.
- The dedup table will favor dedup-able data and prune blocks that show no dedup potential.
- Combining metadata improvements with properly configured storage, including dedicated metadata flash devices, will improve the sustained dedup performance by over an order of magnitude for larger systems.
Kudos to the Development Team
Led by Allan Jude of Klara, the development team included engineers from Klara and iXsystems. The development was completed in less than a year thanks to the strong OpenZFS expertise of the team. The project was primarily sponsored by iXsystems for use in TrueNAS, the industry’s most used storage platform.
“Klara is very proud of the speed and functional quality of the Fast Dedup development.” said Allan Jude. “We look forward to completing the integration and testing work needed for this to be a standard OpenZFS capability, with no extra cost.”
By contributing the software now, Klara and iXsystems are working to ensure it will have the quality needed to be included in OpenZFS 2.3. The contributed software has gone through preliminary validation and we look forward to wider community testing. Making it available to the OpenZFS community will allow enhanced review and testing with a broader range of use cases and capacities.
“The OpenZFS Community is very excited to receive this major contribution of Fast Dedup software,” said Matt Ahrens. “Fast Dedup has been a major user desire for many years and we are pleased to see that Klara and iXsystems have developed the feature in-line with our general requirements. We look forward to testing and integrating the new software.”
OpenZFS 2.3 is expected to take most of the next nine months to mature and reach release quality and status. These Fast Dedup features will be integrated with TrueNAS SCALE “Electric Eel” along with other expected OpenZFS improvements.
TrueNAS with Fast Dedup
Test software, not to be used for production, is expected in Q2 for TrueNAS SCALE. This will allow validation of functionality and performance prior to a production release.
A formal TrueNAS release with Fast Dedup included is planned for the second half of 2024. This will be in both TrueNAS SCALE Community Edition and TrueNAS Enterprise appliances, including the TrueNAS F-Series, M-Series, and R-Series.
“Fast Dedup has been a longstanding desired feature of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the performance.” said Kris Moore, SVP of Engineering. “These attributes will significantly improve the economics of OpenZFS storage relative to Cloud storage and proprietary storage, and our team could not be more ecstatic to see this come to fruition.”
Contact the Developers
For more information or to volunteer for the ongoing testing and validation processes, please contact one of the primary developers.
iXsystems is the company behind TrueNAS. You can make an impact in the community by getting to know TrueNAS SCALE now, developing on nightly builds in March, and testing closer to RELEASE later this year. If we can also help your company with TrueNAS Enterprise appliances, book a time to chat with a product specialist.
Klara is a leading developer of OpenZFS and FreeBSD software. Interested in Klara developing software for your OpenZFS project? Contact Klara to learn more about their offerings.
The post Fast Dedup is a Valentines Gift to the OpenZFS and TrueNAS Communities appeared first on iXsystems, Inc. - Enterprise Storage & Servers.
Continue reading...
