iXsystems and Klara reasserted their joint commitment
to open source with the contribution of the Fast Dedup feature to the OpenZFS
and TrueNAS communities. In a move sponsored and supported by engineers from
iXsystems and the development efforts of Klara, the contribution of Fast Dedup
significantly enhances sustained deduplication performance for ZFS and TrueNAS
storage systems, marking a new milestone in storage efficiency and performance.
Deduplication performance has been limited in ZFS and
OpenZFS since the inception of the file system due to its metadata handling.
Fast Dedup ameliorates this with the introduction of a re-engineered metadata
structure dynamically sized to operate in RAM or on dedicated flash devices for
expedited processing. These advancements collectively enable up to 20 times
faster performance and 5 times more usable capacity for certain workloads while
greatly improving reliability for systems with deduplication enabled. The
technology outperforms legacy dedup by employing algorithms and strategies that
scan, identify, and eliminate duplicates, while keeping metadata in RAM and
SSDs.
"The OpenZFS Community is very excited to receive this
major contribution of Fast Dedup software," said Matt Ahrens, co-creator of ZFS
and the OpenZFS Project. "Fast Dedup has been a major user desire for many
years and we are pleased to see that Klara and iXsystems have developed the
feature in-line with our general requirements. We look forward to testing and
integrating the new software."
Key Fast Dedup Advancements Include:
- Metadata size is now dynamically sized to fit on either
RAM or dedicated flash devices to avoid hitting the performance penalty wall.
- Metadata structure completely re-engineered to enable
efficient updates using a log append process, greatly improving performance for
large updates such as deletions.
- Dedup table now favors dedup-able data and prune blocks
that show no dedup potential.
Metadata improvements combined with properly configured
storage, including dedicated metadata flash devices designed to improve
sustained dedup performance by an order of magnitude or greater for larger
systems.
"Klara is very proud of the speed and functional quality
of the Fast Dedup development," said Allan Jude, FreeBSD Engineering Manager at
Klara "We look forward to completing the integration and testing work needed
for this to be a standard OpenZFS capability, with no extra cost."
"Fast Dedup has been a longstanding desired feature
of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the
performance." said Kris Moore, SVP of iXsystems Engineering. "These attributes
will significantly improve the economics of OpenZFS storage relative to Cloud
storage and proprietary storage, and our team could not be more ecstatic to see
this come to fruition."
The donation of Fast Dedup technology is a testament to
the shared vision and collaboration between iXsystems and Klara, underscoring
their dedication to the advancement of open-source storage solutions. Expected
in Q2 for TrueNAS SCALE, Fast Dedup will be available as test software, not to
be used for production. This will allow validation of functionality and
performance prior to a standard product release later in 2024.