Blog Revolutionising Research Data Storage with TrueNAS

[StorageEngineer]

Revolutionising Research Data Storage with TrueNAS​

Introduction:​

In 2015, I found myself confronted with a pressing issue within the research community – the escalating challenge of managing and preserving vast amounts of data generated by researchers. As data resolutions soared, the need to retain this information for over a decade after publication became apparent, creating a significant strain on existing storage systems. Many universities were grappling with outdated infrastructure, struggling to cope with the sheer volume of data, and researchers resorted to makeshift solutions like multiple external hard drives, small NAS boxes on desks and sometimes even makeshift storage server setups stored in broom cupboards.

Recognising the urgency of the situation, I embarked on a mission to find a solution that would revolutionise the way we approached data storage in the academic world.
​

Identifying the Problem:​

The crux of the issue lay in the inadequacy of existing storage systems. Legacy systems were often incapable of handling the increasing data loads efficiently, leading to challenges in backup times, data security, and cost-effectiveness. It was clear that a paradigm shift was needed to address these challenges comprehensively.
​

Rethinking Data Storage Costs​

One of the fundamental aspects of data storage is its cost structure, consisting of both hardware and software expenses. After extensive research, I observed a pattern where the cost of software remained proportional to the hardware, and additional tools were often required for backup purposes. This prompted me to explore alternatives that could not only reduce costs but also offer enhanced features crucial for a research-intensive environment.
​

Enter ZFS and TrueNAS:​

My quest led me to the discovery of ZFS and its integration with TrueNAS. This combination proved to be a game-changer. TrueNAS provided a versatile solution by enabling data sharing via Samba, a widely used method within the University. Its snapshot functionality addressed the need for recovery points in case of accidental or malicious data deletions. Furthermore, the replication feature allowed for secure backups to secondary systems, mitigating the risk of data loss.
​

Cost-Effective Centralisation:​

One of TrueNAS's standout features was its cost-effectiveness. It became the centralised solution for researchers across the university, integrating seamlessly with Microsoft Active Directory for secure authentication. This not only enhanced data security but also facilitated easier collaboration, encouraging researchers to share data across disciplines.
​

Financial Impact:​

The financial implications of adopting TrueNAS were ground-breaking. In 2015, the cost to store and backup 1TB of data was a staggering £225. Contrastingly, implementing TrueNAS, including backup and a hardware refresh every five years, reduced the cost to a mere £35. Extrapolating this to a petabyte scale, the university stood to save almost one million pounds every five years for every petabyte built. Costs continued to tumble as the years went by and the service grew and currently, we are at £15/TB/Yr which includes that secondary backup system and a hardware refresh every five years. It’s hard to imagine how costs could reduce even further but we shall see.
​

Benefits Beyond Cost Reduction:​

The adoption of ZFS and TrueNAS brought about more than just cost savings. These systems offered additional advantages, including data integrity through checksums, periodic scrubs for error detection and correction, inline compression for space optimisation, and more. The comprehensive suite of features ensured not only efficient data storage but also robust data management practices.
​

Transformational Impact:​

The implementation of ZFS and TrueNAS marked a transformative phase in our approach to data storage. The first two systems built (diagram below) boasted 360TB of RAW capacity, a testament to the scalability and efficiency of this solution. The shift from legacy systems to a more modern, adaptable infrastructure not only addressed the immediate challenges but also positioned the University for future advancements in research data management.
​

Scaling Up: 40PB and Counting:​

Buoyed by the remarkable success of the initial implementations, I went on to build numerous TrueNAS servers over the next eight years. The total storage capacity now stands at 40 petabytes, solidifying TrueNAS as the backbone of our university's research data infrastructure.
​

Conclusion:​

In addressing the pressing issue of data storage challenges in the academic realm, the journey with ZFS and TrueNAS exemplifies the impact of innovative thinking and strategic technology adoption. As researchers continue to push the boundaries of data generation, having a robust, cost-effective, and feature-rich storage solution is imperative. The story of TrueNAS and ZFS stands as a testament to the power of reimagining existing paradigms to meet the evolving needs of the research community.​