Blog LinusTechTips says “Goodbye Windows, Hello TrueNAS”

[JoshDW19]

Video editing is one of the more demanding NAS workloads these days. The move to 4K and 8K video means that massive files need to be transferred with reliability and speed. LinusTech Media is one of the organizations that has these challenges, and they recently migrated from Windows storage server to TrueNAS 12.0 for its speed and reliability.

As always, Linus Sebastian and Jake Tivy produced a thoroughly entertaining and technically educational video. It’s well worth the 20 minutes to watch.

Linus Media previously used a Windows Storage Spaces server for video editing and TrueNAS as a reliable backup and archive of their data. They identified several key problems with their Windows storage that hindered their business operations:

1. 1. Insufficient throughput to the drive storage
   2. Excessive writing of unprotected data to RAM
   3. “Blue Screen of Death” crashes happening too frequently
   4. Storage “hanging” under high write loads – disrupting the whole business
   5. The storage was too slow – making editing more difficult

In looking for a replacement, they were looking for a Linux system with the latest ZFS, Samba, and preferably a UI. While TrueNAS SCALE meets those requirements, it’s still in BETA, so we recommended TrueNAS CORE to them because they were running a production workload that was critical to their business. TrueNAS CORE 12, which has a decade of inherited development and community testing, has been in a release state for over a year. This made it a very mature, excellent choice for a mission critical use-case of this nature.

They built a high-end system with a 32 core AMD EPYC processor, 256 GB of RAM, dual 100Gbe NIC, and Gen4 NVMe drives. Performance was excellent and demonstrated the ZFS file system operating at 14-18GB/s internally within the system, far in excess of what the Windows system could achieve.

They then set about testing it with their video editors and ongoing transfers via SMB. When storage performance is adequate, the editors see very little lag and the system can stop on a dime, without “runaway footage”. The TrueNAS system delivered that performance and can likely sustain it for many more editors.

The reliability of TrueNAS under load comes from the magic of OpenZFS which is engineered for great stability and robustness. Unlike Windows, OpenZFS limits the amount of DRAM used for “dirty” or unprotected data. Its design also behaves much more consistently under high load and hence supports more critical workloads like video editing and virtualization storage. FreeBSD and TrueNAS have also had extensive field testing (millions of machine years) as a dedicated storage environment which has largely eliminated the system crash issues of the overly complex and general purpose Windows Server environment.



Technical Notes


The video included a few technical recommendations which are worth discussing.

Enable TRIM on SSDs: TRIM probably isn’t required for the performance seen. However, the primary reason for not enabling TRIM by default is that some SSDs behave badly with TRIM enabled. The Kioxia CD6 SSDs used in the video do not have this issue.​

Disabling Compression: They disabled compression for storing video files. That’s not a bad decision, but might not have been necessary. OpenZFS is very good at detecting and skipping over incompressible data, so the penalty for leaving the default (compression on) isn’t likely to be noticeable in most cases.​

Using ARC only for Metadata: For applications where data is not shared directly between many clients, it’s not a bad strategy. It reduces the amount of CPU used for managing a large ARC with data. If the cache hit rate would otherwise be low, then it is worth considering. For most use-cases, the use of ARC is beneficial.​

SMB Multi-Channel: Multi-channel does improve the maximum performance seen by a single client but is rarely needed. It is not enabled by default because there are corner cases, particularly over a WAN, where a file can be corrupted. However, this is unlikely to happen in a LAN workgroup environment. The latest Samba in TrueNAS SCALE now has multi-channel readily available.​

Single vs Multiple VDEVs: It was indicated that a single VDEV performed about as well as two VDEVs. This is probably dependent on the testing approach being used. For a single client it is likely to be true, but for many clients, more VDEVs can significantly improve overall performance.​

Multiple boot drives: Yes, TrueNAS lets you mirror boot drives internally. For older SATA SSDs and HDDs, we did recommend mirror drives. For newer and high quality M.2 SSD boot drives, we are finding a single boot drive to provide enterprise level system reliability.​

RAID-Z1 vs RAID-Z2: They used RAID-Z1 (single parity) for their pool. This is generally fine with good quality SSDs (test or burn-in the SSDs first!) which resilver quickly and fail infrequently. As they indicated, they also have a good backup plan. With HDDs, we recommend RAID-Z2 for reliable redundancy on the local storage pool.​

Get Jake to deploy TrueNAS: Linus made a comical jab at some previous issues he had with FreeNAS. In his defense, he was often trying to push the envelope on new features/ hardware and trying to do it while doing a video. Jake sets up his systems methodically in the background, tests them thoroughly, and makes sure they work. As much as we love Linus, new TrueNAS users should aim to be more like Jake . Alternatively, give iXsystems a call and we’ll get you a turnkey appliance that removes any guesswork.​

The TrueNAS Software


TrueNAS CORE is free, just like the older FreeNAS. TrueNAS 12.0-U6 was released on October 5, 2021 and includes a number of fixes, about ten improvements, and a few platform enhancements. These include:

• • NFSv4 HA improvements (Enterprise version)
  • More extensive SMB regression testing with various SMB options
  • SMB shadow copies feature has been fixed (bug in U5)
  • OpenZFS dedup performance improvements by defaulting to SHA-512 algorithm
  • Better webUI snapshot filtering by name
  • UI improvements for disk temperature monitoring and UEFI booting
  • Ongoing improvements to UI & middleware performance for large drive counts
  • Improved UI for plugin updates/upgrades
  • Various M-Series and R-Series webUI improvements

Please check out the updated TrueNAS documentation even if you don’t upgrade today. We’re extremely grateful for all the contributions received thus far and encourage the community to keep the suggestions coming!

TrueNAS SCALE will perform similarly to the version Linus and Jake tests. TrueNAS SCALE is entering the release-candidate stage on Tuesday 26th October. Performance testing of SCALE will be happening over the coming months and business production use of SCALE should start in Q1 2022.



TrueNAS Hardware Platforms


TrueNAS 12.0-U6 is compatible with all of the iXsystems platforms from the TrueNAS Minis, to the power-efficient X-Series, all the way up to the flagship High Availability (HA) M-Series. There is also an R-Series product line that can run CORE, Enterprise, and SCALE editions of TrueNAS. All of these can be updated via the web UI and include graphical enclosure management.

For an HA solution with similar performance to the system built by Linus, look at the TrueNAS M50 or M60. These systems will continue to operate even if one of the controllers were to fail. Combined with Enterprise support, you will have a 24×365 system with five nines of availability (99.999%).

If you have any questions or comments, we’d love to hear them on the community forums, on the TrueNAS subreddit, or in response to this blog. If you need additional information on how TrueNAS can streamline, accelerate, and unify data management for your business, please contact us, or visit the newly redesigned truenas.com and download the TrueNAS version that best fits your needs.





The post LinusTechTips says “Goodbye Windows, Hello TrueNAS” appeared first on iXsystems, Inc. - Enterprise Storage & Servers.

Continue reading...

 

[Basil Hendroff]

If you have any questions or comments, we’d love to hear them on the community forums,

Thanks to @horizonbrave for alerting the community to this earlier. Some feedback in the thread Linus Tech Tips deploying TrueNAS for their production server. While the feedback is generally critical, and rightly so, I think the positive to take away from this is that TrueNAS, against its Linux competitors, was singled out as the preferred ZFS platform.

 

[SweetAndLow]

But why raid z1 and not a stripe or mirrors

 

[horizonbrave]

Thanks to @horizonbrave for alerting the community to this earlier. Some feedback in the thread Linus Tech Tips deploying TrueNAS for their production server. While the feedback is generally critical, and rightly so, I think the positive to take away from this is that TrueNAS, against its Linux competitors, was singled out as the preferred ZFS platform.

Actually part of my post intent was to alert the team on this (in case they were not aware which I doubted) to prevent and help LTT from fucking up and turning the opportunity of a good advertising window into the exact opposite. I'm glad Linus and Jake identified TrueNAS as a good fit for their highly critical business use case.

PS: I just wish they'd be a sustainable/profitable (and fisable for the consumer) solution for iXsystems to penetrate markets others then the North American one (I'm based in Australia).

 

[jgreco]

But why raid z1 and not a stripe or mirrors

Well, a stripe would leave no redundancy, which is generally bad for ZFS, and mirrors wastes a lot of space. If performance is good enough with RAIDZ1, and you're not storing mission-critical data, and your devices have good reliability, then RAIDZ1 is probably a decent choice. Since it sounds like this is mainly for video editing, and they have some sort of backup plan, it's probably fine.

 

[jgreco]

Actually part of my post intent was to alert the team on this (in case they were not aware which I doubted) to prevent and help LTT from ****ing up and turning the opportunity of a good advertising window into the exact opposite. I'm glad Linus and Jake identified TrueNAS as a good fit for their highly critical business use case.

So it took nearly six years, but at least they finally went from this sad state of affairs to something that is hopefully going to be workable and reliable.

I'm not sure generating Youtube videos is a "highly critical business use case", but having a well-designed storage platform is good for productivity.

 

[horizonbrave]

this sad state of affairs

sorry link is not openable ("You do not have permission to view this page or perform this action.")

I'm not sure generating Youtube videos is a "highly critical business use case",

perhaps bad wording, but I guess for a very profitable company and its employees must be a pretty important task being the backbone for the production of their products

 

[jgreco]

sorry link is not openable ("You do not have permission to view this page or perform this action.")

Mmm, interesting. Community discussion of https://www.youtube.com/watch?v=QWPM_cKN-6U ...

 

[Samuel Tai]

@jgreco, the link works for me, but that thread is archived under the Team TrueNAS>Archive: Old Threads board, which is restricted to moderators.

 

[jgreco]

@jgreco, the link works for me, but that thread is archived under the Team TrueNAS>Archive: Old Threads board, which is restricted to moderators.

Yeah, I know, which is why I brought the relevant link out.

 

[danb35]

prevent and help LTT from ****ing up and turning the opportunity of a good advertising window into the exact opposite.

...which they've already done once with FreeNAS, and countless other times with other products. Really, IMO, iX' only sensible move would be to distance themselves from this moron. But they apparently feel that "there's no such thing as bad publicity."

 

[jgreco]

...which they've already done once with FreeNAS, and countless other times with other products. Really, IMO, iX' only sensible move would be to distance themselves from this moron. But they apparently feel that "there's no such thing as bad publicity."

Okay, we gave them a well-deserved scolding six years ago (and a few other times IIRC) about all that. It's possible for things to change. Six years is more than an eighth of a person's professional life, and it could well be that in the meantime, lessons have been learned about the difference between Youtube performance stunts and serious things needed for the success of the business.

The job of creating those fundamentals for the success of the business tends not to be a glamorous affair, but when someone is taking that seriously, it should be considered a success. Perhaps that is unwarranted optimism, but I do believe things can change and improve.

 

[danb35]

lessons have been learned about the difference between Youtube performance stunts and serious things needed for the success of the business.

I'd hope so, but perhaps I'm less optimistic than you. And his decision to use RAIDZ1 for his storage doesn't add to my optimism. But with that said, since I watched his monstrosity of a pfSense build, I haven't seen any reason to watch his channel, so it's entirely possible he's learned a thing or two since then.

 

[Arwen]

@danb35 - But Linus is so bad, it's good!

I do watch some of the videos just to learn what not to do.

Plus, occasionally he actually finds a real problem that needs reporting and fixing. Like with AMD Epyc CPUs with a full boat of NVMe drives has some interrupt problem on Linux kernels... I don't remember the details, but I was glad someone found the problem. (It can't exist with Intel because no Intel CPU has 128 direct connect PCIe lanes... so that makes it new.)

 

[diskdiddler]

I saw that video, I sure hope they can eek out the final performance from it to leave Windows in the dust.

 

[Herr_Merlin]

I don't get why they disabled compression to get more speed.. the endresult with LZ4 should be more speed than without.

 

[HoneyBadger]

I don't get why they disabled compression to get more speed.. the endresult with LZ4 should be more speed than without.

Video data tends to be completely incompressible, and when you're at the level of "measurement" that comes with a 24-bay NVMe system, the overhead from even attempting compression could be significant.

 

[Shane from IT]

Video editing is one of the more demanding NAS workloads these days. The move to 4K and 8K video means that massive files need to be transferred with reliability and speed. LinusTech Media is one of the organizations that has these challenges, and they recently migrated from Windows storage server to TrueNAS 12.0 for its speed and reliability.

As always, Linus Sebastian and Jake Tivy produced a thoroughly entertaining and technically educational video. It’s well worth the 20 minutes to watch.

Linus Media previously used a Windows Storage Spaces server for video editing and TrueNAS as a reliable backup and archive of their data. They identified several key problems with their Windows storage that hindered their business operations:

1. 1. Insufficient throughput to the drive storage
   2. Excessive writing of unprotected data to RAM
   3. “Blue Screen of Death” crashes happening too frequently
   4. Storage “hanging” under high write loads – disrupting the whole business
   5. The storage was too slow – making editing more difficult

In looking for a replacement, they were looking for a Linux system with the latest ZFS, Samba, and preferably a UI. While TrueNAS SCALE meets those requirements, it’s still in BETA, so we recommended TrueNAS CORE to them because they were running a production workload that was critical to their business. TrueNAS CORE 12, which has a decade of inherited development and community testing, has been in a release state for over a year. This made it a very mature, excellent choice for a mission critical use-case of this nature.

They built a high-end system with a 32 core AMD EPYC processor, 256 GB of RAM, dual 100Gbe NIC, and Gen4 NVMe drives. Performance was excellent and demonstrated the ZFS file system operating at 14-18GB/s internally within the system, far in excess of what the Windows system could achieve.

They then set about testing it with their video editors and ongoing transfers via SMB. When storage performance is adequate, the editors see very little lag and the system can stop on a dime, without “runaway footage”. The TrueNAS system delivered that performance and can likely sustain it for many more editors.

The reliability of TrueNAS under load comes from the magic of OpenZFS which is engineered for great stability and robustness. Unlike Windows, OpenZFS limits the amount of DRAM used for “dirty” or unprotected data. Its design also behaves much more consistently under high load and hence supports more critical workloads like video editing and virtualization storage. FreeBSD and TrueNAS have also had extensive field testing (millions of machine years) as a dedicated storage environment which has largely eliminated the system crash issues of the overly complex and general purpose Windows Server environment.



Technical Notes


The video included a few technical recommendations which are worth discussing.

Enable TRIM on SSDs: TRIM probably isn’t required for the performance seen. However, the primary reason for not enabling TRIM by default is that some SSDs behave badly with TRIM enabled. The Kioxia CD6 SSDs used in the video do not have this issue.​

Disabling Compression: They disabled compression for storing video files. That’s not a bad decision, but might not have been necessary. OpenZFS is very good at detecting and skipping over incompressible data, so the penalty for leaving the default (compression on) isn’t likely to be noticeable in most cases.​

Using ARC only for Metadata: For applications where data is not shared directly between many clients, it’s not a bad strategy. It reduces the amount of CPU used for managing a large ARC with data. If the cache hit rate would otherwise be low, then it is worth considering. For most use-cases, the use of ARC is beneficial.​

SMB Multi-Channel: Multi-channel does improve the maximum performance seen by a single client but is rarely needed. It is not enabled by default because there are corner cases, particularly over a WAN, where a file can be corrupted. However, this is unlikely to happen in a LAN workgroup environment. The latest Samba in TrueNAS SCALE now has multi-channel readily available.​

Single vs Multiple VDEVs: It was indicated that a single VDEV performed about as well as two VDEVs. This is probably dependent on the testing approach being used. For a single client it is likely to be true, but for many clients, more VDEVs can significantly improve overall performance.​

Multiple boot drives: Yes, TrueNAS lets you mirror boot drives internally. For older SATA SSDs and HDDs, we did recommend mirror drives. For newer and high quality M.2 SSD boot drives, we are finding a single boot drive to provide enterprise level system reliability.​

RAID-Z1 vs RAID-Z2: They used RAID-Z1 (single parity) for their pool. This is generally fine with good quality SSDs (test or burn-in the SSDs first!) which resilver quickly and fail infrequently. As they indicated, they also have a good backup plan. With HDDs, we recommend RAID-Z2 for reliable redundancy on the local storage pool.​

Get Jake to deploy TrueNAS: Linus made a comical jab at some previous issues he had with FreeNAS. In his defense, he was often trying to push the envelope on new features/ hardware and trying to do it while doing a video. Jake sets up his systems methodically in the background, tests them thoroughly, and makes sure they work. As much as we love Linus, new TrueNAS users should aim to be more like Jake . Alternatively, give iXsystems a call and we’ll get you a turnkey appliance that removes any guesswork.​

The TrueNAS Software


TrueNAS CORE is free, just like the older FreeNAS. TrueNAS 12.0-U6 was released on October 5, 2021 and includes a number of fixes, about ten improvements, and a few platform enhancements. These include:

• • NFSv4 HA improvements (Enterprise version)
  • More extensive SMB regression testing with various SMB options
  • SMB shadow copies feature has been fixed (bug in U5)
  • OpenZFS dedup performance improvements by defaulting to SHA-512 algorithm
  • Better webUI snapshot filtering by name
  • UI improvements for disk temperature monitoring and UEFI booting
  • Ongoing improvements to UI & middleware performance for large drive counts
  • Improved UI for plugin updates/upgrades
  • Various M-Series and R-Series webUI improvements

Please check out the updated TrueNAS documentation even if you don’t upgrade today. We’re extremely grateful for all the contributions received thus far and encourage the community to keep the suggestions coming!

TrueNAS SCALE will perform similarly to the version Linus and Jake tests. TrueNAS SCALE is entering the release-candidate stage on Tuesday 26th October. Performance testing of SCALE will be happening over the coming months and business production use of SCALE should start in Q1 2022.



TrueNAS Hardware Platforms


TrueNAS 12.0-U6 is compatible with all of the iXsystems platforms from the TrueNAS Minis, to the power-efficient X-Series, all the way up to the flagship High Availability (HA) M-Series. There is also an R-Series product line that can run CORE, Enterprise, and SCALE editions of TrueNAS. All of these can be updated via the web UI and include graphical enclosure management.

For an HA solution with similar performance to the system built by Linus, look at the TrueNAS M50 or M60. These systems will continue to operate even if one of the controllers were to fail. Combined with Enterprise support, you will have a 24×365 system with five nines of availability (99.999%).

If you have any questions or comments, we’d love to hear them on the community forums, on the TrueNAS subreddit, or in response to this blog. If you need additional information on how TrueNAS can streamline, accelerate, and unify data management for your business, please contact us, or visit the newly redesigned truenas.com and download the TrueNAS version that best fits your needs.





The post LinusTechTips says “Goodbye Windows, Hello TrueNAS” appeared first on iXsystems, Inc. - Enterprise Storage & Servers.

Continue reading...

cool stuff!