-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum has become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Custom Compressor on free NAS
- Thread starter Bytales
- Start date
Ive been looking at some compressors here
and there are some that seem interesting.
It was just a thought i was toying with if it would be possible. I ran a free Nas VM under ESXI. but dumped ESXI, and currently run Linux Mint, where i managed to created a custom kernel with ZFS 0.8.0rc4 built in, which is what im running. Need to return to the FreeNAS Server to properly Export my pool, before ill be able to Reimport in Linux (didnt knew exporting was a Thing), and once done, im going to test my load (a Cryptocurrency blockchain) with all compression methods ZFS has, see what Transfer rates i get.
Im running it all on a WD Gold 10TB single drive.
The blockchain is pretty big at allmost 140gb, and wanted to see what Kind of numbers i get with different compression algorithms. Getting ZFS directly onto Linux, seems easier than installing FreeNAS as a VM under QEMU/KVM.
Ill post here what results i get withh the different compressors ZFS has built in.
Since i wont be running FreeNAS anymore, i guess im rather stuck with that ZFS 0.8.0rc4 has built in, and i think ive seen that FreeNAS itself has a lower Version of ZFS, if im not mistaken.
The on the fly compression is pretty dope. I dont know how much CPU something like this Needs, but the FreeNAS VM had assigned 4 virtual CPUs, whereas here in Linux, it has native acces to the full 32 Core Epyc CPU i have, so hopefully, all will work better.
and there are some that seem interesting.
It was just a thought i was toying with if it would be possible. I ran a free Nas VM under ESXI. but dumped ESXI, and currently run Linux Mint, where i managed to created a custom kernel with ZFS 0.8.0rc4 built in, which is what im running. Need to return to the FreeNAS Server to properly Export my pool, before ill be able to Reimport in Linux (didnt knew exporting was a Thing), and once done, im going to test my load (a Cryptocurrency blockchain) with all compression methods ZFS has, see what Transfer rates i get.
Im running it all on a WD Gold 10TB single drive.
The blockchain is pretty big at allmost 140gb, and wanted to see what Kind of numbers i get with different compression algorithms. Getting ZFS directly onto Linux, seems easier than installing FreeNAS as a VM under QEMU/KVM.
Ill post here what results i get withh the different compressors ZFS has built in.
Since i wont be running FreeNAS anymore, i guess im rather stuck with that ZFS 0.8.0rc4 has built in, and i think ive seen that FreeNAS itself has a lower Version of ZFS, if im not mistaken.
The on the fly compression is pretty dope. I dont know how much CPU something like this Needs, but the FreeNAS VM had assigned 4 virtual CPUs, whereas here in Linux, it has native acces to the full 32 Core Epyc CPU i have, so hopefully, all will work better.
Last edited:
- Joined
- May 17, 2014
- Messages
- 3,611
ZFS on Linux is close, (maybe a month or 2), to implementing ZSTD compression. Don't have any figures on how or if it's better, but, it does seem to add choices. Including selectable levels of compression verses time to compress.
Note that FreeBSD stopped their own tree of OpenZFS. They will start porting the ZFS on Linux, (which will make some concessions in source code to make it easier to compile on FreeBSD).
Note that FreeBSD stopped their own tree of OpenZFS. They will start porting the ZFS on Linux, (which will make some concessions in source code to make it easier to compile on FreeBSD).
So i did some tests. 141gb blockchain 2 mb ldb files, 73.441 files,
LZ4 1.27x, LZJB 1.22x, GZIP1 1.27x, GZIP9 1.00x, ZLE 1.22x
Write speed was about the same from a Smasung 950 Pro NVME SSD to the HDD, aprox 170 Mb/S
Zou know whats curious ? When coping, i dont hear the HDD spinning. But the whole deal is eating copius amounts of ram, ive seen the ram filling from 34% to 90% and the system has 192gb of RAM.
I compiled the latest Ubuntu Kernel 5.1.9 and the latest ZFS 0.8.1, so that is what im running.
PS: Its as if somehow the whole deal is copied through ram, its amazing how much RAM it eats.
LZ4 1.27x, LZJB 1.22x, GZIP1 1.27x, GZIP9 1.00x, ZLE 1.22x
Write speed was about the same from a Smasung 950 Pro NVME SSD to the HDD, aprox 170 Mb/S
Zou know whats curious ? When coping, i dont hear the HDD spinning. But the whole deal is eating copius amounts of ram, ive seen the ram filling from 34% to 90% and the system has 192gb of RAM.
I compiled the latest Ubuntu Kernel 5.1.9 and the latest ZFS 0.8.1, so that is what im running.
PS: Its as if somehow the whole deal is copied through ram, its amazing how much RAM it eats.
