7zip under FreeNAS/FreeBSD

[Mark Holtz]

Stupid question time... how do I install the CLI version of 7zip onto my personal FreeNAS server so that I can create/extract 7z archives directly on the server through the shell? I have been doing it from my Windows box, but then I'm reading the files from a Windows share, compressing them, and then sending the compressed archive back to the server consuming network traffic. Why shouldn't I be able to do this directly on the server itself?

Having used Linux several years ago, there was a command such as aptget that would download and install the package onto the server. Like I said, this is several years ago and a different OS, but I'm sure FreeBSD has something similar.

 

[garm]

You can’t install anything on FreeNAS in practice. What you need to do is to spin up a jail and install p7zip from ports.

But your workflow sounds flawed. If you intend to archive content on the server using CLI, why not use tar?

 

[Mark Holtz]

If you intend to archive content on the server using CLI, why not use tar?

I'm on a Windows machine, and have adjusted the default save locations or used symbolic links to save material on my FreeNAS server using SMB shares rather than on my local computer. Thus, zip/7z archives is preferred over TAR archives, especially when you are trying to archive GBs of saved data.

 

[garm]

Still don’t see your point. Windows isn’t part of the equation when you want to do it on FreeNAS. Both winzip and 7z will handle *.tar.xz or *.tar.gz just fine. “Trust me, I’m an engineer” , I actually do this for a living.

 

[Chris Moore]

Still don’t see your point. Windows isn’t part of the equation when you want to do it on FreeNAS. Both winzip and 7z will handle *.tar.xz or *.tar.gz just fine. “Trust me, I’m an engineer” , I actually do this for a living.

Just a note of extra information, I found that extracting archives directly in the server was significantly faster than extracting the same archive on windows and then copy / move the data to the server. The only problem I ran into was the owner of the files.
The user that runs the extraction is the owner so you might need to chown.



Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[sretalla]

So clearly you can install 7zip in a jail with no issue:

pkg install p7zip

You may or may not be able to just copy the binary from the jail to a location on your pool (not the boot drive) and run it from there if you chmod to allow execution from the FreeNAS host (needing to specify the path and not just p7zip at the CLI).

 

[Chris Moore]

So clearly you can install 7zip in a jail with no issue:

pkg install p7zip

You may or may not be able to just copy the binary from the jail to a location on your pool (not the boot drive) and run it from there if you chmod to allow execution from the FreeNAS host (needing to specify the path and not just p7zip at the CLI).

Yes, you can create a generic jail and add software to it.
I am not sure how the new jails with iocage work, but the old jails with warden allowed a directory (or more) in the storage pool to be mounted with write permission from inside the jail. Files could be opened and extracted at the command line in the jail but the files were still in main storage.

 

[sretalla]

I am not sure how the new jails with iocage work

Iocage jails allow nullfs mounts via the iocage fstab command, so essentially works the same as warden jails.

I guess you could opt for either method, run your zip/unzip work in the jail, or see if there's a single binary that can be copied out of the jail and run from a pool location (hence will survive reboots and upgrades).

 

[Ericloewe]

Are you sure you'll benefit from this over ZFS compression?

 

[Chris Moore]

Are you sure you'll benefit from this over ZFS compression?

It might be the difference between lz4 and gzip-9 (or more) because I get almost 2 to 1 compression with gzip-9 on one of my servers at work. The lz4 compression is fast, but you can certainly get better compression with a more CPU intensive algorithm. Also, it depends on the data type that is being compressed.

 

[danb35]

It might be the difference between lz4 and gzip-9 (or more)

Of course, you can set compression on a per-dataset basis--choose an appropriate dataset for the files, set that dataset to gzip compression, put files there.

 

[Chris Moore]

I have been doing it from my Windows box, but then I'm reading the files from a Windows share, compressing them, and then sending the compressed archive back to the server consuming network traffic. Why shouldn't I be able to do this directly on the server itself?

I guess that the question is, why are you making zip files? You can use filesystem level compression on the NAS, what is the purpose of creating the file, to obtain higher compression?

 

[Ericloewe]

There's still the metadata and padding advantage of an archive, but for larger compressible files, ZFS compression is nearly as good when used correctly. Zstd looks particularly promising, once that arrives. Allan Jude reports GZIP-like compression at higher throughputs for the higher levels of zstd. Lower levels trade compression for performance. With decent CPUs, you can get GZIP-like compression without bottlenecking the disks.

 

[Mark Holtz]

I guess that the question is, why are you making zip files? You can use filesystem level compression on the NAS, what is the purpose of creating the file, to obtain higher compression?

Backups. And, since the best examples are real-world examples, I'll use one here, albeit the most extreme example I can quickly find.

Yes, I'm a gamer, and have a nice collection of games. The largest game, file-size wise, is Middle Earth: Shadow of War, which is a whopping 92.1 GB download. Go ahead, pull out your download calculator website, and figure out how long it will take to download on your Internet connection. I have a 20 Megabit connection, so it would take at least 11 hours to download. That's why I always perform a Steam game backup of the download once completed so that I can restore the game locally. Even 20 minutes is better than 11 hours.

But, Steam backups aren't that efficient, and I swear, has not changed since Steam was released in September, 2003 when Windows XP was still fairly new and FAT file systems reigned supreme. It was designed to backup to DVDs and CDs, not big storage devices like FreeNAS. If you look at a steam backup, you will notice that the largest file size is going to be around 1,058,XXX,XXX bytes for a CSD file and 36,xxx bytes for the corresponding CSM file, thus we are dealing with a multi-parted archive. Each DVD-sized "disk" has five of those file pairs, and there are 22 of those DVD-sized "discs" (not to mention a 2,804 byte sku.sis). (Slight oversimplification here, but bear with me).

Now, imagine trying to backup that share to a off-line storage. I'm backing up the share to a 8TB portable drive using FreeFileSync. FreeFileSync only backs up changed files, but it first has to determine what files are changed. Which is quicker to compare--a large 92GB archive, or 242 tiny files? Even the smallest of Steam backups still have three files: the CSD, it's corresponding CSM, and a sku.sis.

Enter 7zip. It is my compression program of choice because 7z archives are slightly smaller than ZIP archives. The compression, however, is not all that great with already-compressed data, but I'm able to squeeze out 300MB of additional compression out of my example game. The main reason, however, is to take all of those small files, and make one big archive. Slight problem.... if I'm creating a 7z (or zip for that matter) archive from a file share, that means that the file data has to be transferred from the FreeNAS server to my local computer, compressed, and sent back to the resulting 7z archive on the file share. Even on a 1 Gigabit network connection, it is still going to take a while.

That's why I'm asking about a better solution. Why not install 7zip on the FreeNAS server, log into shell, and make the archive process a local process. Will it be faster? Don't know, but I won't be flooding my personal network. The FreeNAS server can be chunking away creating the archive while I'm gaming or working. Most of the time, my FreeNAS server is sitting idle.

Both winzip and 7z will handle *.tar.xz or *.tar.gz just fine. “Trust me, I’m an engineer”, I actually do this for a living.

There are good reasons for having a tar archive over a 7z archive in a *nix world. One very good reason is file permissions. But, because I'm creating an archive from a Steam backup, file permissions do not play a role here.

And, that is not what I'm asking here. I want to be able to create 7z archives. That is what I, the customer, wants to do. “Trust me, they call me an engineer, but I also deal with end users.” That's how I earn my paycheck.

 

[Chris Moore]

That's why I'm asking about a better solution. Why not install 7zip on the FreeNAS server, log into shell, and make the archive process a local process. Will it be faster? Don't know, but I won't be flooding my personal network. The FreeNAS server can be chunking away creating the archive while I'm gaming or working. Most of the time, my FreeNAS server is sitting idle.

FreeNAS is a heavily modified version of FreeBSD, but the modifications have turned it into an "appliance" that you are not supposed to be able to install software on. You might be able to force it, but it might not survive a reboot, much less survive a version upgrade. What should work is what we were discussing earlier. Create a jail and install the software in it, share the directory where the data is to the jail by mounting the directory, then you just go to the jail to issue your 7zip commands instead of doing it in FreeNAS. Effectively, it is the same, you just don't modify the FreeNAS to do it.

 

[Chris Moore]

That is what I, the customer, wants to do.

PS. You are not a customer unless you are paying for something. Don't get an attitude.

 

[garm]

Backups. And, since the best examples are real-world examples, I'll use one here, albeit the most extreme example I can quickly find.

Yes, I'm a gamer, and have a nice collection of games. The largest game, file-size wise, is Middle Earth: Shadow of War, which is a whopping 92.1 GB download. Go ahead, pull out your download calculator website, and figure out how long it will take to download on your Internet connection. I have a 20 Megabit connection, so it would take at least 11 hours to download. That's why I always perform a Steam game backup of the download once completed so that I can restore the game locally. Even 20 minutes is better than 11 hours.

But, Steam backups aren't that efficient, and I swear, has not changed since Steam was released in September, 2003 when Windows XP was still fairly new and FAT file systems reigned supreme. It was designed to backup to DVDs and CDs, not big storage devices like FreeNAS. If you look at a steam backup, you will notice that the largest file size is going to be around 1,058,XXX,XXX bytes for a CSD file and 36,xxx bytes for the corresponding CSM file, thus we are dealing with a multi-parted archive. Each DVD-sized "disk" has five of those file pairs, and there are 22 of those DVD-sized "discs" (not to mention a 2,804 byte sku.sis). (Slight oversimplification here, but bear with me).

Now, imagine trying to backup that share to a off-line storage. I'm backing up the share to a 8TB portable drive using FreeFileSync. FreeFileSync only backs up changed files, but it first has to determine what files are changed. Which is quicker to compare--a large 92GB archive, or 242 tiny files? Even the smallest of Steam backups still have three files: the CSD, it's corresponding CSM, and a sku.sis.

Enter 7zip. It is my compression program of choice because 7z archives are slightly smaller than ZIP archives. The compression, however, is not all that great with already-compressed data, but I'm able to squeeze out 300MB of additional compression out of my example game. The main reason, however, is to take all of those small files, and make one big archive. Slight problem.... if I'm creating a 7z (or zip for that matter) archive from a file share, that means that the file data has to be transferred from the FreeNAS server to my local computer, compressed, and sent back to the resulting 7z archive on the file share. Even on a 1 Gigabit network connection, it is still going to take a while.

That's why I'm asking about a better solution. Why not install 7zip on the FreeNAS server, log into shell, and make the archive process a local process. Will it be faster? Don't know, but I won't be flooding my personal network. The FreeNAS server can be chunking away creating the archive while I'm gaming or working. Most of the time, my FreeNAS server is sitting idle.



There are good reasons for having a tar archive over a 7z archive in a *nix world. One very good reason is file permissions. But, because I'm creating an archive from a Steam backup, file permissions do not play a role here.

And, that is not what I'm asking here. I want to be able to create 7z archives. That is what I, the customer, wants to do. “Trust me, they call me an engineer, but I also deal with end users.” That's how I earn my paycheck.

Myself and others already gave you the desired solution.. and less crappy ones..

If your desire to use 7z over native tar you need to set up a jail..

If you don’t want a jail you can use tar.. 7z brings nothing of value to the process.

Your backup solution is based on smb shares? If so you could proabobly make your life a lot easier by setting up a jail doing the off site backups for you. That would short circuit the need of 7z and “flooding your network”.

(1) Have steam write files to a smb share (2) make a snapshot of the share (3) mount the snapshot to the backup jail (4) make a deduplicated archive of the snapshot (5) and send off to cold storage.

Or better yet, have the off site be ZFS and just zfs send to that location. Then there is no need to alter the files in any way, which is always the best backup scenario.