lz4 compression and replication

[elangley]

If I have a PUSH replication volume that is compressed should the PULL volume be setup compressed as well? If it is would this result in "double" compression?

 

[eraser]

I don't use replication myself, but my "gut instinct" is that data is transparently uncompressed from the source dataset and then sent over the network to the destination dataset. If you want to compress the data written to the destination dataset you should enable compression on the destination dataset.

Hopefully someone else will chime in and confirm if this is true.

 

[Dusan]

The replication stream actually retains the compression (i.e. zfs send of a compressed dataset/snapshot will generate a compressed replication stream).
Replicating from an uncompressed to uncompressed dataset or compressed to compressed dataset is simple -- nothing changes (and there is no "double" compression).
When you replicate from an uncompressed dataset to a compressed dataset the stream will be uncompressed, but the data will be stored compressed on the destination (the receive will compress it based on the destination dataset setting).
And now the interesting case. Replicating from a compressed dataset to an uncompressed dataset will send a compressed stream and it will also be stored compressed on the destination (i.e. it will not be uncompressed by the receive operation). ZFS doesn't have a problem with this -- there can be both compressed and uncompressed blocks in the same filesystem.

 

[jgreco]

... as long as the destination supports the compression that was used. I assume. Might be interesting to try...

 

[Dusan]

... as long as the destination supports the compression that was used. I assume. Might be interesting to try...

I think the receive will fail in that case. The send doesn't know anything about the destination. It just "blindly" creates the stream. The receive has no channel to communicate with the send, so it doesn't have any other option than to fail when it does not "understand" the stream.

 

[jgreco]

Assuming it is performing validation on the data blocks. Haven't looked at the code and am on a cell phone so at a disadvantage.

 

[Dusan]

Too late (1am here) to try it, but not too late to check the source code :). zfs receive will fail with "pool must be upgraded to receive this stream" (EZFS_BADVERSION) when you try to receive a stream originating from a newer ZFS version (or if feature flags are in use the receiving pool must support all active features that are present in the stream).

 

[cyberjock]

So can I assume that dedup will follow the same path?

 

[Dusan]

Dedup is a different beast. The stream is normally always "non-deduped", even when you are replicating between two deduplicated pools. There is a zfs send option (-D) to generate a deduplicated stream:

-D Generate a deduplicated stream. Blocks which would have been sent multiple times in the send stream will only be sent once. The receiving system must also support this feature to receive a deduplicated stream. This flag can be used regardless of the dataset's dedup property, but performance will be much better if the filesystem uses a dedup-capable checksum (eg. sha256).

This option controls the deduplication of the stream and allows you to even send a deduplicated stream between two non-dedup pools. The deduplication of the data on destination is always controlled by the destination pool setting and it is independent of the stream deduplication.
Non-dedup -> dedup replication will result in the data being deduplicated on the destination.
Dedup -> non-dedup replication will result in the data not being deduplicated on the destination (even if you use zfs send -D as that only controls deduplication of the stream).

 

[elangley]

Fascinating.

Here are my results from replicating a vmware data set that was configured PUSH = Compressed and PULL = Compressed (over the WAN 2000 miles).

It went through fine and the data shows compressed on the PULL side.

The original data was set was 17GB so pretty good space savings from compression at 35%.

noprobs noted in this thread that he has done some custom development on FreeNAS relating to compression. Not sure what it is though since it already works for me.

 

[noprobs]

Just reading this post and I had a slightly different understanding

When you "zfs send" a dataset which is compressed the send stream is not decompressed. For that reason I have local code on my system to compress (gzip/xz/lzma option) see also https://bugs.freenas.org/issues/2227. At destination end whether the dataset is compressed on disk depends on compression setting for that dataset (on remote system)

NB Oracle have a command (zfs receive -o) to change properties on replicate set, eg you could force compression property to be different on remote (vs local) system. This has not yet been carried to Illumos https://www.illumos.org/issues/3753

 

[Dusan]

I guess he compresses the replication stream independently of ZFS compression. Simplifying this a bit, the replication works like this: zfs send -> ssh tunnel -> zfs receive. You can compress the stream by doing: zfs send -> gzip -> ssh tunnel -> gzip -d -> zfs receive.

 

[Dusan]

It maybe worked differently before, but the stream is compressed in the current ZFS version (a 42k stream to transfer a compressed snapshot containing 100MB of zeros):

Code:

[root@freenas] /mnt/tank# zfs create -o compression=lz4 tank/test
[root@freenas] /mnt/tank# dd if=/dev/zero of=test/testfile bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.053930 secs (1944326240 bytes/sec)
[root@freenas] /mnt/tank# zfs snapshot tank/test@snap
[root@freenas] /mnt/tank# zfs send tank/test@snap | wc -c
  43496

 

[noprobs]

Elangby, if both sides dataset have compression enabled then I would expect the net result to be savings on both side; however were you able to monitor how much actual data was sent over the WAN?

My local code is indeed zfs send > gzip/xz/lzma > throttle > ssh > gzcat/xzcat/lzcat > zfs receive.

I wanted to have lz4 option as well but lz4 executable don't appear to be in FreeNAS

 

[noprobs]

Let me check/redo some of my earlier testing

 

[noprobs]

Using some real world data - mainly a mix of office files

NAME PROPERTY VALUE SOURCE
SlowVol/Work_Archive compressratio 1.39x -
SlowVol/Work_Archive recordsize 128K default
SlowVol/Work_Archive compression lz4 local
SlowVol/Work_Archive usedbysnapshots 1.83M -
SlowVol/Work_Archive usedbydataset 3.18G -
SlowVol/Work_Archive usedbychildren 0 -
SlowVol/Work_Archive dedup off default
SlowVol/Work_Archive logicalused 4.26G -

zfs send > file; zfs send | gzip >file; zfs send | lzma >file; zfs send | xz >file
4.3G Jan 25 12:55 auto-20140119.0900-4w.snaps
2.8G Jan 25 12:59 auto-20140119.0900-4w.snaps.gz
2.4G Jan 25 14:09 auto-20140119.0900-4w.snaps.lz Too slow for me
2.4G Jan 25 14:09 auto-20140119.0900-4w.snaps.xz Too slow for me


I therefore use gzip to compress replication streams which in my set-up has significant benefits as my WAN link has limited bandwidth

 

[John Doe]

Hi all,

I am not able to do a replication task from freenas 9.3 to any OpenZFS / ZOL if I enable lz4 compression. (works "fine" between freenas)

Replication of anydatast@auto-foo failed with /usr/local/bin/lz4c: Command not found. cannot receive: failed to read from stream Error 33 : Write error : cannot write compressed block

I was wondering out why zfs receive was calling lz4c not present at that path in any OSX/Linux box. Like Dusan explains a compressed dataset is always sent compressed.

For this reason I did some test anticipated by Dusan:

1) send from v5000, v5 feature@lz4_compress enabled to v5000, v5 feature@lz4_compress disabled is sent compressed and stored compressed

2) send from v5000, v5 feature@lz4_compress enabled to v28, v5 I can still see the feature@lz4_compress, and set it to disabled but it works and is stored compressed! [ zpool create -o version=28 in an updated box]. Strange since v28 afaik shall not have lz4 flag

NAME PROPERTY VALUE SOURCE
delme version - default
delme2 version 28 local
bash-3.2# zpool get all | grep feature
delme feature@async_destroy enabled local
delme feature@empty_bpobj active local
delme feature@lz4_compress active local
delme2 feature@async_destroy disabled local
delme2 feature@empty_bpobj disabled local
delme2 feature@lz4_compress disabled local
​

3) send from v5000, v5 feature@lz4_compress enabled to v11 pool, v3 ZFS finally gives me error ;)

bash-3.2# zfs send delme@snap | zfs receive delme2/sega
cannot receive new filesystem stream: pool must be upgraded to receive this stream.
warning: cannot send 'delme@snap': Broken pipe
bash-3.2# zfs get all | grep version
delme version 5 -
delme@snap version 5 -
delme2 version 3 -
bash-3.2# zpool get all | grep version
delme version - default
delme2 version 11 local
​

4) creating a v26 pool makes a v5 ZFS but inside it i can do a dataset with zfs create -o version=4 pool/dataset and it supports lz4 comrpession receive! So I suppose test #3 fails not for the feature flag missing and is not a problem of ZFS version.
I am not able to create directly a v26 (for example) pool with a v4 zfs but I can create a v5 zfs with a older version dataset (any command to do it direvlty)?

I need to make some test with original solaris box or similar to understaind compatibility constraints because of feature flags. I SUPPOSE that if I create a v28 pool in freenas I can send/import it on solaris ONLY IF no lz4 stream is ever received! So a send receive operation might break some compatibility configured during pools creation (v26/v28 pools with no feature can receive comrpessed info and store it compressed as Dusan says.). This might be a problem for someone.

Moreover I am sending in this test huge files filled with zeros that are actually compressed 10:1 or 100:1 but compression and refcompression keeps indicating 1,0% or 1,15%

I am still investigating why lz4c is called by middleware. It shall be scripted in a pipe since I can't see any call to it in zpool history at booth ends.

[I suppose zfs send foo@bar | lz4c | ssh lz4c -d | zfs receive foo/bar]
If that is true that would mean that double compression is done with vaste of resource?

Just to make this post even more boring, I am also having following problem: First send/receive between freenas NEVER works if you flag "Recursively replicate and remove stale snapshot on remote side:"
I must wait first time to get done and after I have to remember to flag it. very annoying. Maybe a bug?

Ciao!

Alessandro

 

[noprobs]

Alessandro,

If you enable lz4 (pigz or plzip) for sending compression stream that it will fail to send if receiving server does not have compression binaries in /usr/local/bin/. FreeNAS does have those binaries your Linux or Solaris server appears not to.

Note this functionality is totally separate to the storage of data in a zfs dataset which can be compressed with lz4 (v5000 feature flag), gzip, zle or lzjb. NB By default when a zfs stream from a compressed dataset it will be received into a compressed dataset.

From memory data in a compressed dataset is sent as uncompressed data stream and then received into a compressed dataset. Let me go back to some test data.

Jon

 

[John Doe]

Jon,

Sure for the bianries,
are u saying that in case the dataset is already compressed the compression flag in the gui shall be disabled not to generate useless overhead?

Please, if you can, test what I was saying: an "compatibility" v28 dataset loose the possibility of being mounted by Solaris( for example) if a compressed data stream is received since it would enable feature flags implicitly (sig)

Alex