Encrypted ZFS pool made with the GUI a lot slower than manually created ones

[flami]

Hi there,

I just tried out the Freenas 8.3.1 Beta3 and build an encrypted ZFS pool . I selected "Force 4096 bytes sector size" "Enable full disk encryption" "Raid-z".
I then tested its thoughput by doing :

Code:

dd if=/dev/zero of=testfile bs=1M count=30000
30000+0 records in
30000+0 records out
31457280000 bytes transferred in 500.918654 secs (62799179 bytes/sec)

as you can see I got around 60 MB/s

how ever if I create the ZFS array myself in the console like this :

Code:

geli init -s 4096 ada0
# etc
geli attach ada0
# etc
zpool create bla raidz ada0.eli ada1.eli ada2.eli ada3.eli ada4.eli ada5.eli
zfs set mountpoint=/mnt/bla bla
zfs mount -a

I get this

Code:

dd if=/dev/zero of=testfile bs=1M count=30000
30000+0 records in
30000+0 records out
31457280000 bytes transferred in 128.580592 secs (244650297 bytes/sec)

So around 240 MB/s
So it is a lot faster.
This is also true for a single drive pool, where I get around 140 MB/s when created manually and 80 when created via the GUI.

Is there something wrong perhaps with how the GUI creates the pool at the moment? Am I missing something ?
I don't know if I can use my manually created pool with the GUI since there is obviously something very different.

 

[flami]

Alright I found out what happens:

When you tick "force 4k.." a variable called force4khack will get set.
This is done in the gui/storage/forms.py file
this is then passed to gui/middleware/notifier.py and eventually calls

Code:

def __prepare_zfs_vdev(self, disks, swapsize, force4khack, encrypt, volume):

in there is a test

Code:

  if encrypt:
            test4k = False
            force4khack = False
            want4khack = False

Even if this test would not be there, the function that calls geli to encrypt the disks is:

Code:

  def __encrypt_device(self, devname, diskname, volume, passphrase=None):

So it does not know about the 4k stuff.

I changed this around a bit to pass the force4khack flag around, however this causes a problem with.

Code:

 def zfs_replace_disk(self, volume, from_label, to_disk, passphrase=None):

That apparently does not know if a volume was created with the 4khack or not. Thus it cannot pass it along to geli when it calls :

Code:

 devname = self.__encrypt_device("gptid/%s" % uuid[0].content, to_disk, volume, passphrase=passphrase)

Is there a way for zfs_replace_disk to figure out if the volume was created with 4k sectors or not? Maybe this could be checked in the forms.py ? The performance with 512 byte sectors on the geli device is pretty bad. I made a pool without the "force 4k" flag and I got this:

Code:

dd if=/dev/zero of=testfile bs=1M count=30000
30000+0 records in
30000+0 records out
31457280000 bytes transferred in 561.772208 secs (55996505 bytes/sec)

Edit:
I made a bugreport:
https://support.freenas.org/ticket/2035

 

[cyberjock]

When I started playing with encryption I noticed the same thing but I'm not experienced enough to dig into the middleware like you did. Congrats on your find. You should create a ticket at support.freenas.org and include this info so it can hopefully be fixed before 8.3.1 is finalized.


Personally, if I were a FN developer I'd say just force everything to 4k and be done with it. 4k is the future and I can't imagine forcing 4k for all devices really will affect non-4k devices.

 

[survive]

Hi guys,

Just chiming in with my agreement with cyberjock.

Even if you don't need 4k today, with your existing drives I think the force 4k option is important to future-proof the system for a bigger RMA replacement drive or when you want to swap out those old small drives for whatever whoppers are available at the time....either way you will be wishing that 4k option was the default.

-Will

 

[flami]

I think the other big issue is that the zfs_replace_disk function does not seem to check for 4k at all. It seems to me that this needs to be passed along. If I saw that correctly we save whether or not a volume is 4k in the database. But I only skimmed over the code.

 

[cyberjock]

The 4k setting is pool wide. If your other disks have 4k set then any disks added WILL be 4k automatically.

 

[flami]

I think the ZFS file system on top of the eli drive will probably be 4k , I guess "/sbin/zpool replace..." handles that.
It is however before where it creates the eli drive where it does not check for the 4k stuff.

I am trying to find the cleanest way to figure out whether or not a volume/pool (seems that according to the GUI's database its all the same) is on top of 4k drives. It is not saved in the storage_volume table. it only knows the FS and about encryption ( also name etc ).

zfs_replace_disk and zfs_volume_attach_group seem to be the only ones that eventually will call __encrypt_device , where only zfs_volume_attach_group knows about 4k sectors being used.

the easiest would probably be to save in the database if a volume has been created with the 4k option.


edit :
More testing. Seems even though I ticked Force 4k , my zpool is still 512 :

Code:

zdb -C teast

MOS Configuration:
[snip]     
                ashift: 9
[snip]

the manually created one is 4k:

Code:

zdb -C bla

MOS Configuration:
[snip]     
                ashift: 12
[snip]

So I guess I can check the pool sector size with zdb, which can fail if there's no zpool.cache file.

I think I'll whip together a patch.

 

[flami]

I created a small patch to create 4k geli devices. I added it here: https://support.freenas.org/ticket/2035
I hope that helps a bit.

Also I noticed that in zfs_replace_disk we do not call random_wipe() like we do in __create_zfs_volume .
So it seems that a replaced disk is not wiped with random data. random_wipe() is only called once in the entire file which is in __create_zfs_volume. This might make the entire random wipe useless as soon as you add or replace a drive. I might be wrong though, I have not investigated that thoroughly .

 

[rickh925]

Flami, are you running with your patch? This is what I get encrypted then not encrypted. Commas added for readability. I currently have one gigabit interface but am going to try some link aggregation in the future so this isn't going to be a big deal for now as I should be able to saturate the single network link for now.

Rick

Code:

[root@freenas /mnt]# ls -al                                                     
total 37                                                                        
drwxr-xr-x   4 root  wheel     512 Feb 18 10:29 .                               
drwxr-xr-x  20 root  wheel     512 Feb  4 10:53 ..                              
drwxrwxr-x   2 root  operator  512 Feb 18 08:38 .snap                           
drwxr-xr-x   9 root  wheel       9 Feb 18 11:29 BabVol01                        
-rw-r--r--   1 root  wheel       5 Feb  4 10:51 md_size                         
[root@freenas /mnt]# cd BabVol01/                                               
[root@freenas /mnt/BabVol01]# dd if=/dev/zero of=testfile bs=1M count=30000     
30000+0 records in                                                              
30000+0 records out                                                             
31457280000 bytes transferred in 101.225920 secs (310,763,093 bytes/sec)          
[root@freenas /mnt/BabVol01]# 


[root@freenas /mnt]# cd TestEnc/                                                
[root@freenas /mnt/TestEnc]# ls -al                                             
total 6                                                                         
drwxr-xr-x  2 root  wheel    2 Feb 18 12:29 .                                   
drwxr-xr-x  4 root  wheel  512 Feb 18 12:29 ..                                  
[root@freenas /mnt/TestEnc]# dd if=/dev/zero of=testfile bs=1M count=30000      
30000+0 records in                                                              
30000+0 records out                                                             
31457280000 bytes transferred in 199.672666 secs (157,544,248 bytes/sec)          
[root@freenas /mnt/TestEnc]#  

 

[flami]

the patch has been cleaned up and has been merged here :
https://support.freenas.org/changeset/13285

( there's an error in the ticket where it says 13286 )

And yup I run the patch, I had to test it somewhere ;).
With my i5 that has AES-NI, plus 6 5400rpm drives, I get around 250 MB/s encrypted compared to, if I remember correctly, around 400 MB/s without encryption. You probably want a CPU with AES acceleration, I believe there is only one sandy-bridge i3 that has AES-NI , all i5, i7 and xeons E-? have it. No idea about AMD .

 

[cyberjock]

Thank you flami for the patch. I did some testing with encryption in Dec and the performance was far below my expectations with an AES-NI CPU. My guess is that the issue I had(and the reason I'm not using encryption now) is the bug you found and fixed. :)

 

[darkryoushii]

I'm being really hopeful here, but was this merged into 8.3.1 RC1 ? I just created a pool for a 2nd time (first time I factory restored before saving the key file) and moved my 3.4TB across for the second time.. really don't want to have to do that all over again.. again.

 

[cyberjock]

It was added in change 13825 on Feb 13. Unless you are using a nightly build this fix hasn't been implemented in any official public beta(the latest as of right now is BETA3). Note that there has been no "release candidate" for 8.3.1.

 

[darkryoushii]

http://sourceforge.net/projects/freenas/files/FreeNAS-8.3.1/RC1/

Then what is this? Since that's what I'm running at the moment :S

 

[cyberjock]

That hasn't been officially announced...yet. Basically its there for a few developers to see if there's any killer bugs before announcing it officially. It has been known that RCs were posted(and then bugs fixed and the RC version changed) before officially being announced. Personally, I consider running unannounced versions even scarier than dailies because multiple compiles can be labeled as "RC1" before its actually released. If I remember correctly 8.3.0 RELEASE was out and then revised before it was officially announced.

I started a thread somewhere here back when 8.3.0 was released because it had been posted for several days with no official word of its release. 8.3.0 was a big deal for me because I wanted zpool v28.

 

[darkryoushii]

Ah I see, that makes sense then. Is it possible for me to know if the patch has been applied to this build though? I have only been running it for 24 hours and there doesn't seem to be any changes made to the files/recompiles as far as I can tell.

 

[William Grzybowski]

Your patch has not been committed, a simpler fix was made to use 4096 sector size for encrypted volumes.

To confirm check with: geli dump gpt/********* | grep sector