massive performance degration with encryption enabled

[Byolock]

I've noticed slow write speeds and high CPU load from the process "z_wr_iss" while writing larger amounts of data to disk. Testing around a few things I think encryption is to blame.
I created a unencrypted and encrypted dataset, both with lz4 compression and otherwise exactly the same options. I executed
dd if=/dev/random of=./ddtest bs=1M count=2048 oflag=direct on both. The results :

Encrypted Dataset : 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 13.4187 s, 160 MB/s
Unencrypted Dataset : 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 6.64577 s, 323 MB/s

I don't think 160MB/s is the expected maximum I should be getting with this CPU?

I tried to check if AES-NI is enabled using this article : https://www.cyberciti.biz/faq/how-to-find-out-aes-ni-advanced-encryption-enabled-on-linux-system/ but got mixed results. All commands and the values returned :

Show : AES-NI Tests

cpuid | grep -i aes | sort | uniq returned :
AES instruction = true VAES instructions = false

The command dmesg | grep AES returned nothing.

The command sort -u /proc/crypto | grep module returned :
module : aesni_intel+ module : crc32_pclmul module : crc32c_generic module : crc32c_intel module : crct10dif_generic module : crct10dif_pclmul module : cryptd module : essiv module : ghash_clmulni_intel module : kernel

The command openssl engine returned :
(rdrand) Intel RDRAND engine (dynamic) Dynamic engine loading support

I watched perf top while executing dd and it this is what it shows :

Code:

  23.01%  [kernel]                                 [k] gcm_pclmulqdq_mul
  10.88%  [kernel]                                 [k] kfpu_end
   8.47%  [kernel]                                 [k] kfpu_begin
   7.06%  [kernel]                                 [k] chacha_permute
   6.49%  [kernel]                                 [k] aes_xor_block
   5.22%  [kernel]                                 [k] aes_encrypt_intel
   3.37%  [kernel]                                 [k] gcm_mode_encrypt_contiguous_blocks
   3.29%  [kernel]                                 [k] aes_encrypt_block
   2.89%  [kernel]                                 [k] aes_aesni_encrypt
   2.80%  [kernel]                                 [k] aes_copy_block
   2.77%  [kernel]                                 [k] gcm_mul_pclmulqdq
   2.29%  [kernel]                                 [k] __x86_retpoline_rax
   1.27%  [kernel]                                 [k] crypto_get_ptrs
   1.16%  [kernel]                                 [k] copy_user_enhanced_fast_string
   0.74%  [kernel]                                 [k] chacha_block_generic
   0.63%  [kernel]                                 [k] _copy_to_iter

Do these results mean AES-NI is used or not? If it is used, what could be the bottleneck in my System (Hardware is in Signature). Software Version is TrueNAS-SCALE-22.02.3. If any other Information is required I'll gladly provide any details.

 

[anodos]

[NAS-115648] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Perhaps this applies to your particular processor. We just froze for BETA2 and so it's not available yet in BETA train, but it should be available in nightlies if you want to try the crazy train.