Feedback on performance with encrypted datasets

[kriegalex]

I've repurposed my firewall mini-ITX board as a TrueNAS. It was aimed at over gigabit speeds with VPN traffic, so it is using an Atom C3558 that supports AES-NI.

While installing gitlab-ce, I've noticed they give some command to test IOPS using fio :

Code:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs=4k --iodepth=64 --readwrite=randrw --rwmixread=75 --size=4G --filename=/path/to/git-data/testfile

Out of curiosity, I tried creating an encrypted dataset. I was expecting it to perform not too bad with encrypted dataset thanks to AES-NI. How boy was I wrong:

non-encrypted results on 2x4TB HGST 7200rpm SATA3:

read: IOPS=5763, BW=22.5MiB/s (23.6MB/s)(3070MiB/136370msec)
bw ( KiB/s): min= 4817, max=57880, per=99.97%, avg=23045.83, stdev=6926.41, samples=272
iops : min= 1204, max=14470, avg=5761.42, stdev=1731.59, samples=272
write: IOPS=1926, BW=7704KiB/s (7889kB/s)(1026MiB/136370msec); 0 zone resets
bw ( KiB/s): min= 1651, max=20184, per=99.97%, avg=7702.56, stdev=2287.62, samples=272
iops : min= 412, max= 5046, avg=1925.61, stdev=571.90, samples=272
cpu : usr=2.95%, sys=30.55%, ctx=257769, majf=0, minf=6

encrypted results:

read: IOPS=658, BW=2635KiB/s (2699kB/s)(3070MiB/1192897msec)
bw ( KiB/s): min= 175, max=16111, per=100.00%, avg=2637.99, stdev=1025.20, samples=2288
iops : min= 43, max= 4027, avg=659.37, stdev=256.31, samples=2288
write: IOPS=220, BW=881KiB/s (902kB/s)(1026MiB/1192897msec); 0 zone resets
bw ( KiB/s): min= 65, max= 5333, per=100.00%, avg=881.63, stdev=347.35, samples=2288
iops : min= 16, max= 1333, avg=220.28, stdev=86.84, samples=2288
cpu : usr=0.34%, sys=85.28%, ctx=142389, majf=0, minf=6

Are those results expected ?

 

[mav@]

In your test you've specified block size of 4KB, while the dataset I guess uses default blocks of 128KB. Depending on several factors it can make ZFS to decrypt/decompress/re-compress/re-encrypt 32 times more data than application (fio) actually writes. So if your goal was to burn as much CPU as you could -- you've succeeded. ;)

 

[kriegalex]

That's what happens I guess when someone uses a command "out of curiosity" without fully understanding how it works behind the scenes :)

 

[MrGuvernment]

Yup, encryption for different tasks can take a hell of a lot of resources!