FreeNAS with Truecrypt encryption

[Femtoaeon]

Hello,

I have got a performance issue by using a truecrypt container in the combination with FreeNAS. I would appreciate help to resolving and to understand the problem.

Test System:
Mainboard: Asus Z87-Plus
CPU: Intel Core i5-4430 4x 3.00GHz
RAM: 32 GB DDR3-1600 MHz
System drive: SSD 60GB 360/550 Flare SA3 PAT
RAID drives: 7 x WD 3TB
LAN: 1 Gbit/s

Every Hard Disk is put to RAID-Z2 configuration. I am aware that no ECC RAM is used in the test system. This will be changed later.


Test 1:
Read and write of 3.78 GB movies from a notebook over Ethernet.
Drive in the notebook: Samsung SSD 850 EVO M.2
Write: 110 MB/s
Read: 110 MB/s

Data rates as anticipated.


Test 2:
A Truecrypt container with a capacity of 20 GB is created on the RAID-Z2. Compression and atime is deactivated. Recordsize is the standard value of 128K. The container is mounted with a notebook. CPU @ notebook: Intel Core i7-6700HQ @ 2.6 GHz (with hardware acceleration). Copying the 3.78 GB data from SDD to container.

At first 600 MB/s are shown. Later the data rate sinks slowly to 2.5 MB/s.

Approximately 30 Minutes copy duration for 3.78 GB is not acceptable.


Test 3:
Test 2 is repeated with an recordsize of 512. There was no significant improvement.

Test 4:
If a QNAP System is used with RAID 6 and 5 x 1 TB the data rates for reading and writing of the container were bigger than 50 MB/s.


Guess:
As far as I am informed ZFS does not use copy on write (COW) but rather redirect on write (ROW). This means that new data as alway written to a new location the the disk which results in a fragmentation of the Container which could probably be the explanation of the performance drop. If this guess is right then I do not understand why the performance drop happens at the first write process.

While writing the “Disk Busy” status was between 40 % and 60 %.

Hopefully someone can help.

 

[garm]

Don’t use truecrypt.. it’s broken, unsupported and discontinued..


https://en.wikipedia.org/wiki/TrueCrypt

 

[Femtoaeon]

Hello garm,

thanks for the reply.
Even when I use VeraCrypt as alternative to TrueCrypt the same behaviour is shown.

 

[garm]

What about swapping and ARC size? I might be wrong on this but, When you add content to a veracrypt container you actually change the entire container as you go, there is no way of knowing what block stores content and what is blank. So you need to load the whole thing into memory on the machine that mount the container and send the whole thing as changes happens. I’m guessing this quickly fills up your ARC and the system starts swapping, bringing your system to a crawl.

The good news is that you should be able so see this happen in the reports. Dealing with big files like this I would read up on the requirements for iSCSI, as I suspect the workload is similar.

Ps. Well actually I might be totally off on the workload thing.. but I still suspect ARC

 

[Femtoaeon]

Again thanks for the reply.

I did the test shortly after turning on the system. At first the arc size was approximately 0.
After starting the copy process the arc size increased in maximum up to 8 GB. It did not rise above. So normally the RAM should be enough or the arc is somehow not used. Since the container has only 20 GB the complete container should fit inside the arc.
At first I had only 16 GB RAM. Then I was told that the RAM could be to less and I upgraded to 32 GB.
I posted here because even with 32 GB RAM the behavior did not change.

Are you sure, that the entire container will change?
I worked with container up to a few TB in size. The latency was minmal.
I would suspect that the container is divided up in small encrypted blocks. If data is written only the corresponding block should be changed. So only the corresponding block have to be read or written. But here I am not sure. It was hard to find information about the encryption process.

How can i access the reports?

I will read a bit about iSCSI.

 

[garm]

No your right, veracrypt will encrypt block by block. Something else is going on

 

[Femtoaeon]

As I read about iSCSI I found out that iSCSI works block-based while SMB works file-based.
Some sources explained that if I want an specific data block read and I use SMB then the whole data until the wanted data block is read and transferred. This could explain the low performance with SMB.
With iSCSI only the wanted data block has to be transfered.

So as a test I configure an SCSI of 50 GB.
The SCSI was integrated to my laptop.
Here write rate of 50 MB/s are possible.
The Truecrypt-container was then placed inside the SCSI-Volume.
The Truecryp-container was mounted.
If I now read towards the Truecryp-Container the same behavour as before is seen.

At first 600 MB/s are shown as transfer rate. Later the data rate sinks slowly to 4 MB/s.

 

[garm]

Well the 600 MBps is not possible on a 1 GbE, you proabobly see windows writing to memory. Then it starts flushing memory to the container on the NAS and that is insanely slow for what ever reason. Do you get the same behavior with a container stored locally?

 

[Femtoaeon]

This seems reasonable.
At 1 GbE the datarate should be limited to 125 MB/s.
Which is more or less the datarate reachable without using the encrypted container.
Even if there is cache at the server the connection is not faster than 125 MB/s.

So the 600 MB/s must be the datarate towards the cache of the notebook.

I copied my test file towards an 20 GB container on my SDD. So i copied from SDD to container at SDD.
The datarate also starts with 600 MB/s. Then the datarate sinks down to an range between 200 MB/s and 300 MB/s.

This cache at the notebook explains why the datarate starts so high.

Any more ideas what could be the reason for the slow copy process from the cache towards the container on the NAS?

 

[Nick2253]

I'm not sure this is the solution that you are looking for, but you might be better off just creating an iSCSI zvol, and populating it with an encrypted file system. For example, on Windows you could use EFS with NTFS.

 

[joeschmuck]

I'm not sure this is the solution that you are looking for, but you might be better off just creating an iSCSI zvol, and populating it with an encrypted file system. For example, on Windows you could use EFS with NTFS.

I like this, I may try this myself just to see how well it works.