TrueNAS 13 way slower than 12

[MikeyG]

As I understand it, cp was tuned in 13 to have lower CPU, with the result of the lower copy performance you see. You should submit a bug or feature request to upstream FreeBSD, so that the end user has the choice of tuning cp for either lower CPU or greater performance.

As for the server side copy, I'm not sure it was tested against copies between datasets, so this may be an OpenZFS 2.1 (in 13) regression vs. 2.0 (in 12). Try asking around on the OpenZFS developer channels:

Developer resources - OpenZFS

openzfs.org

Ok, I started looking into contacting the OpenZFS developers, but I'm having some trouble figuring that out, and I get the feeling I'm going to run into both limits of my own technical knowledge and time to pursue this with them. Wouldn't something like this be an issue that IX would care enough about to look into though? Aren't they selling some high end all flash systems that would experience this problem? Copying files between datasets shouldn't be such an obscure thing to be doing where a big performance hit wouldn't matter.

 

[MikeyG]

In case anyone runs into this or wanted to know, I brought it up with the OpenZFS developers. They were able to replicate the problem very quickly. Far as I can tell, here is the issue and fix: https://reviews.freebsd.org/D36194

Is there any way to tell how long this might take to make it into a release of TrueNAS? I'm not familiar with how this stuff works.

 

[malco_2001]

In case anyone runs into this or wanted to know, I brought it up with the OpenZFS developers. They were able to replicate the problem very quickly. Far as I can tell, here is the issue and fix: https://reviews.freebsd.org/D36194

Is there any way to tell how long this might take to make it into a release of TrueNAS? I'm not familiar with how this stuff works.

We have an upcoming 13.0-U2 release scheduled for August 30. The lead of our OS team is a reviewer on this at FreeBSD. I will sync with him to try to get this included and report back.

 

[malco_2001]

I have confirmed this will be included in 13.0-U2.

 

[MikeyG]

I have confirmed this will be included in 13.0-U2.

Excellent! Thank you for such a swift response. I look forward to testing it.

 

[MikeyG]

I just installed 13.0-U2. Happy to report some improvement on this. Transfers via disk are back to about 1GBps and transfers from ARC at about 2GBps. Still slightly slower than it was on 12.0-U8 but I'm grateful to get back most of the speed I had.