ZFS: Extreme long seek times with RAIDZ2 and Seagate Archive

[ElGusto]

Hello!

I am using a configuration for some time now, consisting of 4 * Seagate Archive 8TB in a RaidZ2.
I am aware, that this might not be the most perfomant configuration, because of:

-RAIDZ2
-SMR in the Seagate Archive hard disks
-and the count of four hard disks (although I also have seen someone saying, that Z2 performance on four hard disks had been improved much).

But I'm ok with that, I am just using the server for archiving, and I don't need very high performance.

But what I am experiencing at the moment is close to unusable:
When I want to open a file on the server through the NFS share it takes between 30seconds and up to one minute, before the transfer actually starts! File transfer rates are ok by the way, they usually reach up to 100MB/s on large files.
The hard disks are very busy during the waiting phase, and keep rattling and working like crazy.
Same when I open directories. Scanning their contents can take up to some minutes (with some hundreds of files in there).

Writes are no problem.
So what could be the reason for this behaviour? I don't have enough knowledge for providing you possibly useful information in advance, so please help me with this, I am gonna provide all logs and whatever you need for solving this problem.

Thank you!

ElGusto

 

[Sakuru]

Please list all of the components in your FreeNAS box.
How full is your pool?

 

[mav@]

The hard disks are very busy during the waiting phase, and keep rattling and working like crazy.

This may be normal behavior for SMR drives -- when requested to do random writes, SMR drives are writing data into special non-SMR scratch zone, since random writing to SMR zone is impossible, after that, in periods of inactivity drive may reorganize data to free the scratch zone for the new writes. The only thing you may do is to let them time to finish.

Same when I open directories. Scanning their contents can take up to some minutes (with some hundreds of files in there).

Make sure that you have atime=off. It should reduce number of not very useful random writes after reads, terrible for SMR.

 

[ElGusto]

Please list all of the components in your FreeNAS box.
How full is your pool?

Hello!
Here's the fill level of the pool:

Code:

[root@freenas] ~# zpool status z2pool
  pool: z2pool
state: ONLINE
  scan: scrub repaired 0 in 5h58m with 0 errors on Wed Jun  1 07:58:37 2016
config:

    NAME                                            STATE     READ WRITE CKSUM
    z2pool                                          ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/2c5d710c-b611-814e-a225-1aa51e68a575  ONLINE       0     0     0
        gptid/cc2da5ee-733a-c148-bf15-d20be715c5f2  ONLINE       0     0     0
        gptid/45adb761-b939-284f-9019-e0ccf35e9c73  ONLINE       0     0     0
        gptid/9ce5247f-331b-3148-9b38-c958d2bd057a  ONLINE       0     0     0

errors: No known data errors

Here is some general info about my system that might be useful:

Code:

FreeNAS is running in a VM on ESXI 6.0
It's an DELL Poweredge T20, with 16GBytes of DDR3 ECC RAM.
The FreeNAS VM currently os the only guest running on this host.
The OS itselfs resides on a vmdk located on an 2,5" SSD (Crucial MX200 250GByte).
All logging is set to go to the system dataset.
The system dataset is placed on the SSD  also.
The hard disks are connected to an IBM-M1015 SAS-Controller in IT-Mode at PCIe-pass through mode using VT-d.
The controller is running fine with decent block write/read speeds in FreeNAS.
The Harddisks are 4*Seagate Archive 8TB.

FreeNAS System Information:

Code:

System Information
Hostname     freenas.lan
Build     FreeNAS-9.10-STABLE-201605240427 (64fcd8e)
Platform     Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
Memory     8162MB
System Time     Thu Jun 02 19:33:45 CEST 2016
Uptime     7:33PM up 14 mins, 1 user
Load Average     0.19, 0.31, 0.25

There are no additional add-in cards or specialities.

Thank you!

This may be normal behavior for SMR drives -- when requested to do random writes, SMR drives are writing data into special non-SMR scratch zone, since random writing to SMR zone is impossible, after that, in periods of inactivity drive may reorganize data to free the scratch zone for the new writes. The only thing you may do is to let them time to finish.


Make sure that you have atime=off. It should reduce number of not very useful random writes after reads, terrible for SMR.

Thanks. I did as you said. Making an SMR drive writing during a read seems really horrible, after having thought about it!
Are there possibly other ZFS features that could do similar?

 

[mav@]

Are there possibly other ZFS features that could do similar?

I don't know others. atime is the only popular example of writes induced by reads.

 

[Sakuru]

"zpool status" gives the layout and health of your pool, "zfs list" shows how full it is.

 

[ElGusto]

Oh, sorry, I swapped them:

Code:

[root@freenas] ~# zfs list z2pool
NAME     USED  AVAIL  REFER  MOUNTPOINT
z2pool  4.28T  9.33T  4.27T  /mnt/z2pool

Also i want to add, that some weeks ago I have changed my share type to NFS, and that it has become a little better after leaving CIFS.

I don't know others. atime is the only popular example of writes induced by reads.

I am curious about tweaks like:
-Alignment. How could I check or change this?
-And could using a secondarycache (=L2ARC?) on the SSD improve on this?
-Other useful ZFS features (maybe stronger compression? There is plenty of CPU time left!)

Or does FreeNAS do this optimal by default?

 

[mav@]

-Alignment. How could I check or change this?

FreeNAS always aligns to 4KB by default now. Not sure anything more can be done there.

-And could using a secondarycache (=L2ARC?) on the SSD improve on this?

L2ARC can help with read speed if your active dataset size is somewhat bigger then system's maximal possible/reasonable RAM size. It won't help much with write performance, which is a weak point of archive drives.

-Other useful ZFS features (maybe stronger compression? There is plenty of CPU time left!)

Default LZ4 compression is not the strongest one. If you like, you may force gzip9. But hardly the compression rate difference will be as big as performance drop.

 

[Bidule0hm]

FreeNAS always aligns to 4KB by default now. Not sure anything more can be done there.

Actually you can but not by modifying the ashift: https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/ ;)

 

[ElGusto]

After some days of usage I'm glad to report back, that disabling atime really made things work like normal. The improvement in time saving actually is about in between factors 2 and 10, depending on what operations I compared.
Thank you all for help!
:)