Optimise for lots of small files?

Status
Not open for further replies.

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
One of my pools has 6 disks in RaidZ1 and can push 540 MB/s write and 600 MB/s read.

I just got everything hooked up to 10Gbe using 2 Chelsio cards. I can saturate the local Raid array when writing large sequential files to the NAS.

However when dealing with lots of RAW photos of 2-3 MB each I only get 80 MB/s.

Is there a way to optimize a Zpool for this kind of usage? Like adjust the block size or something? Or is there a complete other approach to deal with this?

Fyi. Don't have the budget the use lots of SSD's :smile:

Cheers!
K.
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
One of my pools has 6 disks in RaidZ1 and can push 540 MB/s write and 600 MB/s read.

I just got everything hooked up to 10Gbe using 2 Chelsio cards. I can saturate the local Raid array when writing large sequential files to the NAS.

However when dealing with lots of RAW photos of 2-3 MB each I only get 80 MB/s.

Is there a way to optimize a Zpool for this kind of usage? Like adjust the block size or something? Or is there a complete other approach to deal with this?

Fyi. Don't have the budget the use lots of SSD's :)

Cheers!
K.
How are you sharing files?
 

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
I'm on OSX so I'm using AFP. All settings on automatic so no jumbo frames. I'll try to enable Jumbo frames see what happens.

Would SMB yield better results?

K.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
I'm on OSX so I'm using AFP. All settings on automatic so no jumbo frames. I'll try to enable Jumbo frames see what happens.

Would SMB yield better results?

K.
You can experiment with different file sharing protocols. I'm not too family with netatalk (AFP). NFS will probably be much faster, but you will want an appropriate SSD to act as a SLOG. Alternatively, if you are in the mood for somewhat destructive experimentation, you can destroy your pool and recreate it consisting of mirrored vdevs. Always remember that one of the keys to keeping good performance with ZFS is to leave a decent amount of free space in your pool.
 

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
I created an NFS share and got R/W of 60 MB/s which was worse then AFP performance. Haven't had the time yet to do fine tuning with options...
 

Steve Brown

Explorer
Joined
May 7, 2015
Messages
80
helloha I have the same issue where I can saturate my 1GB link to PC (90 MB/s to 100 MB/s) but when doing a bunch of small files I get 25 MB/s. I even have a LAGG setup on dual 10GB on the freenas box along with a switch that supports it. This is all on CIFS so I just always assumed it was the crappy CIFS lol.
 

zerg

Dabbler
Joined
Sep 26, 2011
Messages
16
CIFS vs iSCSI on HP Gen7 Microserver N36L running 9.10
iSCSI is as twice better then CIFS on Queued Random Read/Write 4KiB (Q= 32,T= 1)
iSCSI:
-----------------------------------------------------------------------
CrystalDiskMark 5.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 116.975 MB/s
Sequential Write (Q= 32,T= 1) : 83.738 MB/s
Random Read 4KiB (Q= 32,T= 1) : 112.804 MB/s [ 27540.0 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 49.606 MB/s [ 12110.8 IOPS]
Sequential Read (T= 1) : 105.690 MB/s
Sequential Write (T= 1) : 97.099 MB/s
Random Read 4KiB (Q= 1,T= 1) : 9.471 MB/s [ 2312.3 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 7.172 MB/s [ 1751.0 IOPS]

Test : 1024 MiB [G: 32.5% (83.1/256.0 GiB)] (x5) [Interval=5 sec]
CIFS
-----------------------------------------------------------------------
CrystalDiskMark 5.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 113.112 MB/s
Sequential Write (Q= 32,T= 1) : 86.436 MB/s
Random Read 4KiB (Q= 32,T= 1) : 51.085 MB/s [ 12471.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 20.433 MB/s [ 4988.5 IOPS]
Sequential Read (T= 1) : 98.153 MB/s
Sequential Write (T= 1) : 86.596 MB/s
Random Read 4KiB (Q= 1,T= 1) : 9.083 MB/s [ 2217.5 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 7.695 MB/s [ 1878.7 IOPS]
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
CIFS vs iSCSI on HP Gen7 Microserver N36L running 9.10
iSCSI is as twice better then CIFS on Queued Random Read/Write 4KiB (Q= 32,T= 1)
iSCSI:
-----------------------------------------------------------------------
CrystalDiskMark 5.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 116.975 MB/s
Sequential Write (Q= 32,T= 1) : 83.738 MB/s
Random Read 4KiB (Q= 32,T= 1) : 112.804 MB/s [ 27540.0 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 49.606 MB/s [ 12110.8 IOPS]
Sequential Read (T= 1) : 105.690 MB/s
Sequential Write (T= 1) : 97.099 MB/s
Random Read 4KiB (Q= 1,T= 1) : 9.471 MB/s [ 2312.3 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 7.172 MB/s [ 1751.0 IOPS]

Test : 1024 MiB [G: 32.5% (83.1/256.0 GiB)] (x5) [Interval=5 sec]
CIFS
-----------------------------------------------------------------------
CrystalDiskMark 5.0.3 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 113.112 MB/s
Sequential Write (Q= 32,T= 1) : 86.436 MB/s
Random Read 4KiB (Q= 32,T= 1) : 51.085 MB/s [ 12471.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 20.433 MB/s [ 4988.5 IOPS]
Sequential Read (T= 1) : 98.153 MB/s
Sequential Write (T= 1) : 86.596 MB/s
Random Read 4KiB (Q= 1,T= 1) : 9.083 MB/s [ 2217.5 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 7.695 MB/s [ 1878.7 IOPS]
Kinda to be expected, but try having multiple clients connect to ISCI. :p
 

zerg

Dabbler
Joined
Sep 26, 2011
Messages
16
In my case I do not need to share contents of "iSCSI" drive. It is just to keep some "private use" files as a raw images for example.
 
Status
Not open for further replies.
Top