ARC Hit Ratio - SMB vs ISCSI Share

[Alexandre Trevizoli]

Hi all!
I need a help. I have two Freenas Boxes, one is using SMB (sharing VHD Files to HYPER-V), other is using ISCSI with Hyper-V too....

Hit Ratio is very low on SMB and very HIGH on ISCSI. WHY?
Have way to tunning it to be better on SMB?
I know that first box, have less memory (16GB) and Second has 32GB, but difference of HIT is so BIG.
Where is my fault?

Thanks a lot!

FREENAS 1 - SMB

FREENAS 2 - ISCSI

 

[kdragon75]

You hit ratio depends on if the requested data is in the cache. That depends on if the data has be previously read and ifs it's still in the cache. That means it depends on the size of the cache and the workload. If you playing movies over smb, you hit rate will suck because you read one big file once and fill the cache with something that you will likely not read again.

This is a bit of a simplified view but should show why it might differ. It's almost impossible to explain your hit rate without knowing EXACLT how your programs/you access the data. Access patterns dictate the efficacy of a cache system.

 

[Alexandre Trevizoli]

Thanks kdragon....I understood.
Only to know, both of FreeNAS are be using to storage VM (big VHD FILES), could you teach me please, if cache system sees the files or the blocks of the files, example, if I have an VHD with 500GB size, it need to be cached fully, or system can cache only the blocks more accessed?
Sorry for my english.
Thanks again!

 

[HoneyBadger]

There's about a 99% chance that you're using the default recordsize=128K on the FREENAS1 host. If provisioned disks as thick and you're having to do a 128K partial record rewrite every time, that could be destroying your random I/O performance as well as leading to it not being able to keep relevant data in ARC, which is yielding the poor hit rate.

 

[kdragon75]

Also to be clear the ARC cashing works on a block level, not file. On a side note, stick with vhdx unless you have a reason not to. (Rant) Or better, use ESXi, proxmox, the free xen, or any other hypervisor under the sun. (/Rant)

 

[HoneyBadger]

Also to be clear the ARC cashing works on a block level, not file.

Technically it works on the "record level" but if it's having to read a 128K record just to find the 4K of data it needs, that's still bad.

 

[Alexandre Trevizoli]

KDragon75, could you explain better please, whats the problem with VHDX? We use it because we have 8 Big Servers using HYPER-CORE (free) in a CLUSTER managed for an Windows 2012R2 licensed. It's a cheaper solution that I found to use all features of HYPER-V give for free.
I was using ISCSI before, but I find very bad performance using ISCSI on this environment, we use only 10GB Ethernet Cards (Melanox) with 10GB Switch.
On FreeNas 1 is SMB (good performance) bad CACHE Hit!!!
HoneyBadger you are right it's 128K record size....I will change for 4K record size and rewrite all files. to check if cache is better.
On FreeNas 2 is ISCSI (bad performance) show good Cache Hit! 128K record size too.

 

[HoneyBadger]

KDragon75, could you explain better please, whats the problem with VHDX? We use it because we have 8 Big Servers using HYPER-CORE (free) in a CLUSTER managed for an Windows 2012R2 licensed. It's a cheaper solution that I found to use all features of HYPER-V give for free.
I was using ISCSI before, but I find very bad performance using ISCSI on this environment, we use only 10GB Ethernet Cards (Melanox) with 10GB Switch.
On FreeNas 1 is SMB (good performance) bad CACHE Hit!!!
HoneyBadger you are right it's 128K record size....I will change for 4K record size and rewrite all files. to check if cache is better.
On FreeNas 2 is ISCSI (bad performance) show good Cache Hit! 128K record size too.

I believe Kdragon is saying that you should use VHDX unless you have a reason not to - in other words, VHDX on shared SMB is better than iSCSI in his opinion. I don't have a great deal of experience with Hyper-V so I'll defer to his knowledge there.

As far as the recordsize, 4K may be aggressively small - you will have a lot of extra metadata and sequential/streaming copies will be slowed by having to be broken into the small parts. Consider a middle ground of 16K - which by the way is the default for iSCSI extents in FreeNAS (the variable is called volblocksize for ZVOLs, and it can't be changed after ZVOL creation.

 

[kdragon75]

I'm ment to imply vhdx is better than vhd. As for record size, Oracle recommend a 64k record size for iscsi on ZFS for cm disks. I don't know how or I this would translate to SMB.