Arc Hit Ratio is at 100%?

[yis]

hey guys!
First i would like to thank everyone who helped me pick the hardware for my FREENAS. You guys are awesome.
I currently have the following hardware:
Lenovo ThinkServer TS140 (I3 )
3 * 3TB NAS drives (western Digital red)
12gb Memory
and 1gb Ethernet

----
my desktop has also 1gb connection -- both connecting to a 1gb switch..
-------
I create a 1TB ISCSI drive on the freenas and i was able to add it to my desktop with no problems. However when i try to copy a large from my desktop to the ISCSI share it start really fast (almost 600mb/second) then seconds later it drops down to 12-13MB ... I had better performance with CIFS share! (around 45mb)
so what am i doing wrong?
I noticed in the reporting that the Arc Hit Ration is at 100%! (first of all, what the hell is that?) second, why is it maxing out?
here are some of the screen shot.. please note those were taken as i am coping 32gb worth of files..

notice here the ARC Hit Ratio is maxed out.

 

[yis]

and oh, i am running FREENAS 9.3STABLE201502110455

 

[Bidule0hm]

First it's not at 100% but at 98.84, there is a difference.

This difference is significant because your ARC hit ratio should range between 90 and 100% for optimal performance (under that adding more RAM will be beneficial; source: cyberjock) so a 1% difference is like a 10% difference.

The ARC is the read cache and the hit ratio is the percentage of reads that come from the cache and not from the drives. So a perfect usage of the cache would give 100% and no cache usage at all would give 0%.

Finally, your hit ratio is good, even very good :D

You can see other members stats on this topic.

 

[yis]

First it's not at 100% but at 98.84, there is a difference.

This difference is significant because your ARC hit ratio should range between 90 and 100% for optimal performance (under that adding more RAM will be beneficial; source: cyberjock) so a 1% difference is like a 10% difference.

The ARC is the read cache and the hit ratio is the percentage of reads that come from the cache and not from the drives. So a perfect usage of the cache would give 100% and no cache usage at all would give 0%.

Finally, your hit ratio is good, even very good :D

You can see other members stats on this topic.

oh.

so why the transfer rate is at 15mb? isnt that too slow for ISCSI? where is the bottleneck? any idea?

 

[Bidule0hm]

This is an entirely other topic.

Generally the main bottleneck is the network. Then for a enterprise usage the main bottleneck can also be the workload (VMs with a lot of small random I/O for example).

For iSCSI in particular I don't know much, you might want to wait an answer from a more knowledgeable member ;)

And I recommend you to make another thread with an appropriate title and to provide more info (hardware specs, workload type, etc.).

 

[yis]

This is an entirely other topic.

Generally the main bottleneck is the network. Then for a enterprise usage the main bottleneck can also be the workload (VMs with a lot of small random I/O for example).

For iSCSI in particular I don't know much, you might want to wait an answer from a more knowledgeable member ;)

And I recommend you to make another thread with an appropriate title and to provide more info (hardware specs, workload type, etc.).

I agree with you. I initially thought that ARC might be the problem (FREENAS noob over here)
Mods please close this thread and i will create another one.

Thanks

 

[cyberjock]

Well, iSCSI is a random-read random-write workload. That means you need gobs of cache. When i say "gobs" I mean at *least* 32GB of RAM to have something resembling reasonable performance, and very likely 64GB of RAM and an L2ARC. You'll also probably need to do mirrored vdevs and lots of them to get good I/O.

You basically are doing all the wrong things with hardware that you can possibly do for iSCSI. :/

With only 12GB of RAM and a RAIDZ1 you are certifiably crazy for even expecting good performance. It starts off going real fast because your writes are being cached in RAM. But the second that cache runs out you are seeing the real performance of the pool. Sucks don't it? Get more powerful hardware that can absorb that suckiness. ;) I've got 32GB of RAM on my main server and I don't even try to do iSCSI. I did once, and it performed so poorly that I quickly realized it was a no-brainer and gave up.

 

[Bidule0hm]

I almost asked for the MFU:MRU hit ratio and the prefetch eff %, you think we can see some relevant info from that?

 

[cyberjock]

I almost asked for the MFU:MRU hit ratio and the prefetch eff %, you think we can see some relevant info from that?

You are welcome to ask, but the numbers are just a hair above meaningless with this hardware and this workload. It's gonna basically say "zomg I'm so f*cked up I don't know up from down!" Prefetch is also going to be good with very little data in the iSCSI (and if its been read nearly sequentially, which *you* as the end user can't even prove without serious ZFS debugging) or very poor if there is a lot of data (or if it hasn't been read sequentially).

 

[Bidule0hm]

Ok. But I'm curious :p In theory the MFU:MRU hit ratio should be very low if the cache isn't big enough for the task.

@yis can you post the result of these two commands please?

Code:

python /usr/local/www/freenasUI/tools/arc_summary.py | grep "Most Frequently Used:"
python /usr/local/www/freenasUI/tools/arc_summary.py | grep "Most Recently Used:"

 

[yis]

Well, iSCSI is a random-read random-write workload. That means you need gobs of cache. When i say "gobs" I mean at *least* 32GB of RAM to have something resembling reasonable performance, and very likely 64GB of RAM and an L2ARC. You'll also probably need to do mirrored vdevs and lots of them to get good I/O.

You basically are doing all the wrong things with hardware that you can possibly do for iSCSI. :/

With only 12GB of RAM and a RAIDZ1 you are certifiably crazy for even expecting good performance. It starts off going real fast because your writes are being cached in RAM. But the second that cache runs out you are seeing the real performance of the pool. Sucks don't it? Get more powerful hardware that can absorb that suckiness. ;) I've got 32GB of RAM on my main server and I don't even try to do iSCSI. I did once, and it performed so poorly that I quickly realized it was a no-brainer and gave up.

Thank you for this information. I really appreciate it. I actually dont mind throwing memory, i just didnt know where the hell is the bottleneck. That's been said, if i didnt want to add more memory, is there any other protocol i can use on my windows box that will give me decent performance? CIFS looked ok, but i know i can do better than that :(..
Again appreciate all your help.

Ok. But I'm curious :p In theory the MFU:MRU hit ratio should be very low if the cache isn't big enough for the task.

@yis can you post the result of these two commands please?

Code:

python /usr/local/www/freenasUI/tools/arc_summary.py | grep "Most Frequently Used:"
python /usr/local/www/freenasUI/tools/arc_summary.py | grep "Most Recently Used:"

here are the info requested.

[root@Storage ~]# python /usr/local/www/freenasUI/tools/arc_summary.py | grep "M
ost Frequently Used:"
Most Frequently Used: 35.21% 820.17k
[root@Storage ~]# python /usr/local/www/freenasUI/tools/arc_summary.py | grep "M
ost Recently Used:"
Most Recently Used: 64.48% 1.50m
[root@Storage ~]#

 

[jgreco]

The reality is that when you're doing iSCSI, you're making life really, really hard for the filer, because it has no idea what is going on and it is just trying to store random blobs of data your PC is shoving at it. It'll do its best, of course, but it needs more resources to do it effectively.

When you use a NAS protocol like CIFS or NFS, it tends to be very different, because the filer is operating in a much more ideal manner, working with files and managing space itself. You'll probably find that it is massively faster to use CIFS or NFS.

Also I'm going to ding you for using "mb/s" and request that you start using appropriate abbreviations. It is very frustrating to guess what you mean.

 

[yis]

The reality is that when you're doing iSCSI, you're making life really, really hard for the filer, because it has no idea what is going on and it is just trying to store random blobs of data your PC is shoving at it. It'll do its best, of course, but it needs more resources to do it effectively.

When you use a NAS protocol like CIFS or NFS, it tends to be very different, because the filer is operating in a much more ideal manner, working with files and managing space itself. You'll probably find that it is massively faster to use CIFS or NFS.

Also I'm going to ding you for using "mb/s" and request that you start using appropriate abbreviations. It is very frustrating to guess what you mean.

Thank you for this and thanks for your feedback. I know getting the wrong term is confusing so i apologize.
what i meant was MByte/sec
Question:
Assuming i added more memory (lets say up to 32GByt). Would iSCSI have a faster transfer rate than a regular CIFs share? Or should i just shut the hell up and use CIFs because my hardware sucks?
The only reason i am experimenting with iSCSI is that in the future I would like to create a SQL/exchange server with the database(s) stored on the storage, if iSCSI is performing that badly then i might just forget about that altogether.

 

[jgreco]

That's a lot more helpful and gives some insight.

CIFS is probably going to be faster for general purpose storage. iSCSI will be better for SQL/exchange, probably, but also has substantially higher resource requirements.

I should write up something on this. More in a bit.

 

[jgreco]

Yeah, here, go read this

https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/

 

[jgreco]

Was that helpful? It comes down to ZFS not-knowing anywhere near as much about the data it's storing when you use iSCSI. You can still make iSCSI awesome but you have to throw big resources at it.

I had a single VM reading at 600MB/sec (that's megaBYTES) off a FreeNAS iSCSI disk over 10gigE, but it was all coming from ARC.

 

[yis]

Was that helpful? It comes down to ZFS not-knowing anywhere near as much about the data it's storing when you use iSCSI. You can still make iSCSI awesome but you have to throw big resources at it.

I had a single VM reading at 600MB/sec (that's megaBYTES) off a FreeNAS iSCSI disk over 10gigE, but it was all coming from ARC. :)

yes it was, thank you for all that information. when i used to be a system engineer i used NetAPP filers so i never had to worry about hitting the limits , it just worked and worked beautifully. but that was a FAS6040 (with 10gigE) ...
but now that i am using horrible hardware i never thought that iSCSI would be hit that hard and perform this badly .. sigh, need to rethink my plans and approach now that i have to work with less than idle hardware.

 

[Bidule0hm]

Most Frequently Used: 35.21% 820.17k
Most Recently Used: 64.48% 1.50m

Yep, that's exactly what I thought. You have a hit MFU:MRU ratio of roughly 0.5 when it should range from 10 to 100 or even more.

What all of this mean? well, the MFU is the list of blocks of data you've accessed many times and the MRU is the list of blocks of data you've accessed recently but only one time. A high MRU and a low MFU hit percentage means that the cache is constantly renewed because it's too small, so it's not very useful.

 

[HoneyBadger]

I think the big issue here is that you don't have either enough or the right kind of back-end vdevs to handle a sustained 1Gbps ingest rate.

In a perfect world, your iSCSI writes would perfectly pass through as sequential writes to the RAIDZ1 vdev and it would be able to keep up because RAIDZ1 handles sequential IO quite well.

But realistically, you're basically demanding your disks write full-speed, all the time, without missing a step. I'm guessing it falls on its face around ten or fifteen seconds after the transfer starts?

 

[yis]

Yep, that's exactly what I thought. You have a hit MFU:MRU ratio of roughly 0.5 when it should range from 10 to 100 or even more.

What all of this mean? well, the MFU is the list of blocks of data you've accessed many times and the MRU is the list of blocks of data you've accessed recently but only one time. A high MRU and a low MFU hit percentage means that the cache is constantly renewed because it's too small, so it's not very useful.

and to increase that, i would need to add more drives/memory? or just memory?

I think the big issue here is that you don't have either enough or the right kind of back-end vdevs to handle a sustained 1Gbps ingest rate.

In a perfect world, your iSCSI writes would perfectly pass through as sequential writes to the RAIDZ1 vdev and it would be able to keep up because RAIDZ1 handles sequential IO quite well.

But realistically, you're basically demanding your disks write full-speed, all the time, without missing a step. I'm guessing it falls on its face around ten or fifteen seconds after the transfer starts?

yes it is, it starts really fast .. i think i saw the first 2 seconds hit 950MByte/sec then start slowing down.... 10 seconds later it goes down to 12MByte/sec and stays there :( so frustrating.