Arc Size Shrinks

[Black Ninja]

My system has 32GB Ram and 2 Pools (10x4TB Disks in RaidZ2 and 6x2TB Raid10). I setup LACP and start testing 2 streams simultaneous streams to make sure I can saturate 2x1GB links. I was able to saturate both links for the most parts but notice unusual behavior in ARC size, that usually didn't happened before - the picture I attached.

Can somebody explain me why this is happening and how bad is it ?!

 

[cyberjock]

Why don't you explain to us what is happening on your end. A chart with a line means nothing to me.....

 

[Black Ninja]

Ok. I am sorry. Let say I reboot the freenas box and the arc almost empty, then when I start copying sequential large files some point arc reaches 23GB and stays like that forever. That's what I consider is normal behavior. Under light usage this arc never changes it's size unless I reboot.

Recently I setup dual link 1GB LACP and start testing it with large files - 2 users write or read simultaneously in attempt to saturate 2 gigabit links to make sure LACP it's working properly , I understand this will add much more stress on the system compared to only 1 user at the time. Then I notice my arc when down to 1.6GB from 23GB(the picture I show) then start climbing again.

From what I know when system need more ram it will take from arc soem, but system already had 7-8 GB Free memory reserved for that purpose and even if needed more ram isn't 22GB too much. All sudden dropped arc from 23GB to 1.6GB .

Is this indication of insufficient memory or arc got flushed out on purpose by the system for some other reason I don't understand ?

 

[cyberjock]

ARC was flushed, most likely because the blocks that were in the ARC were for a file that was deleted. If you copy a 20GB file and it all ends up in the ARC, then you delete it, the ARC will suddenly shrink by 20GB. ;) You can't cache data that no longer exists on the pool, right?

 

[Black Ninja]

Oh my god . I did delete a lot of big files after being copied , because I was just testing speeds. So that's what caused the arc to do that ? I didn't know arc don't keep data after it's deleted, it make sense but I didn't know it was aware of it.

So arc changing it's size it's not indication of inefficient memory ? Is there anything else I can track to feel the system to make sure it has enough memory , besides waiting for the pool performance to drop drastically ? I am trying never go get to that point ?

 

[cyberjock]

Read the thread on arc hit rates and such (it's been created in the last week). There's some info there, but it is important to remember that using fake data does NOT work. You must use your real data (or a copy of the real data) and you must use it in the same way that you normally would use it. In essence, stand up the server and use it in production, then monitor it accordingly.

 

[Bidule0hm]

So arc changing it's size it's not indication of inefficient memory ?

No, the ARC is alive :D more seriously, it adapts to the workload so if there is big changes in it (like big delete, big copy, ...) they can provoke some big changes in the ARC too, it's normal.

Is there anything else I can track to feel the system to make sure it has enough memory , besides waiting for the pool performance to drop drastically ? I am trying never go get to that point ?

The thread quoted by cyberjock is in my signature. From the stats posted by some members there is some indicators that the ARC would be happier with more RAM:

• The hit ratio is under about 90%
• The hit MFU:MRU ratio is under 10:1

If you see these cases the ARC is (way) too small and the performances are more than likely very low:

• The hit ratio is under 80%
• The hit MFU:MRU ratio is under 1:1
• The MRU and/or MFU ghost hit is over a few percents
• The stats (hit ratio and hit MFU:MRU ratio) are good just after a reboot but are worse and worse with the time (it the exact opposite on a good server...)

Note that these stats are useful only after the ARC has warmed-up (it can take weeks to do it...) and with production data/usage (if you do some testing for other things you can ruin the stats temporarily).

 

[cyberjock]

There's the link. :P

Keep in mind that even if the hit ratio is <90%, and even if the MFU:MRU is under 10:1, if you are still getting good performance you shouldn't worry too much. It is entirely possible that your workload is not capable of hitting a 90% ratio. This is just one example of many that are out there on why ZFS is more of an art than a science.

 

[Black Ninja]

Thanks guys. I'll check the link you recommend , I think I might read it already.Not exactly the answer I was hoping for but it I still appreciate it.

I understand that arc hit ratio and MFU:MRU ratio are relevant for performance but this is cache sort of performance which is more relevant where cache is needed like datacenter. I mean what cache does - it caches , it don't make the thing faster. Here is example which happens to be exact way my freenas is used:

You have 20-30 TB pool for media storage. Everyday you add new movies , and read ( watch) old ones. So you have a lot of moving around but , nothing pretty much is read twice, except some metadata perhaps. Correct me if I am wrong but , in this case I will have a very bad arc hit ratio and bad MFU:MRU ratio , but this doesn't mean the server will move files slow, right ? This is what cyberjock was also suggesting.

I gave my Freenas (Supermicro X9SCM-F 32GB DDR3 ECC 1600 RAM) the max memory that can support. I can't add more and you know why. Now I am trying to understand what is the max usage I can get from it. Cyberjock said (on other treads) that can work till one day because extremely slow because of memory related issues. I am trying to understand what will make it so slow and how to monitor the memory in order to avoid getting to this point.

 

[Bidule0hm]

Keep in mind that even if the hit ratio is <90%, and even if the MFU:MRU is under 10:1, if you are still getting good performance you shouldn't worry too much. It is entirely possible that your workload is not capable of hitting a 90% ratio. This is just one example of many that are out there on why ZFS is more of an art than a science.

Yes, it's why I put them in the "would happier" rather than the other list "(almost) dying server", and as with all benchmarks and stats analysis you need to interpret them according to your particular usage :)

Just to be sure: not the "MFU:MRU ratio" but the "hit MFU:MRU ratio", the first refer to the sizes ratio and the second to the ratio of the number of times the caches are hit ;)

I mean what cache does - it caches , it don't make the thing faster.

Yes, and NO. The whole purpose of a cache is to make things faster :D

nothing pretty much is read twice, except some metadata perhaps.

Yes, but that's why we have the prefetcher. It can pre-fill the cache with data that it thinks you'll need just after.

I am trying to understand what will make it so slow

You mean what kind of workload? answer --> random small R/W (like iSCSI), it's the worst type of workload you can have.

 

[Black Ninja]

Bidule0hm, Yes I mean what kind of workload. For small R/W like ISCSI I would use small SSD pool, I wound even dream performance from my mechanical pools(10x4TB Disks in RaidZ2 and 6x2TB Raid10).

But fortunately my needs are way more modest:), I just need to be able to saturate 2 gigabit links with 2 simultaneous streams of Read or Write to the server with large sequential reads (movies). I am able to do that for the most part except when both streams are wiring to the pool , then I can only saturate 1GB to the server. Perhaps I need ZIL.

I have 16bays in my server I can make my 10x4TB pool to be 10x6TB like cyberjock , or my 6x2TB Raid10 to be with bigger hdd , but I don't know how this will affect the pool ,will it became slower ? Did adding the second pool will need more memory to function or it doesn't matter how many pools you have just the capacity ? Will almost empty pool need the same resources like pool 80% ? If you know how it works , you can plan your optimal server, right ?

The goal is to saturate 2 gigabit links to the server and put the max storage possible for 16 bays in some raidz2 redundancy. That's all. I don't need to run plugins, jails, vm, iscisi or anything else, I would build new server for anything else.

 

[Bidule0hm]

I think it's not related to the pool because for example with a 8 drives RAID-Z3 16% full I have the throughput to saturate more than 4 gigabit links. The RAID-Z3 is slower than any of the other ZFS RAIDs and I've made no tweaks for the speed, so with a RAID-Z2 with more drives or a striped mirrors vdev you should have even better pool throughput than me.

To rule out the pool you can do the well known dd R/W benchmark (of course replace tank with your pool name):

Code:

dd if=/dev/zero of=/mnt/tank/tmp.zero bs=2048k count=50k #disable shares and compression for the test!!!
dd if=/mnt/tank/tmp.zero of=/dev/null bs=2048k count=50k #disable shares and compression for the test!!!

Yes the pool space usage induces performance variations: the more you fill the pool the lower the performances. Will usually set the limit at 80% full and the critical limit at 90% ;)

 

[Black Ninja]

I do agree with you on the first and last part , but not in the middle.:)
My pool with more drive and raidz2 should be faster - I agree, I also agree not to go over 80% on pool capacity.
What I strongly disagree is that you can saturate 4gb links with your pool using mechanical drives.I think it's a common mistake people make when evaluating performance , because don't take other factors about HDDs in consideration.

The benchmark you suggest is synthetic, and reading with 500MB/s don't mean you can saturate 5x 1GB links.

Here is simple fact and tested my self:
My pool can read sequentially for one client with speed above 600MB/s , my SSD can do the same with 450 MB/s, but when you start pulling data from 3 client simultaneously then my SSD can supply each client with a 114MB/s but my mechanical pool barely will keep up with 2. If some links are reading some writing it's even worst for the mechanical drives. It's just the nature of the mechanical drives , the head can't be in 2 places in the same time. And even if single HDD can read with 190MB/s and you start another copy in the same time , the speed will not be divided in half , it will drop to 30MB/s approx. for each stream. So I don't think you can saturate more than 2x1GB links with your pool.

 

[Bidule0hm]

Hey, good find!

You're right. The more exact sentence would be "I can saturate a 4 Gig link for one client" :)

"So I don't think you can saturate more than 2x1GB links with your pool." I've two NICs but I can't be bothered to test this, I know it would be useless for me because of unmanaged switch, pain in the ass LAGG management (a bit like the jumbo frames thing) and every other machines on the network have only one NIC (and no, there's not enough users to see any benefits too) :P

"It's just the nature of the mechanical drives , the head can't be in 2 places in the same time." So, hello the loved ARC, again :D

 

[cyberjock]

Just to be sure: not the "MFU:MRU ratio" but the "hit MFU:MRU ratio", the first refer to the sizes ratio and the second to the ratio of the number of times the caches are hit ;)

Yeah, that's what I meant.

 

[Black Ninja]

Hey, good find!

You're right. The more exact sentence would be "I can saturate a 4 Gig link for one client" :)

"So I don't think you can saturate more than 2x1GB links with your pool." I've two NICs but I can't be bothered to test this, I know it would be useless for me because of unmanaged switch, pain in the ass LAGG management (a bit like the jumbo frames thing) and every other machines on the network have only one NIC (and no, there's not enough users to see any benefits too) :p

"It's just the nature of the mechanical drives , the head can't be in 2 places in the same time." So, hello the loved ARC, again :D

I don't use mumbo-jumbo frame on Gig network either:), but I like lagg with 2 links. I don't have many users but, the idea is when 1 computer start uploading to the freenas 500GB data that take hours , the other link is available for access , otherwise you saturate 1 GB link and you can't use the machine for anything else till upload is done. I like Arc , I start looking at ZIL option more and more, if it does what i think it is it might make thing very nice.:D

 

[Black Ninja]

Yeah, that's what I meant.

Is that all you have to say ? I was hoping you'll tell us more about how the things work ?:D

 

[cyberjock]

Is that all you have to say ? I was hoping you'll tell us more about how the things work ?:D

Sorry. Kind of bored of explaining it. :/

 

[Black Ninja]

Sorry. Kind of bored of explaining it. :/

Don't ever to be bored explaining if there are people willing to learn.:)Remember everything you know you did learn from others too.:)

 

[Black Ninja]

ARC was flushed, most likely because the blocks that were in the ARC were for a file that was deleted. If you copy a 20GB file and it all ends up in the ARC, then you delete it, the ARC will suddenly shrink by 20GB. ;) You can't cache data that no longer exists on the pool, right?

I can't thank you enough for this answer. That's exactly what happened and I am so relieved. For now...