L2arc size via arcstat; odd size

[aufalien]

Hi,

I'm running FreeNAS v9.1.1 on one of my servers (soon to be latest).

It has 960GB of L2arc spread across 4xIntel 520 240GB SSD.

Upon executing "arcstat.py -f l2size" I get;

4.2TB

Seems REALLY odd to me. Any guidance is appreciated.

 

[Yatti420]

Specs please.. Do you really need a cache device?

 

[aufalien]

For sure, we really need it. What sort of specs?

 

[HoneyBadger]

For sure, we really need it. What sort of specs?

Processor, motherboard, memory amount (and ECC/non-ECC, though prepare for badger attack if it's the latter), drive spindle count, zpool and vdev configuration, type of data stored, access protocols (CIFS/NFS/iSCSI), number of clients accessing.

 

[aufalien]

Oh sorry, duh, mutli tasking.

42x3TB Seagate ES.2
128GB mem (ECC, don't ask)
4xIntel 240GB 520 SSD - L2Arc
2xOWC 50GB Enterprise Pro - ZIL (tested better then the 100GB Intel DC3700 BTW)
Duel 10Gb SolareFlare Server NICs
LSI 92016-16e
Intel 2430x2 Hexcore CPUs on Intel 2400SC2 mainboard

NFS only, ~250 client mounts NFS V3, nolocks.

13xRaidZ vdevs.
3xdisk each.

So what does arcstat -f l2size really mean?

 

[cyberjock]

For starters, you should never have more L2ARC than 5x system RAM.. so you're already making a serious thumbrule mistake.

So that begs the question of what else you are doing wrong. Choosing to add an L2ARC isn't just a simple "yes/no" answer. There's a range of answers that apply for various situations.

I'm sorry, but I won't go into much detail because the info is in the forums if you choose to go read it.

Good luck.

 

[HoneyBadger]

Are you using compression on your zpool?

You probably want arcstat.py -f l2asize (note the A in there) - which should show the actual (compressed) size of L2ARC.

(Interested about the OWC SSDs as well but I'll save that for a message or separate conversation.)

 

[aufalien]

Cyber, I have a huge working data set and require the 960GB of L2Arc. My turn over rate is weekly (changed from daily) pretty much.

There are no issues with adding large L2Arcs if you have the RAM to support both it and properly sized ARC. I've plenty of RAM and my ARC is not starved by any means.

Honey, yes LZ4 but my impression that L2Arc compression didn't make its way into FreeNAS till 9.2 (I asked Josh about it)? The l2asize doesn't work for me on this box or my 9.2.1.2 box. Too bad that would have been a good stat to see.

 

[cyberjock]

Cyber, I have a huge working data set and require the 960GB of L2Arc. My turn over rate is daily pretty much.

If its a daily turnover then you shoudl already know L2ARC isn't a good choice for your situation.... So that's 2 major mistakes.

There are no issues with adding large L2Arcs if you have the RAM to support both it and properly sized ARC. I've plenty of RAM and my ARC is not starved by any means.

Keep talking.. that's #3 bro. You clearly haven't done your homework on L2ARCs.

Honey, yes LZ4 but my impression that L2Arc compression didn't make its way into FreeNAS till 9.2 (I asked Josh about it)? The l2asize doesn't work for me on this box or my 9.2.1.2 box. Too bad that would have been a good stat to see.

That is correct.. L2ARCs don't have compression at this time.

Anyway.. good luck. Nice to see people still don't do their homework, flip switches, then wonder why they aren't getting amazing performance.

 

[aufalien]

Here we go, some one asks for help and it turns into a flame war. I think you need to retire form this gig.

Post correction, cache turn over is weekly.

 

[HoneyBadger]

Honey, yes LZ4 but my impression that L2Arc compression didn't make its way into FreeNAS till 9.2 (I asked Josh about it)? The l2asize doesn't work for me on this box or my 9.2.1.2 box. Too bad that would have been a good stat to see.

Well, there goes that theory. I tried it on a Solaris box and I'm getting l2asize=0 so maybe it's not pulling the settings right. l2size is 1.8T which is also way wrong.

 

[aufalien]

Thanks for checking HoneyB. I'm migrating to 9.2.X soon and will see what comes of the numbers.

Those Intel 520 SSDs are sorta weird anyways, there power light stays on 24x7 were as all other drives/SSDs are off. Plus during boot time, FreeNAS hangs on those drives when configured in the zpool, cycling through them for like 10 min.

I can see the read light hit them twice in order, one by one for 2 cycles until ~10 min goes by.

My vendor is super cool and swapping em for Samsung Pros for me to test.

 

[HoneyBadger]

Thanks for checking HoneyB. I'm migrating to 9.2.X soon and will see what comes of the numbers.

Let me know. It's always a bit of a mess in my head between FreeBSD/FreeNAS/Solaris ZFS.

Re: the blinking/hanging Intel SSDs ... are they going through a SAS expander by any chance?

 

[aufalien]

Why yes they are my good man. The Samsungs are behaving by the way.

I'll test the Intels direct to the mobo controller to check behavior.

 

[jgreco]

I'm going to disagree with cyberjock here, basically an established pool, workload, and L2ARC setup is actually the ideal way to validate whether or not it is stressy. The upper limit of potentially workable L2ARC setups can be in excess of 10:1, but of course if you tell people that, then they get the massive L2ARC device and it turns out that it isn't actually viable for their pool, at which point ... bad things.

If you've actually verified that you're not stressing your ARC you ought to be fine.

 

[aufalien]

Yes, its currently 15GB due to the nature of our workload.

I predict triple/quad that during very very very bizzy periods so all good. Right now we are simply very bizzy :)

The questions begs, why the odd size when running arcstat?

Oh and thanks much jgreco. You're a long time poster here and I appreciate the kindness.

 

[jgreco]

No clue about the misreported size, sorry. Only one box here running 9.2.1.2 and it has no need of it.

 

[aufalien]

No worries, since its a 9.1.1 box with these seemingly funky Intel 520s, I'll not worry and see how it does with 9.2.1.X and Samsungs.