To add L2ARC or not to add

[anodos]

System:

• FreeNAS 9.2.1.3
• Supermicro MBD-X9SCI-LN4F-O motherboard
• Xeon E3-1220V2 CPU
• LSI 9211-8i HBA
• zpool = 6disk RAIDZ2 vdev
• 32GB ECC RAM

Symptom:
Uncached browsing of directories on CIFS shares containing large amounts of media files with Windows ACLs is painfully and annoyingly slow (taking up to a minute to load a large directory). I have decided to work around the issue by caching all data loaded when browsing directories. In essence, I want to throw more hardware at the issue.

Possible solution:
I am not very familiar with zfs, but I believe the information I have listed below from arc_summary indicates that my ARC is undersized for my current workload. RAM is maxed out and so I'm thinking of adding an L2ARC since I can't go out and get a new computer.

Code:

ARC Total accesses:                    1.78b
    Cache Hit Ratio:        96.60%    1.71b
    Cache Miss Ratio:        3.40%    60.38m
    Actual Hit Ratio:        93.47%    1.66b
 
    Data Demand Efficiency:        94.62%    255.32m
    Data Prefetch Efficiency:    0.74%    16.57m
 
    CACHE HITS BY CACHE LIST:
      Anonymously Used:        1.90%    32.59m
      Most Recently Used:        4.44%    76.13m
      Most Frequently Used:        92.32%    1.58b
      Most Recently Used Ghost:    0.06%    949.83k
      Most Frequently Used Ghost:    1.29%    22.05m
 
    CACHE HITS BY DATA TYPE:
      Demand Data:            14.09%    241.59m
      Prefetch Data:        0.01%    122.47k
      Demand Metadata:        82.66%    1.42b
      Prefetch Metadata:        3.25%    55.71m
 
    CACHE MISSES BY DATA TYPE:
      Demand Data:            22.75%    13.73m
      Prefetch Data:        27.24%    16.44m
      Demand Metadata:        32.01%    19.33m
      Prefetch Metadata:        18.01%    10.87m
                                Page:  3

Questions:
I have attached the output of arc_summary.py for reference (note that these numbers reflect 60 days up-time).
Does you think throwing in an L2ARC is a reasonable response (assuming that samba / FreeNAS are properly configured)?
If so, is a 240GB Intel 730 a good choice? If not, what would you recommend?

Let me know if I am making some newbie mistakes or am asking the wrong questions.
Additionally, any recommendations for reading material regarding how to interpret arc_summary, arcstat, etc. would be much appreciated.

 

[cyberjock]

Well, from what I've seen adding an L2ARC with <64GB of RAM typically makes the box slower. Yes, you can throw more hardware at ZFS and make it slower. There's well understood reasons for this and I don't want to explain it again, you can search if interested.

Your L2ARC should also never exceed 5x your ARC. In your case you are probably looking at a 25-28GB ARC best case, so I wouldn't go over 120GB, and if you run any kind of jails or other software you might even have to go as low as 80GB.

So what's my opinion: Don't consider an L2ARC until you have 64GB of RAM minimum. Of course, for you this also means a whole new system. :(

Also keep in mind an L2ARC may not even appreciably help as it only stores info after you've requested it several times. So not necessarily useful for this task either.

 

[no_connection]

"Support DOS File Attributes:"
Is that something you need?
Unchecking that solved my directory problem.

 

[Maq]

"Support DOS File Attributes:"
Is that something you need?
Unchecking that solved my directory problem.

That has helped me with speed a lot, however, it messes up the permissions a bit because only the owner can change the permissions with that option unchecked.

 

[cyberjock]

That has helped me with speed a lot, however, it messes up the permissions a bit because only the owner can change the permissions with that option unchecked.

That's why you should always research what something does before doing it. I'm not saying you didn't or that you blindly did it and are upset. I'm just saying that what works for his situation might not work for others. 95% of the servers I've dealt with could not be improved with this setting change. The change in behavior for permissions makes this unusable for them.

 

[Maq]

That's why you should always research what something does before doing it. I'm not saying you didn't or that you blindly did it and are upset. I'm just saying that what works for his situation might not work for others. 95% of the servers I've dealt with could not be improved with this setting change. The change in behavior for permissions makes this unusable for them.

Yup, totally agree and I did research. I was expecting this to happen, but I was also hoping that the "Inherit Owner" and "Inherit ACLs" options would help with the permissions. Still a lot to look into.

 

[anodos]

I need extended attributes.
I've been able to reproduce the problem with my ARC only 60-70% full, which probably means 'browsing data' is being flushed from ARC. This makes sense because vfs.zfs.arc.meta_limit is set to around 7.5GB. Some rough unscientific benchmarks (lots of rebooting and browsing folders and viewing vfs.zfs.arc_meta_used in arc_summary.py) indicates that browsing directories of media files accesses about 4-8GB metadata per 1TB of stored files. So in order to cache the entire media collection's browsing data (for lack of a better term) I will need to cache about 32-64GB metadata, which I think would require a minimum of 96GB ARC. Apart from browsing directories performance has been very good. That's why I have been considering adding an L2ARC with the parameter "secondary cache = metadata", and running a nightly cronjob to have smbclient browse the shares / cache the browsing data.
I may end up getting a 100GB Intel S3700 and testing it anyway just because it is much cheaper than buying a new computer (and hey - who doesn't like having an extra SSD lying around).

 

[cyberjock]

Yeah, you are very confused. Let me explain a little...

You are talking about ZFS metadata. Metadata cannot, by definition, use more than 30% of your ARC space. This is hard coded and cannot be changed by you without changing code and recompiling ZFS. That 4-8GB of metadata per 1TB of storage is also very inaccurate. Any given quantity of data will have a totally different amount of metadata based on the number of files, number of snapshots, permissions, etc. I'm actually dealing with a user that had about a bajillion 1-4MB pictures, and they have a boatload of metadata because of his storage needs. He's about to upgrade to 128GB of RAM in "hope" of handling the problem. But trying to even throw out a number for metadata size is a big joke(the joke isn't on you however). Anyway, if you think you need 32-64GB of ARC just for metadata, you really need 196GB of RAM(at the lower end) to accommodate 64GB of metadata. Not exactly cheap!

Now, you can set the secondary cache to metadata, but here's where things get dicey. If you add an L2ARC you will begin gobbling up the metadata space in the ARC with L2ARC index, so performance is very likely to drop. This is one reason why I always recommend 64GB of RAM minimum before considering an L2ARC. Thinking you're going to trade ARC for L2ARC for a win when you are already having performance problems isn't going to give you a net gain. L2ARC is slower than the ARC(duh?) and trying to trade ARC for L2ARC is only useful if your ARC is being enough to handle its duties while now handling the new responsibility of the L2ARC.

In short, I stand by my comments above about the 64GB of RAM being needed before considering an L2ARC. If you only care about metadata you can solve that problem right now...set primary cache to metadata. But when performance is in the toilet don't say I didn't warn you(more warnings below).

Also, you are probably asking for a whole new world of problems if you set secondary cache = metadata. It can work in places where directory listings are all that matters and you do not care how fast or slow the files load. I've met a whole one person that had a situation where this was true. It was a multi-million dollar server that had a very very specific purpose and no other.

 

[no_connection]

From what I have read Z2 isn't that good for IOps as you are limited to about what a single disc can do.
Would adding two more drives and running them all as mirrors in four vDevs improve things?

Also are powerd disabled?

 

[anodos]

Okay. Common sense kicked in. I've decided to revisit the provisional assumption "CIFS is configured correctly" and work forwards from there. I will do the following:

1) Thoroughly verify that I've set up CIFS correctly.
2) Experiment with different CIFS configurations (i.e. creating more shares with less data).
3) If (1) and (2) don't solve the problem, then upgrade hardware.

On a side-note, I will continue reading more about ZFS. Once upon a time I thought "it's just a file system, how complicated can it be?".

These days it feels like my work process is: read, ask questions, read more, wait and think, read more, read more, wait and think. Once in a while I will actually do something. :)

 

[Atomic]

Hi,

I have a simular problem, see thread: http://forums.freenas.org/index.php?threads/freenas-performance-probleme.21626/

My Motherboard supports max 16GB ECC RAM, what would happen if I install a 256GB SSD as RAID-cache and/or a 256GB SSD as a L2ARC-cache?

 

[anodos]

Hi,

I have a simular problem, see thread: http://forums.freenas.org/index.php?threads/freenas-performance-probleme.21626/

My Motherboard supports max 16GB ECC RAM, what would happen if I install a 256GB SSD as RAID-cache and/or a 256GB SSD as a L2ARC-cache?

In my case, the problem is probably caused by several factors most likely including:

1) a convoluted and disorganized file tree.
Combined with
2) gobs of media files
And possibly
3) samba misconfiguration
And possibly
4) samba / zfs / freebsd inefficiencies.
Resulting in
5) lots of random disk IO and pressure on my ARC which is probably causing my performance problems.

Adding an L2ARC is not a solution to (5) since it will probably just increase pressure on the ARC. The hardware solution is to upgrade your computer to on that supports more RAM.

If that is not an option try breaking your share into several datasets / shares and try simplifying your file tree (don't nest lots of folders).

 

[cyberjock]

Did you even read this thread Atomic? I mean, I already provided the advice in this thread. Please read the thread before posting again.

 

[anodos]

Okay, I found a solution that works for me. I rearranged my directories. Browsing shares is now instantaneous. ARC size after browsing to a large media-filled directory (after a reboot) improved from 1.6GB [before restructuring] to 61.70MB [after restructuring]. It seems like my hardware is good enough for my needs. I will report back once I have completed re-arranging them (and perhaps give a howto).

 

[tromba]

Hey anodos,

I have a similar "jumble of media files and metadata" situation and was wondering how that "how-to" was coming along?

Thanks a bunch in advance.

Cheers
M

 

[anodos]

Hey anodos,

I have a similar "jumble of media files and metadata" situation and was wondering how that "how-to" was coming along?

Thanks a bunch in advance.

Cheers
M

My real life job is getting in the way of writing tutorials. The concept is simple enough. Sort your data into 'projects' or some other regular grouping that contains no more than 30GB of data or 10k files. Place your project folders directly in a dataset to create a flat file tree (no branches and minimal subfolders) then recreate your original file tree using symbolic links.

If your original filetree was like the following

Code:

/mnt/Tank/Videos/ridiculously/long/and/annoying/path/MainProjectFolder/[movie1.mp4 movie2.mp4 movie3.mp4 ...] 

Post-restructure it would look like this:

Code:

/mnt/Tank/Videos/MainProjectFolder/[movie1.mp4 movie2.mp4 movie3.mp4 ...] 

If in the past you mapped /mnt/Tank/Videos on your windows workstations, you will need to create a new directory under /mnt/Tank/Videos that will become the target for explorer mappings. For instanace "mkdir /mnt/Tank/Videos/VideoRoot"

Then create hidden folders containing the 'connective tissue' for the your new file tree based on symbolic links

Code:

mkdir /mnt/Tank/Videos/{._L1_ridiculously, ._L2_long, ._L3_and, ._L4_annoying, ._L5_path}

then link them together

Code:

ln -s /mnt/Tank/Videos/._L1_ridiculously/ /mnt/Tank/Videos/VideoRoot/ridiculously
ln -s /mnt/Tank/Videos/._L2_long/ /mnt/Tank/Videos/._L1_ridiculously/long

and so on

As you can tell, this is a process that you will want to automate. I wrote a script to do that for my dataset, but it is very specific to my data. I started writing something generic that can be used by others, but other obligations are getting in the way.
I hope this gives you an idea of the direction to go in.

 

[tromba]

Interesting idea. Thanks a lot for getting back to me.

 

[anodos]

Interesting idea. Thanks a lot for getting back to me.

It works. I've now scaled it up to 9TB of data consisting of about 4 million files shared via CIFS. Performance was atrocious prior to the change. Saved me from having to buy a new computer.