Performance whilst being used for CCTV Storage

[DaveY]

Hi Dave,

This is a really good point - the graphs are showing an L2ARC size of 8.4T!!! Attached are graphs but it looks like this might be where the issue is coming in? Top is now reporting 156GB Total ARC

Now we're getting somewhere! Lots of things you can do to troubleshoot here. Try the stat and summary tools, arc_summary.py and arcstat.py and see if you can find some coorelation.

Double check your disks and make sure you didn't accidently use the SSD + your vdev or one of the HGST drives as cache! I only say this because you say you're using 8TB drives and you got an 8.4TB cache! I really doubt it's because of compression artifacts. with a 240GB drive, you would have to be doing a 35x compression ratio on your video files to get to 8.4TB. If you truly have 8.4TB of L2ARC, then removing it should take care of it. 128GB of RAM and 8.4TB of L2ARC is a bad combination. Some memory is needed for mapping. I believe there's a post out there about the proper RAM/L2ARC size ratio.

Since it's an archive, perhaps you have the luxury of rebooting. My suggestion would be to remove (all) your L2ARC devices for now and see if the suttering goes away. At 10% cache hit and the overhead of needing to map your L2ARC space, you're probably better off without it.

Good luck!

 

[DaveY]

come to think of it, i think you can detach L2ARC online. I can't remember. It's being awhile.

 

[Ericloewe]

Of course you can. Same for SLOG.

 

[chriswiggins]

Thanks guys I’ll give it a go today!

Am I right in thinking that an incorrectly configured read cache could also cause the breathing issue I’m seeing with writes? The stuttering after lots more debugging appears to be caused by the NAS not accepting any IO for 2-4 second blocks at a time. This happens for writes too. Are the two potentially related?

 

[chriswiggins]

Ok update here - removing the L2ARC device (confirmed was the SSD) hasn’t made any difference so we can rule that out? Graph still shows L2ARC which is pretty confusing?!

 

[DaveY]

Hmm, if the graph is still showing 8.3TB, then that would be concerning. You can run zpool status to see.
I'm wondering if maybe the issue is on the client end? It's suspicious all 3 of your NAS are all having the exact same performance issues at the exact same time. What client are you using to do the playback. From your screenshot, it looks like a Windows VM running on Hyper-V? Maybe the stuttering is caused by the VM? what format is the video in? mpeg4? Can you just download the file to the client first and then play it locally so we can rule out the NAS as the culprit? Or maybe try playing it back on a diffrent client machine?

Also, I forgot to mention this, but I've had similar performance problems using autotune. What you've read on this forum is right. autotune for some reason is not very good at the settings. Once I removed them, performance went back to normal. Just something else you can try.