400TB+ ZFS File server with large L2ARC

[Jyo]

The question is if the working set of your team will be way larger than the 256G you already have as primary cache. Plus, the data needs to be copied to and from the L2ARC. For everything that is in RAM the L2ARC is of absolutely no use. Once RAM is full, the L2ARC will start to fill up by read operations. So the only way to warm up the L2ARC is by reading data.
There is a persistence option now - it's off by default, I don't know if it is supposed to work just now. It is quite new, too - just like special vdevs.

L2ARC really depends on the question if the same blocks will be frequently read multiple times.

I do understand what you're saying Patrick, the editors tend to work on large files (which stay on the Working File Server) and open them in the video editing application. Each project can vary quite widely though depending on the scale and scope and the codecs involved. But for the sake of estimating one can take a range of 50GB to 200GB+ per project and multiple projects are worked on simultaneously (within the week you can say), for 2-3 editors.

I agree with the special vdev recommendation, totally makes sense for the archived AV media dataset.

What I'm exploring is to create a large enough SSD based cache that can accelerate whatever they work on without having to have a separate zpool which is totally SSD based. This would allow me to have a much larger dataset on disk (70-100TB) with a 10-14 TB L2ARC, than having to spend the budget on about 20-30 TB of SSD based pool.

Would things like pre-fetching help?

 

[Jyo]

Is it your intentions to serve up the video files to the clients and have them edit the video files on their local machine or to have them edit the content on the file server directly? Also, the size of the files to be worked on are of course a major consideration. Just curious.

Hi Joe, we keep files on the file server and open them in the client side application. These are large multi-GByte video and audio files, sometimes images. Its a lot easier to do things this way when we work in the team since it saves the effort of moving files to and fro before and after projects and can be reviewed easily during the iterative editing process by the team.

Is an aggressively tuned L2ARC over a large dataset a sensible way to do this? Is there any tuning that would help with pre-fetching sequential data (as that is the nature of temporal data - video and audio)? Can one tune the ARC to prefetch at file level? folder level? I understand it is normally block level, but exploring suitable options.

Whats the most aggressive way to read cache data in a ZFS File server?