How large of a cache drive

[beowu!f]

Hey all,

I have a 512GB SSD for my boot drive. I know I could repartition it to use most of it as a cache drive.. but not really worried about doing that at this point.

My m/b has slots for 2 more NVMe SSDs. I also have 8 SATA 3 slots, of which 5 will be used for HDDs. That leaves 3 open for SSDs.

I have a 250GB SATA3 SSD just sitting unused. It's older.. Samsung 850, but barely used.
I was also looking to buy an NVMe SSD drive IF it makes sense to do so and the 250GB may not be enough OR maybe I would use that drive for other stuff...

I know there are a variety of ways you can configure TrueNAS to utilize drives (other than the main pool.. in my case 5 HDDs).

What I am wondering is.. would there be any benefit in adding both the 250GB SATA3 SSD AND an additional 1 (or even 2) NVMe SSDs.. or more so.. HOW much of a difference in terms of how I use it.

SO.. I am using TrueNAS for a few things like most people. First and foremost as a primary redundant backup of family photos/videos, and some documents. I DO have a 2nd Synology NAS with 32TB storage.. which originally I got to handle large RAW video files of sports of my kids I used to do. Now that that is over.. I wanted to make use of this Threadripper workstation (gen 1, 16 cores) that was just sitting around for 2 years unused.. and I built myself a 10u network rack for all my network gear.. so figured.. rather than this old desktop system I have in use right now for UNRAID (which I discovered is NOT what I wanted).. make use of a more capable server that is not in use. The reason is, I am ripping a ton of 4K blurays, and I want utmost quality for my main home theater playback (plan on getting the next gen nvidia Shield when it comes out.. IF it comes out).. but also have a few kids (college, etc) that want to watch movies from time to time so I enable remote access to them only and they will need much "leaner" versions of the videos. So I will need the ability to transcode a lot of 50GB+ 4K HDR videos in to much smaller stream capable optimized versions. Hence.. the good fit for this threadripper 16 core system.

Now that that is out of the way.. I don't store stuff that often.. so write speed being slow is fine for me. Hence why I think RAIDZ1 is the right path for the 5 drives. BUT, because we have a few devices that sometimes are all streaming from Plex.. I wanted to make sure that the Plex server, and TrueNAS too, can handle the performance needs when necessary. With that in mind, hence this long post with context as to why I am asking this.. I am unsure of what SSDs will add and if 2, 3 or more would make things a lot better, or not worth the added cost.

SO.. I am OK with buying a 500GB NVMe SSD drive to add to the m/b 2nd slot.. and I could even consider a 3rd one IF it made sense. The question I guess is.. how to use it.. and how much benefit does it add?

I ALSO plan to run some VMs on this machine.. e.g. as a hobbyist developer who likes to mess around with Kubernetes, etc.. I was thinking when this server isn't transcoding stuff, I could utilize TrueNAS and VMs to run some K8s nodes, and a message broker. AS well, HomeLab server.

So.. would having 2 SSDs (outside the boot one) be good and use one for cache and the other for storing VMs/HomeLab/etc? Or is there a better way to configure all this with 2/3/4 SSDs to optimize overall performance?

Thanks.

 

[Etorix]

Streaming videos is unlikely to benefit from a L2ARC. Do you expect to stream the same film again and again?
If you're confident that you have SATA ports to spare, the 850 could serve as boot drive.
If you plan to have VMs, i.e. block storage, in addition to bulk storage (raidzN), then get a second 512 GB NVMe and make a mirror for your VMs.

 

[beowu!f]

That's a good call on the 850 being a boot drive. Didn't think about that. I'll give that a spin so the 500GB NVMe can be used for cache. What's the benefit of 2nd 512GB for VMs? Why make it mirror? For speed?

 

[Etorix]

Reliability. Without redundancy, you'll lose your VM at the first error—but if it's just a lab, this may be fine.

 

[beowu!f]

Reliability. Without redundancy, you'll lose your VM at the first error—but if it's just a lab, this may be fine.

OK.. was thinking that was what you meant. Yah.. this is more just a playground/lab. Not worried about it being redundant.

 

[NugentS]

Don't bother with a cache - you won't see any benefit for your use case - not on a pool of videos and depending on your memory may actually hinder. Please post you full hardware specs

 

[Constantin]

in this instance, the likely best use of the L2ARC is as a metadata-only caching device.

Especially if you have TrueNAS 12, enabling a persistent metadata-only L2ARC can make significant browsing improvement once it is “hot”. Ditto for any protocols that constantly traverse directories like rsync.

even better, there is no risk of data loss if the L2ARC goes belly-up. So you can use virtually any SSD.

 

[chruk]

I think this depends on the level of simultaneous access.

If you have one or two users at same time reading video files, providing the fragmentation isn't terrible on tiny access sizes, then it should be fine.

I think given a large enough record size (default 128k is probably plenty), and its only a few devices streaming video, then I don't think you need anything exotic, you shouldn't need a L2ARC at all for that.

 

[beowu!f]

Full hardware:
CPU: Threadripper Gen1 16 core
RAM: 48GB
GPU: 1030ti 2GB
SSD: 512GB 950 Pro NVMe PCIe3
(2nd/3rd slots available).
HDDs: 5 x 16TB X18 EXOS 7200rpm


SO.. from the last 3 comments.. I am unsure if I should use the 850 250GB SSD drive as the BOOT/OS, and then assign the 512GB 950 Pro as cache.. or don't bother with cache, and just leave 950 as boot/os drive and no extra SSD needed?

Sorry, not entirely sure what L2ARc and TN12 is.

 

[jgreco]

What I am wondering is.. would there be any benefit in adding both the 250GB SATA3 SSD AND an additional 1 (or even 2) NVMe SSDs.. or more so.. HOW much of a difference in terms of how I use it.

Aside from the previous discussion of "not that likely to be beneficial to your use case", also be aware that you need a crapton of RAM like 64GB+++ before L2ARC has a good chance of being useful.

 

[beowu!f]

Aside from the previous discussion of "not that likely to be beneficial to your use case", also be aware that you need a crapton of RAM like 64GB+++ before L2ARC has a good chance of being useful.

Well.. I have 48.. though oddly I thought I had 64. May have some bad sticks but haven't checked in to that yet. I am OK if it's not needed. In my case I am mostly going to copy files to it from time to time, and stream from it. It is for home.. not work related stuff.
I sometimes do video editing, but I have 8TBs on my workstation and likely more if I need it.. so I don't typically think I would work off of a NAS. I am setting it up with 10gig network though, so it could be possible but again that is hobbyist stuff, doesn't happen that often and in my case I seldom work with anything more than 500GBs of data. Easy enough to copy to the local SSDs and work on it that way than over network.

 

[Etorix]

TN12 = TrueNAS version 12
L2ARC = Level 2 ARC, i.e. the "cache" this thread is about from the beginning…
Level 1 cache, the "ARC" itself is the RAM. Adding a L2ARC takes up RAM space for a L2ARC table, so reduces the available space for the primary cache, ARC. The common rule of thumb for sizing is L2ARC≤5*RAM, so even 64 GB would seem a bit small.

If you feel adventurous, you might try to set up a L2ARC from a partition on a NVMe drive. (Or the SATA boot drive, but I'm not sure how useful is a L2ARC on a single SATA drive… shared for drive activity. It's probably still better than getting random small files from a raidz array though.)

 

[beowu!f]

TN12 = TrueNAS version 12
L2ARC = Level 2 ARC, i.e. the "cache" this thread is about from the beginning…
Level 1 cache, the "ARC" itself is the RAM. Adding a L2ARC takes up RAM space for a L2ARC table, so reduces the available space for the primary cache, ARC. The common rule of thumb for sizing is L2ARC≤5*RAM, so even 64 GB would seem a bit small.

If you feel adventurous, you might try to set up a L2ARC from a partition on a NVMe drive. (Or the SATA boot drive, but I'm not sure how useful is a L2ARC on a single SATA drive… shared for drive activity. It's probably still better than getting random small files from a raidz array though.)

Thank you for that info. I am using latest TrueNAS so TN12. So using the 512GB NVMe as L2ARC would not be beneficial? I mean, if I don't need it fine. I just assumed having an SSD be in the middle of reads/writes might make it faster.. but really not concerned. With a 10gig interface, I doubt I can saturate the network with 5 drives.

 

[aufalien]

Beowulf,

I've a similar use case in having an asset, video, media server. Your best bet is no L2ARC or SLOG devices and the pool set to async. I used 1M block sizes. Also increase your RAM if possible.

Leave hardware offloading enabled however it is advisable to disable it when running jails/VMs. My servers are strictly NFS.

Of course this is how I roll so to each his/her own.

I feel stupid commenting as I'm having issues of my own but since your use case is sorta like mine, I thought to chime in.

 

[beowu!f]

Beowulf,

I've a similar use case in having an asset, video, media server. Your best bet is no L2ARC or SLOG devices and the pool set to async. I used 1M block sizes. Also increase your RAM if possible.

Leave hardware offloading enabled however it is advisable to disable it when running jails/VMs. My servers are strictly NFS.

Of course this is how I roll so to each his/her own.

I feel stupid commenting as I'm having issues of my own but since your use case is sorta like mine, I thought to chime in.

Thank you for the reply. I can't go above 48GB.. honestly out of money to spend, but the m/b has 8 slots and that is what I have. I could increase to 128.. but man.. if 48TBs and 48GB of RAM is not enough.. then I am way over my head. This is mostly to play movies/tv on 1 or 2 devices in the house. In a couple years all the kids will be gone so it will literally be 2 devices tops, usually 1. The purpose of the large mount of storage is to try to store the maximum quality 4K HDR I can, to avoid needing to use bluray. I think my family photos/videos/etc is about 3TBs or so in size, and I have a 2nd Synology NAS (which I may just get rid of and move family stuff to a couple HDD backups and store them in safe.. which I need to do anyway) rather than waste more power on a NAS that wont be used much. May look to sell it as it has 5 8TB HDDs in it as well. I only figured I could use it as an optional asset/K8s node server (and/or VM host) due to the amount of RAM and storage space.

Hope you figure out your issues.

 

[aufalien]

Ah I see...

Welp rubber meets the road is best so try it as is and see how well it plays.

Also 4K HDR playback depends on file format as much as hardware used.

I think you've plenty of hardware for a few 4K streams but of course like I said, it really depends on how that 4K HDR content was encoded.

And thanks for the well wishes.

 

[Constantin]

I found a metadata only L2ARC beneficial even if just for browsing more quickly through directories. With 48GB of RAM, you have more than enough to entertain a 512GB L2ARC. The usual minimum is 32GB of RAM.

I would simply try it out and see if there is a difference. A L2ARc can be removed at any time without issues. An Async=Always pool has its own issues, but to each their own. I also run a Z3 setup here, so I’m more cautious than default.

 

[aufalien]

I found a metadata only L2ARC beneficial even if just for browsing more quickly through directories.

Very interesting. I may try that one myself!

An Async=Always pool has its own issues, but to each their own.

What have you noticed?

 

[aufalien]

Beowulf,

Seeing that your server is largely read media, I'd disable atime as well.

While I like Constantins advice of using L2ARC for metadata only which seems to improve browsing, you probably won't have very large numbers of files in your directories. I have thousands in a single directory (image sequences) and my browsing across NFS seems fine using various UIs like Dolphin, Konqueror, Finder, Explorer etc...

But test and tune to taste.

 

[Constantin]

Maximizing the performance of the pool by disabling sync also opens you up to corruption during data transfers / writes should the power fail, etc while it’s happening.

I prefer allowing the OS to decide that which is sync required, and what is not sync required. But it all comes down to use case. My pool is primarily for dormant data, not scratch, etc. Hence the Z3, and so on.