TrueNas scale ssd cache

[HITMAN]

Hello Guys,

I’m building my new home NAS and I’m going to use the Scale version. It will be mostly used for NextCloud, Plex, TimeMachine, torrent (qbittorrent) and as a backup server.
I’m going to use “mirror” option for vdev.
I checked couple of guide and can’t still decide is it good idea to use my nvme drive as cache disk for that mirror vdev. Mostly writes coming from uploading media to nextcloud and downloading torrents. Reads I guess mostly from plex.



Any ideas are welcome,

Thanks!



My hardware is:

MB: ASUS PRIME A320M-K, Socket AM4
CPU: AMD Ryzen™ 5 1600AF, 3.6GHz, 19MB, Wraith Stealth Cooler
RAM: 32GB
Disk: 120GB SSD Transcend MTS820 nvme
Disk – Boot: 128GB A-data
Storage 2 x 4GB WD Red

 

[jgreco]

We don't generally recommend adding L2ARC (what you're calling "cache") until you have 64GB or more of ARC, because it is generally difficult for ZFS to pick out traffic patterns within a limited amount of RAM. At 32GB, it's possible but questionable. L2ARC isn't useful if you're not storing relevant stuff in it, and since L2ARC eats into the ARC for the pointers, it hurts you if you're significantly too small. If you already HAVE the disk and want to try, there's zero harm in trying and watching the ARC/L2ARC stats to see what the outcome is.

 

[HITMAN]

Thanks jgreco,
i have the disk laying around so i will do some testing and post here. I opted for 32GB ram because most of the time there are 10-20 images or 1-2 small clips are uploaded to NextCloud. Maybe download 1-2 tv shows a month and watching plex 1 very rarely 2 concurrent streams. Backups are weekly 1x full / 3x Incremental (size ~80GB). All of the above at the moment working on Qnap TS251+ 8GB ram Celeron j1900.
I just want to set it and forget for hardware changes next 5-6 years. :)

 

[twhiteWC]

Thanks jgreco,
i have the disk laying around so i will do some testing and post here. I opted for 32GB ram because most of the time there are 10-20 images or 1-2 small clips are uploaded to NextCloud. Maybe download 1-2 tv shows a month and watching plex 1 very rarely 2 concurrent streams. Backups are weekly 1x full / 3x Incremental (size ~80GB). All of the above at the moment working on Qnap TS251+ 8GB ram Celeron j1900.
I just want to set it and forget for hardware changes next 5-6 years. :)

I'm interested in your setup. What did you end up doing? I know L2arc is not really what most people hope for in a home NAS with the cost of entry maxing out the ram on a lot of boxes.

I've heard of some adding LVM Write cache or Write-back Cache to underlying volumes and importing those LVs into the pool instead of raw disks. And with the limitations in ZFS Caching (and the low cost of high endurance, high performance Gaming NVMe drives) that may become more of a thing, though it is fairly complex and discouraged by many.

Just wondering where you landed, now a year out.

 

[jgreco]

I've heard of some adding LVM Write cache or Write-back Cache to underlying volumes and importing those LVs into the pool instead of raw disks.

DO NOT DO THIS. You risk catastrophic pool corruption, for all the same reasons you are not supposed to use a RAID controller with cache. ZFS carefully structures writes and cache flushes out to disk in order to maintain the consistency of the pool, and adding additional layers in between is a really bad idea.

 

[HITMAN]

I'm interested in your setup. What did you end up doing? I know L2arc is not really what most people hope for in a home NAS with the cost of entry maxing out the ram on a lot of boxes.

I've heard of some adding LVM Write cache or Write-back Cache to underlying volumes and importing those LVs into the pool instead of raw disks. And with the limitations in ZFS Caching (and the low cost of high endurance, high performance Gaming NVMe drives) that may become more of a thing, though it is fairly complex and discouraged by many.

Just wondering where you landed, now a year out.

Hello,
sorry for the late reply i'm not using any cache as per Jgreco advice, all is working well, no problems with NC, plex, traefik, qbittorent,minecraft.
Never had to restart cause of low memory or etc. Even Realtek NIC which is installed on NAS between my Pc is working with no problems so far.

P.S. Never use middle-man between ZFS and DISKs, meaning HW raid, not passing disks if you are using Truenas in VM etc.

 

[twhiteWC]

Thanks for the feedback! So 32gb RAM, 2x7k and no L2arc, and you're having really good performance?

 

[twhiteWC]

DO NOT DO THIS. You risk catastrophic pool corruption, for all the same reasons you are not supposed to use a RAID controller with cache. ZFS carefully structures writes and cache flushes out to disk in order to maintain the consistency of the pool, and adding additional layers in between is a really bad idea.

Thanks for the warning. Yeah I see this discouraged almost everywhere it is suggested. At the end of the day no one cares how well things ran for a time if it ended up crashing badly and losing data or causing a bunch of downtime. I'm sure eventually there will be better options to tier storage in ZFS and maximize larger SSD volumes, rather than just continual recommendations to max out your ram and avoid L2arc. Right now it makes a great system look a little behind. But again most of us value reliability over raw performance.