To Slog or to Sync=Disabled for a Read only pool of stagnate data (Media Server)

[Jaslo]

First off, this is for home. This is not something getting pounded every minute with transfers. This is just for storage of backups, my families media, and when I'm moving data around for people fixing their PC's and doing backups for them and such. 90% of the servers resources are used to READ from 2 data pools. I have read the forms, and posts on various sites and I have a good understand on the ZFS technology however, I feel my question is in a gray area that's not really touched upon in posts due to my data on my freenas being stagnate / data that is only read from.... So I'm trying to optimize my Freenas Build, for what I am using it for, with the hardware I have as I am limited to 16GB RAM due to the Motherboard and a UPS and SLOG's are cheaper then upgrading MB, Chip, and RAM and a happy wife is a way better life.

Question:
For optimal performance in my case, can I safely ZFS Sync=Disabled on my 2 data pools that 97% of the time are only read from to give better throughput when writing to these pools and reading from them without the chance of losing data? My Emby (Media Server) Jail would be reading from these Data Pools and if Transcoding is needed, it would be done on my SSD pool of which is housing the Jail. Data that would be written to these pools would be coming from a PC / Drive other then the Freenas server so therefore, if the power went out while copying or the freenas server crashed for some odd reason while copying, the Data already on the drives would be safe?!? It would just be the data I was copying over during the time of the outage that I would potentially need to recopy as those files would be incomplete or corrupt.

If there is a risk to my existing data, I don't want to put that data at risk. Would a SLOG (Small SSD / optane / pcie x1 nvme) and a UPS system(due to lack of supercapacitor) be a better alternative due to my Hardware RAM being maxed? OR is there some alternative I should do to get the most performance with data safety in mind? (PS, in the future, pools will be rebuilt to ZFS2 when i get some more drives for Xmas and my birthday hehe)

System:
Forgive the Frankenstein build... It's what I had, and it works, and my wife inst complaining about money....
Freenas 11.3 u2.1
AMD Phenom II x4 970
16GB DDR3 RAM
LSI Sun pcie2 x16 IT Mode SaS (4 ports used)
onboard 6x sata3 (all 6 used)
onboard 2x sata2 (unused)
Rivo PCIe SATA Card, 8 Port Marvell 9215 Non-Raid pcie x1 (future expansion.... tested 7 500gb drives to make sure it worked in zfs2)
3x Roswell 4bay hotswapable
1x 4bay SSD

Pools:
Stagnate Data pools (possibly doing Sync=Disabled):
Pool 1 (most important data) = 4x 3TB Iron Wolf ZFS1
Pool 2 (extra storage / secondary data) = 4x 4TB Barracuda ZFS1
Pool 3 (My Jails (only a small Emby Server) Sync=Enabled) = 2 Crucial Bx500 480GB SSD Mirrored

Back story:
Primary use is to store data such as my wifes, families, friends pictures, videos, files, with the odd backup when i am repairing somesone's pc or im migrating from one pc to the other. The only Jail running on it is a freebsd running Emby for the kids and wife so we don't have our kids skating across the floor with DVD's and blurays shouting look daddy, I'm like Elsa. I guess you can say its overkill for what I use it for and why don't i use a External... well you can ask my wife about when she dropped my 4tb lacie, or when the dog jumped up on my desk trying to get a snack and knocked my 8tb seagate to the floor while I was at work, or when my 3 year old thought my 8TB external was a toy and dropped it down the stairs with his paw patrol truck while I was at work. The clouds open up and the higher power says, they hate me and to get a real storage system.... my life story.... externals are bad.... freenas is good

Since i said F' it and made a Thermaltake Armour case into my NAS B**** with the hot swappable drive bays, I have had 0 issues with " act of god hardware failures" and I love every moment of it for over a year now. (Probably mostly due to the sheer size of the case, it weighing over 80 pounds, and is out of the kids and dogs reach and the wife has a restraining order of 5 meters from it)

 

[Yorick]

Well, what’s sync writing to it now? This is all SMB shared? Any Macs that are accessing it?
The macs would use sync write, and it’s okay to disable that on the SMB service itself. Or the dataset if you feel like it.

Windows machines won’t sync write in the first place, so the issue never arises.

I don’t see NFS or iSCSI, so that’s not a concern.

You might be able to just do nothing and achieve the same result :)

 

[PhiloEpisteme]

For optimal performance in my case, can I safely ZFS Sync=Disabled on my 2 data pools that 97% of the time are only read from to give better throughput when writing to these pools and reading from them without the chance of losing data?

I mostly agree with @Yorick except that where a system wants to use sync writes your question above is technically going to have an answer of "no". If you could use sync writes and choose not to; there is the change chat a power loss or similar could cause you to lose a transaction etc. Depending on what data you are sending that might matter; or it might. The right answer will invole a certain amount of what you plan to use the system for. In your case it sounds reasonable to not use sync writes and avoid a SLOG, enjoy better performance, and accept the very small risk.

Pool 1 (most important data) = 4x 3TB Iron Wolf ZFS1

FWIW, I think its worth mentioning that for any data you care about a good backup strategy is important. :)

 

[Jaslo]

So the emby Jail is the one doing all the work streaming via DLNA and https. Any transcoding would be the emby server (SSD jail on the freenas) copying the file to the local jail, transcoding it, then distributing it. The jail has the Mount points set from the pools. when i run zilstat -t 5 i do see writes and its not a constant 0. especially if i am streaming. this is when i started accessing my media server and attempted to stream a mp4 santa message.

[~]# zilstat -t 5
TIME N-Bytes N-Bytes/s N-Max-Rate B-Bytes B-Bytes/s B-Max-Rate ops <=4kB 4-32kB >=32kB
2020 Apr 30 20:05:47 0 0 0 0 0 0 0 0 0 0
2020 Apr 30 20:05:52 0 0 0 0 0 0 0 0 0 0
2020 Apr 30 20:05:57 273376 54675 134160 1540096 308019761856 376 376 0 0
2020 Apr 30 20:06:02 180512 36102 124216 1028096 205619704512 251 251 0 0
2020 Apr 30 20:06:07 213480 42696 99424 1216512 243302569344 297 297 0 0
^C


Copying to and from the shares would be SMB as thats easiest for me due to my setup and the pcs I work on. So for my setup, there is no advantage period from a zlog or anything. My performance for this system is at the mercy of the drives, sata controllers, and ram and 0 benefit from a cache?

 

[Jaslo]

Here is the Zil a few mins into the mp4


2020 Apr 30 20:14:23 277432 55486 138144 786432 157286393216 6 0 0 6
2020 Apr 30 20:14:28 109752 21950 101992 393216 78643262144 3 0 0 3
2020 Apr 30 20:14:33 162984 32596 90840 786432 157286524288 6 0 0 6
2020 Apr 30 20:14:38 442280 88456 294280 1048576 209715655360 8 0 0 8

not sure if this helps with any recommendations......

 

[PhiloEpisteme]

My performance for this system is at the mercy of the drives, sata controllers, and ram and 0 benefit from a cache?

Read performance benefits greatly from cache and that cache comes in two forms; the ARC and L2ARC. The ARC is something that exists on all systems and it lives in memory; it is extremely fast. The L2ARC is an additional device you can add which expands the read cache capacity of your machine at the expense of consuming some ARC to keep an index of the L2ARC. This is a bit of a simplification; but should get the point across.

Writing has no such cache. The ZIL is not a cache; it is the ZFS Intent Log and it stores transaction groups prior to when they have been fully committed to the pool. This is extremely useful when you're using sync writes and don't want to wait for your system to calculate checksums etc of all of the data ON TOP of writing the data to permanent store. In the event of a power failure before data was written from ram to the pool the ZIL can be used to get that data back.

A SLOG is a Separate zfs intent LOG. Simply put; it is when you don't want your ZIL to live on your pool for performance reasons so you purchased a different device to serve as the ZIL.

So, to be a bit strict about terminology; no, your write speeds will not improve from any sort of cache because there is no write cache. :)

If you're seeing a ton of ZIL usage and your write speeds are not where you want them you have two easy options.
1. Disable sync writes. This will give you the best performance; at the expense of the protection the ZIL provides.
2. Add a fast, PLP SLOG device. This will not be quite as fast as option 1 but should give you write performance for sync writes.

Note that a SLOG will do nothing if you're not using sync writes or seeing a ton of ZIL usage. Nor will it do anything for the read side of things.

One way to experiment is to do your typical work flow with sync writes turned completely off and check your performance. If it is significantly better you may want a SLOG or to turn off sync writes. Which one you choose depends on how much you care about sync writes and whether you can disable them.

 

[Yorick]

I don’t see how streaming would ever tax the write capabilities of the system, we are talking 20Mbit/s or less. I also don’t expect Emby to use sync writes. As long as the dataset isn’t set to force sync, you are all set there. Lastly, this is an SSD, which means if there were sync writes, and those were slowing things down, you’d need an Optane or something along those lines for a secondary intent log, something that’s considerably faster than the ssd itself.

Let’s try this angle: Is there a performance bottleneck you are observing when you are streaming, or is this just idle curiosity?

You say emby is transcoding: from which codec to which, and what is the cpu utilization when it does?

 

[PhiloEpisteme]

Let’s try this angle: Is there a performance bottleneck you are observing when you are streaming, or is this just idle curiosity?

This is a great approach. Identify the bottleneck and then fix it. :)

 

[Jaslo]

To be honest I don't have any real bottlenecks or major issues. I'm just looking for advice on what I can do based on my usage and main hardware to optimize this freenas server build to get the best performance out of it as possible. I know RAM is the biggest and then my processor. But adding arc or l2arc would be cheaper for me to do the afull system overhaul. So if a arc or l2arc would benefit my emby and storage ill do that and sorry for being a noob when it comes to this... I have been reading and experimenting for the last 2 years and now I'm at the point of wanting some expert direction to go in without spending massive amounts of money. (I can get away with a 500 budget without my wife killing me hehe) so I just want to make sure I'm spending it in the right area.

In the future I'm going to add another ironwolf and do zfs2 on my pool1 (I have a 2 externals that I do a backup after major changes then take the externals and lock them in a cabinet just incase) im going to create a new pool in the future of Iron wolfs or equivalent in zfs2 to replace my pool 2 in the future. Then use pool 2 will be reused as my temp storage for migrating storage and backups and so on. The biggest backup I've had to do of someone's server was 5tbs. So 4 x 4tb drives in zfs would be fine for temp storage.

All help and pointers is greatly appreciated and thank you Yorick and PhiloEpisteme for you taking the time to breakdown and explain some things to me to help me understand the directions and pointers your recommending and why. GREATLY APPRECIATED

 

[Yorick]

More ARC is not likely to help at all. Easy to check: Run zfs-stats -a from SSH command line and look at the ARC stats. Unless you have a lot of ARC misses, more RAM isn't going to help read speeds. It will, however, allow you to run bigger VMs and such :).

In the future I'm going to add another ironwolf and do zfs2 on my pool1

I am assuming you already know this, and: That will entail taking a backup of pool1 on your externals; destroying pool 1; recreating pool1 as raidz2; and restoring from backup.

 

[Jaslo]

More ARC is not likely to help at all. Easy to check: Run zfs-stats -a from SSH command line and look at the ARC stats. Unless you have a lot of ARC misses, more RAM isn't going to help read speeds. It will, however, allow you to run bigger VMs and such :).




I am assuming you already know this, and: That will entail taking a backup of pool1 on your externals; destroying pool 1; recreating pool1 as raidz2; and restoring from backup.

Yes sir. Plan is when I have the extra drives on hand, do a full backup to my external, destroy the pool and then recreate with the new drives in ZFS2.

When I started out I put dummy data on the pools and practiced the do's and don'ts and simulated drive failures and how to resilver and restore a degraded pool, swap drives and setup the automated short and long smart checks and scrubs and such before I put any important data on the pools. Played around for about a month and did lots of scenarios just so I was comfortable and knew if I do this, this is the result lol.