Ok I want my expectations to be in line with reality. It also was not my intention to come off defensive, cocky, or all knowing. You are a wealth of information and I believe I need to ask the right questions with enough supporting information to get an adequate response. I have read about ZFS and more specifically FreeNAS forums for 1000s of hours over many years. What is apparent to me now is ZFS must have overhead that is not well defined in the FreeNAS manual, Oracle’s white papers, or FreeBSD knowledge base.
ZFS does have overhead. I generally tell people "ZFS protects your data at virtually any and all costs". Those costs translate into needing more power than just some old Pentium 4 with 512MB of RAM. The more performance you want, the more hardware you'll need to buy. It is linear to a point, then it becomes exponentially more expensive.
It is my fault for not including my use-case and specs in my previous reply. I’m working on my signature. I honestly did not want this thread to turn into my specific problem/situation and wanted to keep it focused on SLOG devices.
Unfortunately, use case is extremely important. In fact, it may be the most important thing in determining if you even can use an slog. If you plan to use CIFS exclusively, adding an slog will be pointless. The hardware won't even be accessed.. ever. So very, very, very important. Cannot be understated.
I am very interested in the interworking of ZFS to accurately hypothesize the hardware required for a desired result.
I unfortunately will disappoint you. This forum really isn't a good place to get that info. It is *a* place to get the info, but that's not something I'm about to jump into on this forum except in small quantities. I'm not into writing books and posting them on the forum. ;) Sorry.
Your quote above had the most striking impact of my fundamental core understanding of ZFS. It is daunting to me not to see how things like working set, ARC, arc hit cache ratio, etc. are not relevant. In other words, I think they are the most relevant aspect.
They aren't. And as you learn more about ZFS you'll figure out what stuff is important and what's not. Nothing wrong with being wrong (and I'm wrong sometimes because of stuff constantly being added and/or changed in OpenZFS).
This conveys to me the speed at which data is hosted is directly attributed to RAM size regardless of working set.
Totally incorrect. If your working set (regardless of the size) can fit in the ARC (regardless of the size of the ARC) you will see maximum performance. Why? Because all of the regularly accessed data is in RAM. How much faster do you want than grabbing data from RAM? But, if the working set is even 1% bigger than the ARC, there is a significant performance hit. It's also an exponential cliff.
Why does a real world experience have this limitation? What is this elusive overhead? How much stuff was stored? What percentage of the pool was being utilized? Why would this system not have throughput around 3.5 GByte/sec?
For that you'd have to understand all of the underpinnings of ZFS, etc. Far beyond the scope of a forum post.
Similar to the OLTP world, ZFS utilizes a comparable system called adaptive replacement cache (ARC). This methodology utilizes an ingenious algorithm to cache recently used and frequently used pages. The pages consist of two groups of data, metadata and working set. The metadata portion is similar to an allocation table. This is commonly calculated at 1GB for every 1TB of data. The default maximum metadata utilization is 1/4th of the ARC. The other 75% of the ARC holds your working set which is the actual cached reads from the pool. These values of 75% and 25% are minimums and maximums respectively and are dynamic. I.e. if you don’t have many files then the metadata portion of the ARC won’t utilize the maximum of 1/4th of ARC.
Not entirely correct. It is possible for the metadata to exceed that 1/4 value (either because you change the tunable) or because the system is stressed and ZFS has decided to go bigger.
If you are not getting enough page hits (too many direct reads from spinning rust) you’re probably ARC/RAM deficient. This is represented in a poor hit cache ratio. You could possibly add L2ARC however, it requires metadata as well which eats a portion of your ARC. Not all of your RAM is dedicated to ARC. FreeNAS has OS, jails, and other stuff that requires resources. My guess is the OS overhead is typically around 1 to 4GB.
Basically, this is correct. There's caveats and exceptions, but basically that is correct.
If the above is true, and you have a sufficient processor, then why if a file is cached in ARC is the bottleneck not your network/FC interface? I mean, what else is holding the system back?
You are limited to your weakest link. What if you have 20ms of latency between the server and the system? Your file server is directly affected by performance of other externalties. You've also oversimplified things because things like sync writes will directly interrupt any other work in process on the zpool to update the slog/zil. If you have no slog you just interrupted whatever other work was in progress to handle a sync write. If you work at a desk and every 10 minutes you get a phone call that interrupts you, even if its just for a minute or two, doesn't that slow you down more than just the time of the call? ZFS works the same way.
To finish my point and give you and my fellow forum members something to poke holes in. Let’s create a fake use-case. We have a bare metal server running mssql locally. The OS is installed on local drives. The data drive on this server is hosted on a FreeNAS box utilizing 10GB iSCSI. This SQL server has one set of 4 databases totaling 8GB in size that reside on the FreeNAS box. The FreeNAS box is not hosting anything else. It has an adequate SLOG device to keep up. It has 2 mirrored vdevs using standard 10k SAS drives. The drives are 146GB each. This yields ~230GB useable disk space with overhead. The storage usage is about 3.5%, so poor performance from iSCSI fragmentation should not be an issue. If we use the methodology above and some slight fudging would the equation below not make sense?
4 (OS overhead) + 1 (metadata) + 11 (working set+ some) + 32 (iSCSI overhead) + 16 (why not) = 64GB of ram on FreeNAS
Would this not yield kick ass database performance without 512GB of ram? If the performance degradation lies with iSCSI screw it use NFS! Why would you not expect to see over 1GB of throughput here? I believe in this situation the ghosts lists won’t have any hits even with iSCSI
flaws.
In your hypothetical example, you have to read the data from the zpool for it to be in the ARC, and it has to have not been expired from the ARC. Have both of those conditions been met? If not, how do you make those conditions met. I can tell you that nobody has a system that magically has the SQL databases loaded into ARC from the get-go. So someone has to take that penalty and do the reading and/or writing.
So lets assume you need to read the entire database into the ARC, but the seek times of those 10kRPM drives will be your limiting factor. Solution: Go all SSD. Now thing are going to be much faster because from the time your SQL machine makes the query for data from the tables, to the time it goes over iSCSI, NFS, CIFS, etc, to the time it takes the zpool to read the data from the zpool's spinning rust (or the ARC if you are lucky) plus the time to respond back through the protocol, network stack, network infrastructure, you've just lost precious ms that the SQL database spent sitting around
waiting for data.
Cyberjock, you answer me so fast. I have a busy life with 3 kids and must administer the infrastructure at my employer’s company. My company unfortunately does not use ZFS. Please take a few days or a week to get back to me. No rush! How is your book going? I would like to buy it.
Yeah, the book is on hold. Life is just too busy for me. Heck, I'm having to force myself to make time for the forums lately. :P
HTH!
