amurray
Cadet
- Joined
- Dec 14, 2013
- Messages
- 5
Thought I'd share my own success story about how implementing a SSD based ZIL made a huge performance improvement in client-to-server file transfer performance (in my system, anyway.)
I'm by no means an expert, this is just how it solved my particular problem on my particular system, hope it helps someone. Be careful with your own data, and be sure to have a look at cyberjock's epic slideshow pinned at the top.
http://forums.freenas.org/index.php...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
I'm not new to FreeBSD, but am fairly new to the ways of ZFS. The ZFS Intent Log has always mystified me (and lots of other people it seems; it's easily the most misunderstood, argued-about feature in ZFS :)
My system is an ever-evolving home NAS project for all your typical uses (backup, media serving, etc.,) has 32GB RAM, 4 core AMD FX CPU, and a main pool made up of six WD RED drives in three mirrored pair vdevs. I'm serving both AFP and CIFS to a mix of Mac OS X and Win8 machines.
Performance has generally been amazing (easily saturating gigabit ethernet during large file transfers) but with one major exception: Trying to copy my Aperture (photo) libraries to the server would take 12+ hours per library (the largest is about 27GB). These libraries are actually just 'packaged' directory structures consisting of thousands of images, directories and XML files. In my case the library was 27GB and 306000+ files. The same thing was happening with the Windows machines (esp. my Steam library and its 600,000+ files).
Basically, big contiguous files transferred wicked fast, hordes of small files took forever.
After a LOT of research and reading here and elsewhere, I was using 'zpool status -v' to watch the pool IOPS and bandwidth, and zilstat since it seemed like there must be some kind of contention going on when transferring loads of small files TO the server.
http://www.richardelling.com/Home/scripts-and-programs-1/zilstat
zilstat showed almost no ZIL activity when doing huge contiguous writes, but would start showing up to 90-100 synchronous I/O transactions per 1 second interval when writing thousands of smaller files. This happened with both AFP and CIFS. Looking at 'top', I would see afpd/smbd often showing a state of 'zil' or 'zio'.
Everywhere I read about the ZIL said something a bit different. "It's only for synchronous writes / It's only for databases / Home NAS users don't need it / It is/isn't a write cache / It's bad for performance / It'll give you gas" and on and on.
Finally, I read this article and it started to make some sense.
http://constantin.glez.de/blog/2010/07/solaris-zfs-synchronous-writes-and-zil-explained
As I now understand it, the ZIL is a non-volatile, store for recording synchronous transactions which have been acknowledged by ZFS but not yet flushed and committed to the pool's disk(s). Hence the term 'Intent Log'. It's not a cache, because the only time it is ever read from is in the event of a power failure or crash, in order to replay any transactions that didn't make it to disk. (It's actually a lot more complicated than that, but I think that is a reasonable summary.)
Going back to zilstat, I saw that amount of data being written to the ZIL (which by default, exists on-disk in your pool) was never more than 3-4 MB per 1 second sample in my case, but with these kinds of transactions going to disk (by default) at a rate of 90-100 times per second, I started to see what was going on.
I added a 2GB ZIL vdev (a partition on an old OCZ caching SSD) to the pool. Since the pending transactions in RAM that the ZIL references are written to disk every few seconds (I saw 5 seconds mentioned somewhere) the ZIL only needs to be big enough to hold a few seconds worth of synchronous write transaction info. zilstat will give you an idea of how many IOs and how much data the ZIL is seeing in your pool.
With the ZIL added to my pool, doing the same file copy that used to take 12 hours now takes less than 1 hour. zilstat shows that the synchronous IOPS being handled have gone from 90-100 up to 900-1200. Using 'zpool status -v', I didn't ever see the allocated size of the ZIL go above 120MB, and that it is indeed being cleared every few seconds. Another benefit I saw was that writes to the vdevs in the pool were now 0 in between pool write operations. Before, they were all constantly busy doing (small) writes, which I guess now frees up the controller to do more/other things.
So now, instead of having to wait for synchronous transaction info to be written to the ZIL somewhere on disk in the pool before sending the ACK back to the client, the ZIL now exists on a much faster, dedicated store (the SSD) and can send back ACKs as fast as it can write to the SSD. The sooner the client gets the ACK, the sooner it'll send more stuff. In my case it sped up my slowest file transfer examples by around 10x. I was amazed at the improvement. Another less quantifiable benefit is that everyday tasks like moving and deleting files and just navigating around the filesystem from the client side feels smoother and more responsive (I can see little bursts of activity in the ZIL as I move and delete things in the directory structure.)
Of course, you'll want to put your ZIL on a device that is write-optimized, has high IOPS, and is non-volatile. I can really see how a normal SSD could die a quick, horrible death due to the constant writes (the one I happened to have around was one of those old SATA III OCZ Synapse caching SSDs which was actually 128GB in size but was provisioned in firmware to show only 64GB, leaving the rest for reallocation.) You can also mirror ZIL devices and/or stripe multiple mirrored ZIL devices for whatever performance and reliability levels you might need.
ANYWAY, hope this helps someone, not sure if this is the 'proper' way to address the problem, but I know it made a huge improvement on my system, and I learned a LOT about ZFS in the process.
AM
(Edited to clean up some terminology and further clarify some points.)
I'm by no means an expert, this is just how it solved my particular problem on my particular system, hope it helps someone. Be careful with your own data, and be sure to have a look at cyberjock's epic slideshow pinned at the top.
http://forums.freenas.org/index.php...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
I'm not new to FreeBSD, but am fairly new to the ways of ZFS. The ZFS Intent Log has always mystified me (and lots of other people it seems; it's easily the most misunderstood, argued-about feature in ZFS :)
My system is an ever-evolving home NAS project for all your typical uses (backup, media serving, etc.,) has 32GB RAM, 4 core AMD FX CPU, and a main pool made up of six WD RED drives in three mirrored pair vdevs. I'm serving both AFP and CIFS to a mix of Mac OS X and Win8 machines.
Performance has generally been amazing (easily saturating gigabit ethernet during large file transfers) but with one major exception: Trying to copy my Aperture (photo) libraries to the server would take 12+ hours per library (the largest is about 27GB). These libraries are actually just 'packaged' directory structures consisting of thousands of images, directories and XML files. In my case the library was 27GB and 306000+ files. The same thing was happening with the Windows machines (esp. my Steam library and its 600,000+ files).
Basically, big contiguous files transferred wicked fast, hordes of small files took forever.
After a LOT of research and reading here and elsewhere, I was using 'zpool status -v' to watch the pool IOPS and bandwidth, and zilstat since it seemed like there must be some kind of contention going on when transferring loads of small files TO the server.
http://www.richardelling.com/Home/scripts-and-programs-1/zilstat
zilstat showed almost no ZIL activity when doing huge contiguous writes, but would start showing up to 90-100 synchronous I/O transactions per 1 second interval when writing thousands of smaller files. This happened with both AFP and CIFS. Looking at 'top', I would see afpd/smbd often showing a state of 'zil' or 'zio'.
Everywhere I read about the ZIL said something a bit different. "It's only for synchronous writes / It's only for databases / Home NAS users don't need it / It is/isn't a write cache / It's bad for performance / It'll give you gas" and on and on.
Finally, I read this article and it started to make some sense.
http://constantin.glez.de/blog/2010/07/solaris-zfs-synchronous-writes-and-zil-explained
As I now understand it, the ZIL is a non-volatile, store for recording synchronous transactions which have been acknowledged by ZFS but not yet flushed and committed to the pool's disk(s). Hence the term 'Intent Log'. It's not a cache, because the only time it is ever read from is in the event of a power failure or crash, in order to replay any transactions that didn't make it to disk. (It's actually a lot more complicated than that, but I think that is a reasonable summary.)
Going back to zilstat, I saw that amount of data being written to the ZIL (which by default, exists on-disk in your pool) was never more than 3-4 MB per 1 second sample in my case, but with these kinds of transactions going to disk (by default) at a rate of 90-100 times per second, I started to see what was going on.
I added a 2GB ZIL vdev (a partition on an old OCZ caching SSD) to the pool. Since the pending transactions in RAM that the ZIL references are written to disk every few seconds (I saw 5 seconds mentioned somewhere) the ZIL only needs to be big enough to hold a few seconds worth of synchronous write transaction info. zilstat will give you an idea of how many IOs and how much data the ZIL is seeing in your pool.
With the ZIL added to my pool, doing the same file copy that used to take 12 hours now takes less than 1 hour. zilstat shows that the synchronous IOPS being handled have gone from 90-100 up to 900-1200. Using 'zpool status -v', I didn't ever see the allocated size of the ZIL go above 120MB, and that it is indeed being cleared every few seconds. Another benefit I saw was that writes to the vdevs in the pool were now 0 in between pool write operations. Before, they were all constantly busy doing (small) writes, which I guess now frees up the controller to do more/other things.
So now, instead of having to wait for synchronous transaction info to be written to the ZIL somewhere on disk in the pool before sending the ACK back to the client, the ZIL now exists on a much faster, dedicated store (the SSD) and can send back ACKs as fast as it can write to the SSD. The sooner the client gets the ACK, the sooner it'll send more stuff. In my case it sped up my slowest file transfer examples by around 10x. I was amazed at the improvement. Another less quantifiable benefit is that everyday tasks like moving and deleting files and just navigating around the filesystem from the client side feels smoother and more responsive (I can see little bursts of activity in the ZIL as I move and delete things in the directory structure.)
Of course, you'll want to put your ZIL on a device that is write-optimized, has high IOPS, and is non-volatile. I can really see how a normal SSD could die a quick, horrible death due to the constant writes (the one I happened to have around was one of those old SATA III OCZ Synapse caching SSDs which was actually 128GB in size but was provisioned in firmware to show only 64GB, leaving the rest for reallocation.) You can also mirror ZIL devices and/or stripe multiple mirrored ZIL devices for whatever performance and reliability levels you might need.
ANYWAY, hope this helps someone, not sure if this is the 'proper' way to address the problem, but I know it made a huge improvement on my system, and I learned a LOT about ZFS in the process.
AM
(Edited to clean up some terminology and further clarify some points.)
