SSD gotchas

[Oko]

I am getting ready to build another FreeNAS box and I would like to have some clarification with respect to use of SSD drives for OS. Namely my original intent is to have to embedded SSDs on which FreeNAS 9.3 will be installed on ZFS partition and mirrored. However as we all know SSD are not as great as PR departments would like us to believe. Just to name a few serious issues.

1. Flash chip storage blocks can only be [re-]written a finite number of times.
2. In order to deal with this and still have a usable product, SSDs use a fancy function in the controller called "wear leveling" which means that it keeps track of how many times the blocks have been written. My understanding is that as you [re-]written the performance exponentially decays.
3. Some drives also have spare blocks because one of the ways to kill SSD drive is to completely fill it.
4. There is a SSD function called "trim" with recent FreeBSD support. With a regular file deletion, all that the drive will be told is to clear some information from the directory and allocation bitmap (or equivalent metadata, depending on the filesystem in use). This means that the SSD will eventually "fill up" all of its storage blocks (due to the aforementioned wear leveling) and when it needs to write new data, it must first erase some storage blocks. That takes time, and it would be better to do in the background when the SSD isn't being heavily used.

I think I covered main problems with SSDs. On the another hand ZFS has its own gotchas which could benefit form SSD drives.

1. ZFS uses a logging mechanism, the ZFS intent log (ZIL) to store synchronous writes, until they're safely written to the main data structure on the pool. The speed at which data can be written to the ZIL determines the speed at which synchronous write requests can be serviced: The faster the ZIL, the faster most database, NFS and other important write operations become. Normally, the ZIL is part of the regular pool on disk. But ZFS offers the possibility to use a dedicated device for the ZIL. This is then called a "log device".
   By using a fast SSD as a ZFS log device, you accelerate the ZIL and synchronous write performance improves. Adding SSD as a ZIL seems no brainier.
   
2. ZFS also has a sophisticated cache called the "Adaptive Replacement Cache" (ARC) where it stores both most frequently used blocks of data and most recently used ones. The ARC is stored in RAM, so each block of data that is found in the RAM can be delivered quickly to the application, instead of having to fetch it again from disk. When RAM is full, data needs to be thrown out of the cache and is not available any more to accelerate reads. SSDs can be used as a second level cache: Blocks that can't be stored in the RAM-based ARC can then be stored on SSDs and in case they're needed, they can still be delivered quicker to the application than by fetching them again from disk. An SSD that is used as a second level ARC is therefore called an L2ARC, or a "cache device".
   L2ARC is less interesting to me personally because I like to put lots of RAM.

Now come my FreeNAS related question. My original plan was to use embedded SSD (ZFS mirror them for the OS). My understanding is that FreeNAS is NanoBSD based (mount only) which seems will help me with SSD gotchas 1 and 4. How does SSD 2 affects FreeNAS. How about SSD gotcha 3? Does FreeNAS include Trim support. How will root on ZFS be affected by use of SSD? Do I have to be aware of any additional issues? Should I use separate mirrored SSDs for ZIL in order to have performance benefit since OS will be mounted read only?

At the end I just want to make clear that I am planning to use regular industrial grade HDD for storage pool. As all of above is related to the installation of FreeNAS itself (particularly FreeNAS 9.3 which supports root on ZFS) and best recommended practices when it comes to SSDs.

 

[cyberjock]

First, I'm gonan preface this with "every SSD is going to be different, and some may be supported a particular way while others may not be supported at all". Unfortunately the manufacturer has the answers you may have, and they won't give you those answers (or they'll give you BS answers because they don't know what this "Z.F.S." thing is.

2. That is handled entirely internal to the controller, so it is independent of the OS, the file system used, etc.
3. Yep, and the SSD will (should) reallocate it automatically on-demand. If data "would have been lost" due to a block of data being unreadable ZFS should catch and repair it during a scrub, assuming you have enough redundancy to do so. (go RAIDZ2!)

The short answer is you've done your homework, there's nothing to worry about, and you should just use the drives as you are planning to. All will work out fine and all the tech will work exactly how you want (give or take trim, which won't be a big factor considering the boot device sees relatively little writes except when doing an OS upgrade).