BUILD Different SSDs in a mirror. A good idea?

[Chicken76]

I plan on building a file server at work that's going to host a few hundred thousand files that need to be accessed fast. The data is currently shared by a windows server on regular hard drives. The speed is acceptable when a single user reads or writes a big file (several hundred megabytes), but when a few users have to access hundreds of small files each, simultaneously, things get really slow. I feel the only way forward is to move to SSDs and something dedicated, like a FreeNAS machine.

Now I'm fairly accustomed to FreeNAS, having used it for the last couple of years, but my experience with it is limited to using regular hard drives; which brings me to this post. I'd like to know:

1. if there are any gotchas with FreeNAS and SSDs,
2. and is using different types (as in manufacturers) of SSDs in a mirror a good idea? I know someone who created a mirror out of identical SSDs only to find one morning the array completely blank. It turned out that the drives had a firmware bug that in some cases after a cold start could revert to factory state (a whole lot of zeroes). The drives were certainly not the cheapest and from a very reputable manufacturer (won't mention the name here). That's something I really want to avoid, and that's the reason for asking this. I understand that in a mirror, the write speed can't exceed that of the slowest drive.

 

[jgreco]

Need to be *read* fast? Or read and written?

Because if they merely need to be read fast, ZFS offers a lot of potential to make that virtually instantaneous in a different way.

First you need to determine the working set size, that is, the amount of data that's being accessed. You can estimate it by taking the average "small file size" and then multiplying by "a few hundred thousand".

ZFS will happily cache data in its ARC in main memory; if your ARC is big enough, everything will come out of the ARC once it's been read once from the pool after boot. But the ARC has to be big for that to happen. Modern machines can easily accommodate 32GB of RAM, so if your small files only sum maybe 20GB or thereabouts, just make sure you have lots of RAM. If you need to go beyond that, there's L2ARC, which is SSD used in this kind of scenario.

Note that both ARC and L2ARC do require that data be pulled from the pool at least once after boot, so a reboot of the fileserver can make things temporarily slow while the ARC "warms up." But once it is warmed up, it is incredible.

Let us know how much data you're talking about.

For reads and writes, yes of course you can also opt to make a pool with SSD's, but then you might not be able to store your big files alongside the little ones, because SSD is *expensive*.

 

[Chicken76]

Thank you for your input on this.

Let me explain how I reached the conclusion that using only SSDs makes sense in my case:
Firstly, the current data amounts to about 300 GB. In the future it might grow in size and number of files, but I can see myself keeping it to around 5-600 GB for the next 3-4 years with careful management and keeping data seldom accessed on the current server.

The "ZIL + L2ARC + regular drives + more RAM to compensate for the presence of L2ARC route" would amount close to the price of a couple of 1 TB SSDs, and performance would not match.

The question still remains: would there be a problem with using different SSD types? (different controllers, different NAND, different firmware and garbage collection, etc.)

 

[jgreco]

I'm a big believer in heterogeneous setups. Try to keep things not too wildly different (ie not SLC and MLC) but different controllers (Intel, sandforce?) seems prudent.