SSD Selection

[Mguilicutty]

Just curious as to whether anyone has looked at SSDs recently and gone through the process of choosing some for their servers. With all the different controllers available now a days there is quite a lot of info to sort through.

My use case in this instance will be to act as ISCSI system drives for VMware and Hyper-v virtual machines in a home lab. My thoughts were to use 4 ~250GB drives in a raid 10 setup. This should provide decent speed and easy expandability down the road if necessary. They will be connected via the onboard LSI2308 in IT mode.

I'm just beginning the sorting process myself and my initial thoughts are...

Powerloss capacitors don't really matter in my case as I'm on a (soon to be generator backed) UPS and this is just for home use.
Should drives have on board garbage collection? Does FreeNAS do trim?
Looking for something cheap but reliable.
Some SSDs work better with compressible data, should probably avoid these.
This array will see a lot of small reads and writes.
Each physical host will have onboard 120GB SSD for swap and such.

And here is too much information...

For now each host is connected to the FreeNAS box via Gb ethernet. The FreeNAS box has 4 Gb interfaces, 2 are lagged for regular access and 2 are reserved for MPIO ISCSI use. I'm hoping down the road to use Infiniband or similar. 10Gb is too expensive still for home use. This is all connected to a Cisco 3750 switch stack.

Normal load is two server VMs that are lightly used for home duties. Their data partitions rest on a separate array on the same FreeNAS box and are accessed via CIFS. VMs can balloon to 10 or more in some situations. The physical servers are Dell C6100 nodes, pci expansion is limited.

Besides my 2 laptops and 2 desktops, 2 phones and a few raspberry pi's I have a wife and three kids who each have phones and tablets and laptops and desktops and A/V gear that is all connected to the network. I was diagramming this all out the other day and it is insane.

 

[Mguilicutty]

Well, it looks like FN does support TRIM with the 9.2.0-Release per this thread. https://bugs.freenas.org/issues/1990

Another concern I had was SLC vs MLC and wear. It seems some manufacturers are minimizing this somewhat by using a caching buffer (like Sandisk's NCache) which also speeds up writes. Need to look into this further.

There are so many "good" options that are all priced at around $170 that I might just pick a few different ones and see what happens.

Now debating whether 2 ~500GB SSDs in a mirror would do just as well. I must admit I don't know much about measuring performance capacity on these systems yet but my goal is max iops and low latency rather than total throughput. I'll be limited to 2x 1Gb Ethernet connections until either FN and Infiniband get happy together or 10Gb prices come down. At this point I don't know what the choke points will be. I'm trying to match or beat 2 locally connected 500 Gb 7200 rpm drives on the C6100s, which are fine 80% of the time. When several VMs boot up or are doing disk intensive activities though things come to a crawl. I'm going the ISCSI route just so I can learn more about fail over and clustering stuff.

 

[herby]

I've been running a couple of Xenserver guests from an iSCSI zvol on a 128GB Crucial M4 and I've been pretty happy.

Monitoring trim from the CLI:

Code:

# sysctl kstat.zfs.misc.zio_trim
kstat.zfs.misc.zio_trim.bytes: 403263307776
kstat.zfs.misc.zio_trim.success: 30243943
kstat.zfs.misc.zio_trim.unsupported: 714
kstat.zfs.misc.zio_trim.failed: 0

I just ordered a pair of 240GB M500's on sale. I'm going to make a raidz with them and the M4 for about 220 GB usable. I'll replace the M4 in a little bit when I have some more fun money to spare.

 

[Mguilicutty]

Thanks for the info Herby. I saw the M500's at Microcenter the other day @ $150 for the 240GB. I was tempted to pick up a couple and may go that route myself. Just curious, why would you go raidz in this application over just a raid 1 or 10? I'd think that would slow things down a little.

 

[Mguilicutty]

Okay, I got to playing around with an SSD that has been sitting on my desk for a while, a Kingston HyperX 3K 120GB. I did a "smartctl -l error /dev/whatever" it was and it returned back "SMART Error Log not supported". Does this make these drives not a good choice for us? SMART is supported and enabled and I guess FN can still run tests and report the results so is this even an issue? I also noticed that it doesn't seem to save a temperature log like other drives I've tested though it does have the min and max temps recorded. On a side note, it is reporting 78C as it's max reached temp! Holy crap... Now that it is in the rack it is reporting and comfy 19C.

Am I making too much of this as a novice smartctl user?

 

[cyberjock]

Well, SSDs are a unique beast. SSDs don't seem to really run SMART tests in the traditional sense. They don't have the same hardware that SMART was designed for. Also, SMART never actually enforced any kind of criteria for a "short" or "long" test. Most manufacturers do a full disk read on long test, but anything else is really unknown unless you work for those companies and know what is going on. And if you are one of those people, you surely can't talk about it anyway.

Anyway, SSDs drive doesn't actually read all the memory cells looking for errors on a long test. For every drive I've seen a short test and long test takes less than 1 minute(some less than 10 seconds). I'm presuming that both are actually doing the same thing(whatever that may be) and the drive is simply supporting both tests because those are both common tests. Your average admin doesn't want to buy some SSD and get lots of error messages stating that the drive doesn't support SMART short tests.

To be honest, I'd probably do short tests every few days(scrubs as you'd normally schedule them) and leave it at that. Personally, there's only minimal value in running tests since most drives, if they start failing in a way that will be found in a SMART test, are almost certainly already causing other problems drawing attention to the drive.

 

[Dusan]

For every drive I've seen a short test and long test takes less than 1 minute(some less than 10 seconds).

I use ADATA SP900 (256GB) for jails and the short test takes about one minute, the long test takes 48 minutes. Of course, nobody knows (and ADATA won't tell) what it actually tests.

 

[cyberjock]

I use ADATA SP900 (256GB) for jails and the short test takes about one minute, the long test takes 48 minutes. Of course, nobody knows (and ADATA won't tell) what it actually tests.

That's interesting! Does the test time vary with the amount of data on the drive?

 

[Dusan]

That's interesting! Does the test time vary with the amount of data on the drive?

I remember the time was 48 minutes even when the drive was empty. Also, there is no SMART test log. smartctl -l selftest returns "SMART Self-test Log not supported".

 

[cyberjock]

I remember the time was 48 minutes even when the drive was empty. Also, there is no SMART test log. smartctl -l selftest returns "SMART Self-test Log not supported".

That seems kind of pointless. So it runs the test, but what if it fails? And how do you check the status of the test that is running? Really really weird. Maybe their 48 minute "test"(whatever it is doing) isn't better thought out as the other guys(woohoo for 1 minute test!). Hmm....

 

[Dusan]

You can check the progress in the smartctl -c section:

Code:

Self-test execution status:      ( 249) Self-test routine in progress...
                                        90% of test remaining.

After a successful run it reports:

Code:

Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.

smartctl code shows that the drive can report various failure modes of the last SMART test via this channel. However, I have no idea if the ADATA really works that way.

 

[herby]

Thanks for the info Herby. I saw the M500's at Microcenter the other day @ $150 for the 240GB. I was tempted to pick up a couple and may go that route myself. Just curious, why would you go raidz in this application over just a raid 1 or 10? I'd think that would slow things down a little.

I suspect it does slow things down a little bit, but I fell victim to raid greed. Too chicken for just a stripe, too cheap for a stripe of mirrors; although that's what I have for my "big" slow pool.

As an aside the 240GB M500s are popping up on sale in the $120 range lately so look for a deal before you bite.