bad performance, Crucial M500 (120GB) vs Intel 330 (60GB) as SLOG

[r00ster]

We are currently building a new NAS for our VMware ESXi servers, but we are getting very bad results from two new Crucial M500 (120GB) SSD's if we use them as SLOG devices (striped and mirrored).

Just to give you an idea what we are working with:
Build FreeNAS-9.2.1.3-RELEASE-x64 (dc0c46b)
Platform Intel Xeon CPU E3-1245 v3 @ 3.40GHz
Motherboard: SuperMicro MBD-X10SL7-F
Memory 32706MB (ECC)
HD: 6x WesternDigital WD2000FYYZ (2TB) in RAIDZ2 (so 7.1 TIB usable)
Controller: LSI 2308 (flashed to HBA mode) (all six HD's are connected to this controller)

We are doing some testing before we migrate to this new NAS, So we first started to do a baseline test using only the HD's without a separate SLOG device. We created a ZFS volume and shared this using an NFS to one of our ESXi servers. We installed a virtual Windows 7 x64 machine and did some benchmarking with HD Tune Pro 5.50 using the Random Access test.

Without an SLOG device

Transfer size Operations / sec avg. access time max. access time avg. Speed
512 bytes 115 IOPS 8.643 ms 51.265 ms 0.056 MB/s
4 KB 110 IOPS 9.071 ms 57.329 ms 0.431 MB/s
64KB 98 IOPS 10.182 ms 87.282 ms 6.138 MB/s
1 MB 6 IOPS 156.015 ms 493.247 ms 6.410 MB/s
Random 11 IOPS 83.850 ms 397.923 ms 6.051 MB/s

Then we wanted to know if we could increase the performance using an two year old Intel 330 SSD (60GB):

Transfer size Operations / sec avg. access time max. access time avg. Speed
512 bytes 2844 IOPS 0.351 ms 5.678 ms 1.389 MB/s
4 KB 2252 IOPS 0.443 ms 5.139 ms 8.798 MB/s
64KB 865 IOPS 1.168 ms 8.031 ms 53.476 MB/s
1 MB 95 IOPS 10.421 ms 185.222 ms 95.956 MB/s
Random 167 IOPS 5.966 ms 147.249 ms 85.03951 MB/s

It seemed we increased the performance by roughly tenfold, So then we started testing with two new Crucial M500 SSD's (120GB) which we wanted to run in mirror mode. we were expecting even better performance then the Intel 330 (60GB) but the results tell us a different story. The newest firmware for the Crucual M500 almost doubled the performance during our initial test but still... see below.

Transfer size Operations / sec avg. access time max. access time avg. Speed
512 bytes 212 IOPS 4.710 ms 11.323 ms 0.104 MB/s
4 KB 213 IOPS 4.690 ms 8.654 ms 0.833 MB/s
64KB 183 IOPS 5.458 ms 8.965 ms 11.451 MB/s
1 MB 13 IOPS 74.334 ms 138.768 ms 13.453 MB/s
Random 24 IOPS 40.888 ms 300.490 ms 12.409 MB/s

We tried everything we could think of to increase the performance.

• Updated the firmware
• Changed AHCI mode to IDE
• Switched SATA controllers
• Using only a single Crucial M500 (tried both)

We cannot explain the bad performance compared to the same test using an old Intel 330 SSD. If somebody has some insight or other things we can try we would love to hear your advice.

We are now thinking about moving on and buying a single Intel S3500 or S3700 SSD, or even the new Intel 730. Or getting a Seagate 600 Pro. But we can sleep a little better if we could find an explanation why these two Crucial M500 SSD's are performing so slow.

 

[eraser]

I believe that writes are generally sequential to a SLOG/ZIL. Here are the relevant specs for your two drive models. You'll notice that your Intel drive is much faster:

• Intel SSD 300 = 400 MB/s sequential write [1]
• Crucial M500 (120 GB model) = 130 MB/s sequential write [2]

Which firmware did you upgrade your M500 to? I believe the consensus out on the web is that firmware MU03 fixed many performance problems. I checked today and see that firmware MU05 was released last month. With that said, the M500 was never meant to be crazy fast - especially the 120 GB model.

The M500 read performance looks pretty good though... you could save those M500 and maybe use one as a L2ARC if your workload ends up needing one.

[1] http://ark.intel.com/products/67286/Intel-SSD-330-Series-60GB-SATA-6Gbs-25nm-MLC
[2] http://www.crucial.com/pdf/productFlyer/ProductFlyer-letter_Crucial_M500_SSD.pdf

 

[cyberjock]

So who's gonna tell the OP that benchmarks on zpools are hilarious to watch and that when it goes into production he'll realize that performance is a fraction of what those benchmarks claimed?

 

[eraser]

So who's gonna tell the OP that benchmarks on zpools are hilarious to watch and that when it goes into production he'll realize that performance is a fraction of what those benchmarks claimed?

Apparently you are. :)

Why do you make that claim (that benchmark numbers are wildly out of touch from real world performance)? (I'm not necessarily disagreeing).

My first guess would be that the Working Set Size (data in regular use) in production is usually much bigger than what a simple benchmark produces so not all of it will fit into the ARC. Therefore the NAS performance seen from the ESXi server will be closer to the performance of the underlying disks instead of the speed of the ARC.

Saying that though, his posted numbers without a SLOG are close to what I would expect from a single RAIDZ2 vdev (IOPS about equal to a single drive)....

edit: oops, should have said "withOUT a SLOG"

 

[cyberjock]

When you do your benchmark, it'll create some test set, then do read and write benchmarks on it. It will likely be a few GB at the most. ZFS will cache it all, so you're testing your ZFS cache speed. Unless you set the actual test set to something several times bigger than RAM the benchmark will be skewed because ZFS will cache as much as it can. Generally, if you can't do 5x your RAM(which is often extremely time consuming) the benchmark result will be artificially inflated. That's why I'm always telling people to put the benchmark tools down. They won't do what you think they are doing, and you won't get numbers that mean a damn thing in relation to reality.

Here's the disparity.. your benchmark dataset is certainly a small fraction of your ZFS cache. But your working set in a production environment will probably be multiple times what your ZFS cache is. That difference is night and day. There is no comparison at all and using those number is penny wise and pound foolish.

I can run benchmarks that can make you think your pool can do 4GB/sec, or I can run benchmarks that make you think it can do only 4MB/sec. With the same hardware. Nobody that looks to a forum for guidance will have the experience and knowledge to figure this crap out. If they did, they wouldn't be in a forum asking for help. ;)

As I've said many many times in this forum, if you plan to run more than a few VMs, your going to be looking at replacing your hardware. 32GB of RAM isn't going to allow you to do what you want to do. :(

As for SSDs, the only SSDs I recommend for ZFS are Intel drives. Not that others might not work, but they won't work as well. There's a thread where someone tested a few others, and on a loss of power ever drive failed except Intel.

 

[eraser]

Thanks for the reply cyberjock! Fully agree that the ARC will wildly inflate benchmark results if the test data is incorrectly chosen.

I think it would be best to calculate/figure out the worst case read performance of a pool (with little to no read cache help at all) and make sure that it meets the needs of the project. Then any speed increase from the ARC is just gravy. Or maybe find a way to disable the ARC for the test (perhaps by setting "primarycache = metadata" or "primarycache=none" on the dataset?). Thoughts?

What do you think about benchmarks for write performance though? Does increasing the amount of installed RAM affect write performance much (once you get past a sufficient amount)? If I recall correctly FreeNAS needs 3 txg's worth of memory for write caching and a txg is typically created every 5 seconds (3 x 5sec = 15 seconds). If one was able to write data to FreeNAS over a perfectly efficient gigabit network connection ( 125 MB/sec) and the zfs pool can write data that fast then it would require at most 1875 MB of RAM. Unless I am missing something... ?

 

[aufalien]

I must agree with CJ here. I've suffered horribly from analysis paralysis brought on by benching. Best thing I did was tweak while in production.

 

[cyberjock]

No, don't confuse transactions with transaction groups. FreeNAS writes a transaction very 5 seconds. So at most you'll need 5 seconds worth of throughput(or roughly 125MB/sec * 5). This is why I tell people that have Gb you don't need a big SLOG. Even a 1GB SLOG is massively oversized. Of course, you won't be able to buy a 1GB SSD that is a racecar.

My thoughts, stop and do what aufalien said. Just start using it and see how it performs. I will warn you though, don't expect high speeds with a system of your caliber. What you are likely about to do is going to result in an very very slow box, and you're going to wonder why the VMs suck so badly. 32GB of RAM just isn't enough when your working set is 2+ magnitudes of that. I do offer consultation for stuff exactly like what you are doing. If you want to go that route, PM me. My guess is that even if I tripled my price, you'll save more money in the end having me help you build it right than trying to build it over and over trying to get a system that can perform.

 

[aufalien]

I forgot to mention that either drive is not well suited as a SLOG. The Crucial looks to have 5ms latency which is abysmal. Couldn't get the Intel latency spec but they traditionally have low latency. And when it comes to SLOG, its all about latency. I found Intel live chat very helpful so you could always get a chat session going to verify latency specs.

For sh%$ts and giggles, you can try;

Set sync to disabled

Or config a RAM disk and make it a SLOG

Then run your tests. Mind you, this is ONLY for testing to see if numbers change. I would normally advice getting the newer eMLC class SSDs like the Intel DC3700 or OWC Enterprise Pro for a ZIL. But before spending yet more money, see if your numbers change and go from there.

 

[cyberjock]

RAMdisk from an SLOG? yeah.. don't even do that.. there's no point in doing that, even for testing.

 

[xhoy]

From top to bottom:
@eraser, at first the disk ran MU3, we updated to MU5 performance increase by almost 50%.

@cyberjock, We know that benchmarks and the real world differ. But this is the only way we are getting *some* insight's. Until know our benchmark seem to match our own experiences. We mounted an NFS export on a ESXi machine, a windows 7 x64 install incl some updates took 6 hours vs 2 (with the slog).

We ran several existences sequential write tests that match this experience.

@eraser we are talking SLOG here, not ARC.

@cyberjocky we ran every test atleast 4 times, where the speed of all 4 times is roughly the same evertime. Without slog, with 330 and M500. I don't think we are testing cache her. We only run write test on a SYNC enabled mounted NFS share since that is the place were SLOG comes in handy. as far as I/we read all documentation the concept of SYNC is that that ever bit has to be committed to stable storage. So it can't be in ZFS cache. Right?
We looked at the forum and read almost every thread that has to do with building a stable FreeNAS setup. Most of the post where your's btw! (thanks for that) :)

@Aufalien, since we like to get it right

@Cyberjocky, We have good/acceptable performance with the Intel 330 SSD, we where just wonder why the M500 do sucks :P

@aufalien, we get that the drive is not suited and it didn't seem to be that unsuited.... We were thing about the DC3700 but these were a lot cheaper :)

And a ramdisk is out of the question, since a power failure would.... :)

So it seems that we need a Seagate Enterprise Pro or an Intel DC S3700 to get it working with acceptable performance.

 

[cyberjock]

Well, a ramdisk is pointless because the ZIL/SLOG is already in RAM. The whole point of an external device is to have a non-volatile storage for whatever is in RAM in the event power is lost between the point at which your server claims the data is in the pool and when it *actually* is written. If you don't care about that few seconds, then sync=disabled is basically like having the SLOG in RAM and no non-volatile storage. It's good for testing, but doing it for real world is pretty dangerous stuff. It's like gasoline and a spark.. just waiting for you to be dumb enough to mix the two together.

As for the M5, I have no clue. I don't recommend anything except Intels because of their performance and reliability.

 

[xhoy]

Well, a ramdisk is pointless because the ZIL/SLOG is already in RAM. The whole point of an external device is to have a non-volatile storage for whatever is in RAM in the event power is lost between the point at which your server claims the data is in the pool and when it *actually* is written. If you don't care about that few seconds, then sync=disabled is basically like having the SLOG in RAM and no non-volatile storage. It's good for testing, but doing it for real world is pretty dangerous stuff. It's like gasoline and a spark.. just waiting for you to be dumb enough to mix the two together.

Yes we know that :)
That is way we are looking at the SSD SLOG i am thinking about replicating the lkcl test because i cannot find any documentation on the seagate 600 SSD (or any other enterprise ssd other then the S3500.

It seems that we accidentally picked a (relativly) very fast write speed SSD. We had this one laying around for a year or so. pure luck!
I found that there is an issue with the M500 and freebsd but i dont know if this had preformance implications... (the samsung (pro) 840 has the same problem)
http://lists.freebsd.org/pipermail/freebsd-stable/2012-November/070812.html But that should be fixed by now?

 

[cyberjock]

No clue if that fix is in FreeNAS or not. The 840 Pro is one of the worst choices. Samsung's 840 Pro uses TLC memory which has a relatively short livespan compared to even "cheap" MLC.

 

[xhoy]

yeah i know it is a bad choise but the bug is the same, but i don't see how disabling a feature could be a performance enhancement/drop

 

[cyberjock]

Disabling what feature? I don't see anything about disabling a feature...

 

[ser_rhaegar]

No clue if that fix is in FreeNAS or not. The 840 Pro is one of the worst choices. Samsung's 840 Pro uses TLC memory which has a relatively short livespan compared to even "cheap" MLC.

Actually the Pro uses MLC while the EVO uses TLC. Not that I'd recommend it still. I use the EVO for my ESXi datastores, not great, not terrible. I tried a pair in my FN box as a mirror but it wasn't very fast.

 

[cyberjock]

Actually the Pro uses MLC while the EVO uses TLC. Not that I'd recommend it still. I use the EVO for my ESXi datastores, not great, not terrible. I tried a pair in my FN box as a mirror but it wasn't very fast.

Oh, you're right! Thanks for the correction. ;) Pro and EVO have 3 letters so it gets confusing in my head. Haha. Samsung's drives never caught my attention until the EVO. And those are TLC and I keep thinking the Pro is from the same technology.

 

[xhoy]

Just al little update.

After we tried the Crucial M500 We also tried a Seagate 600 PRO ssd (100GB version). The results where some what better then the M500 (5%) but still really really slow.

We then switched to a Intel DC S3700 100GB and performance are great! We now fully utilize the 1GB Ethernet port with write/read simultaneous. Write speeds of 80mb/s are nothing special!

SO After a real world testing: Just buy a S3700 as an SLOG!

 

[joz]

Just wanted to add I had 3 m500 on MU03 firmware and they crashed the server eventually. There was a bunch of low level IO related errors I didn't notice until I checked the console. I believe it was from the bugs with SMART reporting on that firmware. But it caused the entire server to crash blocking io to other storage pools.

My initial indicator that something was wrong was actually errors in another storage pool with good drives. I assumed the WD disks were going bad but then they all started to go bad.
I logged into the console and say the errors on the ssd devices and realized that io to those devices was totally blocked. I removed the drives, and re-silvered the other devices in the other pool and everything was fine.

I have since upgraded to MU05 firmware. The issue hasn't resurfaced after a week. But I haven't put a ton of load on it yet. I wouldn't put those drives in a fileserver until the MU05 firmware is flashed.