NVMe drive for slog

[Rand]

So while I am playing around with my new build I wondered whether a NVMe drive would be a good slog device.
I have read @jgreco's recommendation for P3700 (or Intel 750) alternatively but haven't seen anyone actually testing them.

So I dont have a 750 but I have an Samsung 950 pro/512MB drive. I know its not a recommended disk due to missing superconductor. But after all its playing around only so what the heck.

My primary use case will be vmware/nfs so I thought I'd run a pretty basic test (not scientific at all), but I am quite suprised at the result - the 950 pro is way slower then expected.


I have run a simple copy from one esx datastore (locally, Samsung Evo 850) to one of three nfs attached
datastores

Datastore 3_SASSSD points to a mirror of HGST SAS 400GB SSDs
Datastore 3_nvme points to a single Samsung 950 pro as datastore
Datastore 3_tank points to a 2x2 mirror of NL-Seagate 7200 2GB SAS HDDs, with another SAS 400GB SSD as SLOG

[root@solesx2:/vmfs/volumes] ls -ltr EVO_850_512/W2k12/W2k12-flat.vmdk
-rw------- 1 root root 42949672960 Nov 7 2016 EVO_850_512/W2k12/W2k12-flat.vmdk

[root@solesx2:/vmfs/volumes] time cp -R EVO_850_512/W2k12/W2k12-flat.vmdk 3_SASS
real 7m 3.09s
user 1m 2.84s
sys 0m 0.00s
[root@solesx2:/vmfs/volumes] time cp -R EVO_850_512/W2k12/W2k12-flat.vmdk 3_nvme/
real 22m 14.83s
user 1m 7.47s
sys 0m 0.00s
[root@solesx2:/vmfs/volumes] time cp -R EVO_850_512/W2k12/W2k12-flat.vmdk 3_tank/
real 7m 19.29s
user 1m 25.72s
sys 0m 0.00s

So despite beating all other disks in speed by at least double (reached >1GB/s with the nvme drive) it does not seem to be a good SLOG device at all (and yes i know high transfer speed does not make a good slog device;))

I now wonder whether this is only the cheapo Samsung and if somebody has tested the Intel 750 with more sucess?

Edit:
Just for completeness sake here is the samsung as slog on the tank pool
[root@solesx2:/vmfs/volumes] time cp -R EVO_850_512/W2k12/W2k12-flat.vmdk 3_tank
/
real 21m 28.50s
user 1m 27.42s
sys 0m 0.00s

 

[jgreco]

So while I am playing around with my new build I wondered whether a NVMe drive would be a good slog device.
I have read @jgreco's recommendation for P3700 (or Intel 750) alternatively but haven't seen anyone actually testing them.

Really? 'Cuz I've been running one here since shortly after they became available.

 

[Rand]

:p
Ok, so I have not found the thread were testresults have been discussed;)

 

[jgreco]

Yeah, finding stuff is always difficult with this forumware unless it happens to have been stickied.

 

[Rand]

So I found this post
https://forums.freenas.org/index.ph...test-box-32cores-and-10gbe.39229/#post-242503

Code:

# zfs set sync=standard storage3; dd if=/dev/zero of=file bs=1048576 count=1048576; rm file; zfs set sync=always storage3; dd if=/dev/zero of=file bs=1048576 count=1048576; rm file; zfs set sync=standard storage3
1048576+0 records in
1048576+0 records out
1099511627776 bytes transferred in 3189.541550 secs (344724033 bytes/sec)
1048576+0 records in
1048576+0 records out
1099511627776 bytes transferred in 13362.986037 secs (82280384 bytes/sec)

That's an Intel 750...

So the 750 (78MB/s) is the way faster then the 950 pro (~30MB/s) unless I miscalculated
82280384 / 1024 (->K) /1024 (->M) => 78MB
42949672960 /1024/1024 = 40960 / 1376s=> 29,7

I am still quite suprised that even the 750 is slower than the "regular" (SAS) SSDs

Edit: New calculation , had the SAS SSD before

 

[jgreco]

I don't know that I'd take a random SAS SSD performing better as meaning anything. All of this is meaningless unless you have a device that is actually providing the log functionality required to support POSIX sync writes. If your SAS SSD, for example, is caching writes in RAM without a mechanism to guarantee them, of course it'll be a lot faster. The tricky problem with all of this has always been to find a low latency device that actually delivers the required characteristics, otherwise it's just easier to turn off sync writes and HOLD ON FOR YOUR LIFE IT JUST GOT 10X FASTER...

 

[Rand]

I agree;
but I'd hope that the HGST SSD400M (HUSML4040ASS600) would be a suitable device

In the end its o/c a question of what do I really need, what can I afford and what has been tested.

From a business point of view reliability and known good experience are key -
for home/enthusiast use it is important but not at extreme cost.

This is just discussing the results that suprised me and hopefully answering the question whether *any* nvme disk will do (it will not) and that cheap nvme drives are not the holy grail (for slog).
Certainly a P3700 for accredited 250 MB/s is a good choice but pricey.

 

[kspare]

i've been running an intel 750 for slog and one for l2arc for awhile. they work great!

 

[kspare]

I currently have my san offline to update to 9.10. I can run some tests on it if you have anything you want to know?

 

[Rand]

Well If you can confirm jgreco's numbers for a sync data write that would be great.

Then I'd just need someone with a 950 to double check my results and we'd have a good baseline.

 

[jgreco]

What's the point of checking with the 950? It's a pointless exercise to check something that isn't useful.

 

[kspare]

950 what?

 

[jgreco]

Samsung 950 Pro. I don't see the point since it has no power loss protection, so it's unsuitable as a SLOG.

 

[Rand]

:) O/c it is totally useless for your use case :)

As I said I (and probably many others not using this professionally) would at least consider it.

So identifying its uselesness performance wise next to it not being recommendable due to missing superconductor seems not totally useless to me.

And additionally I wonder whether this kind of performance is typical for nvme (likely not) so I appreciate any input on other nvme drives.

 

[jgreco]

It doesn't matter if it's "professionally" or "not". Once you decide to go with a device that isn't a suitable SLOG, you are basically just MUCH better off turning off sync writes and dropping the issue entirely. It's one of those things you can't really do halfass. Either your data is protected correctly under POSIX semantics or it isn't. The "isn't" really isn't a big deal in many cases.

 

[kspare]

He's right you know...

 

[jgreco]

He's right you know...

Who, me?

 

[kspare]

Yes.

 

[jgreco]

Yes.

Well of course I'm right. I'm always right. It sucks to be always right.

 

[Rand]

He's right you know...

I know;)
Never was in disagreement about this (or the topic). Was just trying to gather info for ppl who dont believe him :p