NFS, ESXi 6, ZIL, and SLOG

Status
Not open for further replies.

Jon K

Explorer
Joined
Jun 6, 2016
Messages
82
Hi all -

So, I have a RAIDZ pool setup in what I like to consider a good compromise configuration between IO, redundancy, and capacity (4 x 3 1TB disk RAIDZ). I've created a dataset and NFS share using NFS4 checkbox, mapped root, and mounted it to my host. I storage vMotion'd a VM to it and am doing:

/test$ sudo dd if=/dev/zero of=/test/tmp.dat bs=2048k count=4k
4096+0 records in
4096+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 73.105 s, 118 MB/s

The above is with sync=disabled on the dataset. Then enable sync and do:

/test$ sudo dd if=/dev/zero of=/test/tmp.dat bs=2048k count=4k
4096+0 records in
4096+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 240.491 s, 35.7 MB/s

Obviously 3x slower.

So, I do have a 250GB SSD for SLOG and 256GB for ZIL. I would think that I would get better performance from this configuration with the SLOG configured with sync writes on NFS. Though, that only applies if the writes are actually hitting the SLOG. Can anyone guide me to better throughput w/ sync enabled?

Thanks!
 

diehard

Contributor
Joined
Mar 21, 2013
Messages
162
The ZIL is highly dependent on an SLOG's latency and performance with a very short queue depth. Which SSD are you using?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
The only way to get faster sync writes are to get a super fast, low latency pool, or to get a proper SLOG. You can test the performance difference by adding the log device to your pool and running your sync test. As mentioned, get a proper SLOG device (low latency, capability to handle heavy writes, internal power backup protection (capacitors)).
 

Jon K

Explorer
Joined
Jun 6, 2016
Messages
82
All, I am using a Samsung 850 Evo (250GB) for SLOG. I know it doesn't have capacitors but my FreeNAS node is on a 3000va UPS by itself with redundant PSUs on the server and the UPS is on an automatic generator. The storage node is for lab only, so I am not terribly concerned. The 850 Evo is capable of 500+ MB/s and 40k IOPS at QD of 1, and 80k+ IOPs at QD32. Thoughts?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
All, I am using a Samsung 850 Evo (250GB) for SLOG. I know it doesn't have capacitors but my FreeNAS node is on a 3000va UPS by itself with redundant PSUs on the server and the UPS is on an automatic generator. The storage node is for lab only, so I am not terribly concerned. The 850 Evo is capable of 500+ MB/s and 40k IOPS at QD of 1, and 80k+ IOPs at QD32. Thoughts?
I tried a Samsung 850 EVO SLOG device when I first experimented w/ FreeNAS about a year ago. It's not really very well-suited to the task. I pulled it and ran with sync=off on my NFS dataset until I installed an Intel DC S3700 a couple of months ago. The S3700 is an entry-level SLOG. There are superior choices, all more expensive, but it has all of the requisite characteristics of low latency, fast writes, high durability, and power backup.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
All, I am using a Samsung 850 Evo (250GB) for SLOG. I know it doesn't have capacitors but my FreeNAS node is on a 3000va UPS by itself with redundant PSUs on the server and the UPS is on an automatic generator. The storage node is for lab only, so I am not terribly concerned. The 850 Evo is capable of 500+ MB/s and 40k IOPS at QD of 1, and 80k+ IOPs at QD32. Thoughts?

Totally unsuited to the task. You have two basic choices:

1) Just turn off sync writes and move on with your life, you will be much happier.

2) Select a proper device, perhaps an Intel 750 HHHL 400GB, or an Intel DC S3710, or some other appropriate SLOG device.

Selecting the wrong device for a SLOG is kinda like not buckling up your seat belt because you believe that since your car has air bags, you're already safe.
 

Jon K

Explorer
Joined
Jun 6, 2016
Messages
82
Totally unsuited to the task. You have two basic choices:

1) Just turn off sync writes and move on with your life, you will be much happier.

2) Select a proper device, perhaps an Intel 750 HHHL 400GB, or an Intel DC S3710, or some other appropriate SLOG device.

Selecting the wrong device for a SLOG is kinda like not buckling up your seat belt because you believe that since your car has air bags, you're already safe.

Interesting - so most documents/reviews indicate the 850 Evo 250GB has an average latency of 0.035 milliseconds or 35 microseconds on 4k writes. The S3710 has an average, according to the datasheet, of 55 microseconds. I know the S3710 has power loss protection. What makes the S3710 better than an 850 Evo for the intended use? Thanks again!
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
All, I am using a Samsung 850 Evo (250GB) for SLOG. I know it doesn't have capacitors but my FreeNAS node is on a 3000va UPS by itself with redundant PSUs on the server and the UPS is on an automatic generator. The storage node is for lab only, so I am not terribly concerned. The 850 Evo is capable of 500+ MB/s and 40k IOPS at QD of 1, and 80k+ IOPs at QD32. Thoughts?
UPS and Generator do nothing to protect your data in the case of a kernel panic, system crash or hard reset. And if you don't care about those possibilities in your environment, then save your cash, and use sync=disabled.
 

Jon K

Explorer
Joined
Jun 6, 2016
Messages
82
UPS and Generator do nothing to protect your data in the case of a kernel panic, system crash or hard reset. And if you don't care about those possibilities in your environment, then save your cash, and use sync=disabled.

Absolutely correct - but again, this is a lab. I have a somewhat nontraditional lack of concern for the data so much as implementing and designing the solution with known limitations.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Interesting - so most documents/reviews indicate the 850 Evo 250GB has an average latency of 0.035 milliseconds or 35 microseconds on 4k writes. The S3710 has an average, according to the datasheet, of 55 microseconds. I know the S3710 has power loss protection. What makes the S3710 better than an 850 Evo for the intended use? Thanks again!
Write endurance, for one thing. Power loss protection, for another (as you pointed out). And Intel seems to understate their SSD performance specs, as noted in this article.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Absolutely correct - but again, this is a lab. I have a somewhat nontraditional lack of concern for the data so much as implementing and designing the solution with known limitations.
Ah, well! If you're running a lab and "don't care", just run with sync disabled like @depasseg suggests. That will give you the absolute best performance! Just be sure not to kick the power cord out of your server. :smile:
 

Jon K

Explorer
Joined
Jun 6, 2016
Messages
82
Ah, well! If you're running a lab and "don't care", just run with sync disabled like @depasseg suggests. That will give you the absolute best performance! Just be sure not to kick the power cord out of your server. :)

I will lean on this eventually, but it'll always be in a lab. Though, my "lab" conveys scraps and pieces, but it's actually a pretty robust setup:

IMG_0364-1024x683.jpg


But yes, I'll try to keep the PSU cords in :) I may pick up the SSD you linked as it does look like the ideal solution. For now, I may run disabled on this dataset. Thanks for all the input!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Interesting - so most documents/reviews indicate the 850 Evo 250GB has an average latency of 0.035 milliseconds or 35 microseconds on 4k writes. The S3710 has an average, according to the datasheet, of 55 microseconds. I know the S3710 has power loss protection. What makes the S3710 better than an 850 Evo for the intended use? Thanks again!

The 850 Evo is no competition to the S3710. I mean, I like the Evo and we have a bunch of them, but if I was going to pummel an SSD with every write made to an array, the Evo's write endurance isn't suited.

But the big point here is the design of the SSD. We know that if there's a power loss, host reset event, etc., the S3710 will do the right thing. The purpose of a SLOG is to guarantee that ZFS can recover every uncommitted sync write made to a pool.

Either a device is actually capable of delivering on this promise or it isn't. Either this property matters to you or it doesn't.

If you do not require sync writes to be recovered, ditch the SLOG, go to sync=disabled, and your speeds will double, maybe triple.

If you do require sync writes and you don't have a device that actually delivers, I don't understand the reasoning.

If you do require sync writes and your SLOG has the correct characteristics, then obviously, great.

I don't see a path forward in there for an improper SLOG device.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yes, it doesn't have the endurance of the P3700 or S3710, but it is inexpensive and for moderate write load environments is probably the best low latency low cost NVMe device out there.
 
Status
Not open for further replies.
Top