Upgrading to 10 GbE, use NFS, will SLOG now make a difference?

[scwst]

So I finally decided to create a 10 GbE connection between my main Ubuntu machine and my FreeNAS box (spex in sig) -- the (used) parts are in the mail. Up till now, I hadn't given a SLOG much thought, because as I understand it, the current 1 GbE connection of somewhere about 100 MB/s is the bottleneck for the spinning metal disks I have in the pool. If all goes well, however, I should be looking at roughly ten times that network speed by the end of the week. I admit to feeling somewhat faint.

Since I use NFS for everything, and (as I understand it) NFS uses sync, re-reading https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/ and https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/ leads me to believe that even a SSD SLOG via SATA III for about 500 MB/s would make things noticeably faster. It just so happens that I have a SATA Intel 540 SSD 120 GB sitting around here I could use to gain some first experience. I would just like to make sure I understand the basic mechanisms involved before I start with any cable stuff.

(I should mention that this would be a temporary setup -- I'm aware that the Intel 540 SSD has no battery backup and something like an Intel 750 PCIe would be what I should really be looking at. Truth be told, I'm rapidly outgrowing the C2550D mobo, so I'm going to have to sit down and figure out a whole new hardware setup anyway in the coming months. I am also aware that the first step should always be max out RAM, and I do have two sockets free. But again, with a new machine coming up and RAM prices what they are, I'm probably going to leave memory as it is and invest the money in the new system.)

 

[MatthewSteinhoff]

1. Benchmark.
2. Add SLOG.
3. Benchmark.
4. Review results.

There is little danger to adding an SSD SLOG, testing, then removing it if you don't see improvement. Or, if you do see improvement, you can always swap the test SSD for a power-protected, high-endurance, low-latency SSD.

In our configuration, we have a very wide stripe of mirrors. We added an SSD SLOG and found no performance improvement. In fact, because the stripe was so wide, the SLOG might have slowed down writes. It was hard to tell. So, we removed the SLOG to simplify the configuration.

Cheers,
Matt

 

[HoneyBadger]

While ESXi does issue sync writes to NFS datastores, I don't know if Ubuntu uses sync on its default mounts.

SSH into your FreeNAS machine, run zilstat, and then write some data. If you see non-zero values, then an SLOG may help you for small bursts of writes.

But if you're getting around 100MB/s on your current config, I'll go ahead and say "no, you aren't using sync writes" and in this case, an SLOG device will just sit idle doing nothing but burning wattage and money.

As far as the Intel 540, it's likely a poor SLOG device as it not only lacks PLP (power loss protection) but also uses TLC NAND with no posted TBW (terabytes written) on the ARK site, so I have to imagine you'll be blowing through the program/erase cycles in a hurry.

 

[scwst]

Thanks for the advice and especially the pointer to zilstat, I wasn't aware of that program. Testing it with the current setup (1 GbE, no SLOG) and a 1.2 GB file, I get (lots of zeros before and after deleted):

Code:

N-Bytes  N-Bytes/s N-Max-Rate	B-Bytes  B-Bytes/s B-Max-Rate	ops  <=4kB 4-32kB >=32kB
  3616	   3616	   3616	 131072	 131072	 131072	  1	  0	  0	  1
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
   496		496		496	   4096	   4096	   4096	  1	  1	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
   688		688		688	   4096	   4096	   4096	  1	  1	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
111160	 111160	 111160	 131072	 131072	 131072	  1	  0	  0	  1
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
   456		456		456	   4096	   4096	   4096	  1	  1	  0	  0
	 0		  0		  0		  0		  0		  0	  0	  0	  0	  0
  1904	   1904	   1904	  16384	  16384	  16384	  4	  4	  0	  0

I take this to mean that, yes, it's using sync with NFS on Ubuntu, and yes, a SLOG would help for these transfers. I'll wait till the 10 GbE is in place and then test a bit more (the Intel SSD has seen some use anyway, so I'm not too worried about it wearing down while I figure out what to do) and then try to figure out if this makes sense. Thanks everybody!

 

[scwst]

Shouldn't we be able to estimate just how long the Intel 540S TLC SSD drive would last as a SLOG?

I checked the Smart parameters Media Wear Out Indicator which is 99, and NAND Written (1 GiB), which was 578 (Intel provides this statistic, Samsung doesn't. Boo!). Roughly, 578 GiB written divided by the capacity of 120 GiB gives us about 5, so statistically speaking I should have five complete writes finished. Since this is a TLC drive, we should have about 1000 P/E cycles originally (based on https://www.anandtech.com/show/6459/samsung-ssd-840-testing-the-endurance-of-tlc-nand), so let's say I have 995 P/E cycles left.

Because of my mildly obsessive bookkeeping, which I can totally quit anytime and does not impact my quality of life at all, I know that I use up slightly less than 1.5 TiB of storage on my FreeNAS per year. The system is used mostly for archiving, but there is some backup from various computers, though usually texts and other small stuff. Obviously there are automatic snapshots being written and erased, but also nothing big (I don't backup caches for the browser, for instance). But let's say 2.0 TiB of writes per year. That would be 2.0 TiB/120 GiB which is 16.6 - so I'd go through roughly 17 P/E cycles per year. With 995 cycles left, that means the drive should be toast in 58 years and a bit. If I (or more likely, my heirs) am still using this drive in the year 2076, we're in some post-apocalyptic scenario. Zombies, I assume.

If these calculations are correct? Never tried this before.

What I probably should do once the 10 GbE is installed is write down the Smart parameters of the drive, then install the SSD as a SLOG, let it all run for a few weeks, and then check Smart statistics again. That would give me real data, which is much more fun.

 

[MatthewSteinhoff]

I checked the Smart parameters

Ugh... I love the idea of SMART but hate its execution.

We put a four-drive strip of SSD mirrors (960GB) in production for two and a half years ago for our XenServer VMs using the least expensive drives we could find at the time. Here are the two SMART codes we track...

* 169, Remaining Lifetime Percentage, is at 99
* 177, Wear Leveling Count, is at 49

Depending on which web forum I believe, we either have 198 years left or just under two years. The SSDs are rated at 360TB TBW and have a three-year warranty so I'm guessing they will die in three years and a day all else being equal.

We paid $210 each in April 2016. When they die, we'll just replace them with the current price leader. Today, that looks to be Samsung at $178. At this price point - and probably less - we're happy to treat drives as disposable items that are regularly replaced. Assuming a three-year lifespan and an $800 spend, we're paying less than 75 cents a day for mind-blowingly fast VMs.

Cheers,
Matt

 

[scwst]

So I have the 10 GbE connection set up and the Intel SSD installed as a SLOG for testing, and after some fooling around with the right SATA ports and whatnot, it works. iperf3 gives me about 1110 MB/sec for straight "memory to memory" testing, which is what you would expect. However, "memory to disk" ( iperf3 -s -F test -f M on the FreeNAS, iperf3 -f M -c <FREENAS> on the Ubuntu client computer) gives me 70 MB/sec, which is about half of what I had expected -- the limiting factor should be the hard drive's speed about somewhere around 120 MB/sec or such, if I understand stuff correctly. Note these are three mirrored VDEVs of two drives each combined to form the pool.

Now, I'm assuming that iperf3 doesn't do sync writes, so the SLOG has no part in this. There seems to be some bug with Ubuntu 18.04 LTS not correctly setting jumbo frames from the desktop interface (stuck with MTX 1500 for the moment), and they've completely changed the low-level configuration, so I'm not going there now. But that should affect memory to memory as well.

In practice, I don't really care, because async writes of this kind will be buffered in memory anyway, and sync writes will go to the SLOG (need to test that for speed yet). Just curious what's going on here.