BUILD replacing pool of WD Red with Samsung 860 Pro

[Hobbel]

Hey out there,

I would like to replace my 6x 1TB WD Red pool (3x mirror, INTEL DC P3700 PCIe as ZIL/SLOG) with 6x Samsung 860 Pro. I searched the forum and found some information, but not definitive answers. It will be used as ESXi datastore via iSCSI (5x 1GBit). 3 disks are connected to the onboard SATA ports and 3 are connected to the HBA. I try to figure out:

• would you recommend the 860 Pro?
• should I use the ZIL/SLOG for the SSD pool? (my search results say -> yes)
• should I use the entire disk space or under-/overprision each one to e.g. 750GB?

You'll find the full specs of the system in my signature + INTEL DC P3700.

Thanks in advance :)

 

[MatthewSteinhoff]

would you recommend the 860 Pro?

I recommend the least expensive SSDs that meet your durability and performance needs. For us, that was four inexpensive (at the time compared to everything else) ADATA Premier SP550 960GB drives. Coming from conventional drives, these are five times faster, give or take. By the time they wear out, we'll be able to get double the capacity and double the performance at the same price point, give or take. We have 19 XenServer VMs hosted from this pool.

SSDs have massively more IOPS and throughput than conventional drives. If you have a 6x1 conventional now and are mostly happy, you may not need 6x1 SSDs. Depending on your storage needs, you might be able to get away with a pair mirrored. Going 4x1 SSD would give you ten times the performance, give or take, what you have now. Look at the specs on the drives you have now and the drives you are buying, be conservative and run the numbers.

should I use the ZIL/SLOG for the SSD pool? (my search results say -> yes)

We tried SLOG with SSD and didn't find any performance improvement so we got rid of the SLOG to simplify the configuration. Your mileage may vary.

should I use the entire disk space or under-/overprision each one to e.g. 750GB?

We're running ours at under 35% capacity mostly because we over purchased on both capacity and performance because we were SUPER conservative. Probably could have gotten by with 960G mirrored or a striped mirror of 500G SSDs.

SSDs typically recommend over provisioning of 15-20%? FreeNAS recommends a reserve of typically 15-20%? That can be the same 20%. On a 1,000G SSD, I'd feel comfortable presenting 900G to FreeNAS and then using up to 800G of that space. SSDs are still new enough in this use case that I haven't seen any hard recommendations I believe.

Cheers,
Matt

 

[Nick2253]

• would you recommend the 860 Pro?

• should I use the ZIL/SLOG for the SSD pool? (my search results say -> yes)

• should I use the entire disk space or under-/overprision each one to e.g. 750GB?

I will also recommend the cheapest SSD you can find. With the increased IOPS and bandwidth of an SSD, even a parity array will give you enough performance for your workflow, and you'll be able to take better advantage of the expense SSD space. Don't forget, a typical SSD has at least 100x IOPS of your WD Red, and at least 3x bandwidth.

The only reason you'd need a SLOG is if the latency of the SSD pool is still too high for your workload (the critical function of a SLOG is to decrease the latency from a write request to completion). You will probably not hurt your performance by keeping the SLOG, but it won't make nearly as much of a difference as with spinning rust.

I would not recommend under-provisioning by default. It's important for a SLOG, because every sync write goes through the SLOG. However, for the large array, the load is split out among all the different pool members. If you only do two drives in a mirror, it might be a good idea to slightly underprovision, but more than that seems unnecessary given the massively improved lifespan of SSDs today.

If you're set on 6 drives, I'd recommend doing a six drive RAIDZ2 with the drives to maximize your storage space.

 

[Hobbel]

@MatthewSteinhoff @Nick2253
thanks for your oppinions. I know that the IOPS are massively increased compared to the current disks. But I'm not sure if "any SSD" is the best choice...

my first idea was to use Samsung 860 Evo 1TB (€300). But after some reading I don't like the TLC for my datastore. Call me pessimistic, but TLC is not known for the best choice regarding the long term guarantee of the specs.
The 860 Pro 1TB (€450) is MLC and has an endurance of 1.2PB. This seems to be a good starting point for a long term use. Of course, these numbers are just written down on paper and in reality the disks can fail before/after all those endurance specs.

This build is not primarily driven by the costs. Of course, saving money normally is a good thing... This replacement will also become my reference system for upcoming builds for my customers.

I will have a look at other drives with MLC (I will have a look at TLC too, but I don't like it). But what I've seen so far is that there are not so many out there, which are cheaper.

for now I have

• no underprovisioning
• using SLOG, because it's builtin

 

[Hobbel]

found some cheaper SSD's around 900-1.000 GB. It's about saving €100-150 per disk, if I would use TLC.

This means 2x 860 Pro vs 3x another SSD. I tend to start with 4x 860 Pro and bet on the longer endurance of the disks. So the replacement/maintenance costs should be lower which over all will result in the same costs.

 

[Nick2253]

but TLC is not known for the best choice regarding the long term guarantee of the specs.

Do you have a source for this? Real-world, long-term stress tests that I have seen show that TLC is reliable: https://us.hardware.info/reviews/41...-with-final-conclusion-final-update-20-6-2013

 

[wblock]

TLC is reliable, but probably will have a shorter life than the others. A third less, since it fits three values into a cell instead of the two of MLC. Given all else being equal, which it probably is not. Actual tests of TLC have shown that it can last a very long time, probably long past the time of obsolescence. I would expect it to last longer than a typical hard drive, particularly now that Seagate has swallowed all the other hard drive companies.

 

[Hobbel]

Do you have a source for this?

This is my result from reading many different sources. of course, it's my interpretation of information - so for my build.
Like wblock said, it has one more state. in my own words, without being an expert in building SSDs: the older the SSD becomes the more likely it is, that the states can't be identified clearly and the cell is markes as "damaged".

yes, most of us know that there is a chance that the lifetime of todays SSDs can exceed the lifetime of the current hardware configuration. But I have doubts about TLC.

 

[Nick2253]

This is my result from reading many different sources. of course, it's my interpretation of information - so for my build.

I was really hoping you had a source for this, because pretty much everything I've been able to find on the internet is blog speculation based on specs. If there is real evidence that TLC is less reliable, I'm very interested in learning about it. What I've been able to find about actual tests is that this seems to not be the case; granted, these are usually not rigorous, or with a large sample size.

the older the SSD becomes the more likely it is, that the states can't be identified clearly and the cell is markes as "damaged".

Age alone has very little impact on silicon. It's the wear from writing that is the real problem. This is why technologies like wear leveling were developed. However, even if a cell is damaged or defective, the controller has means of marking it as such (similar to how HDDs work). The big concern is obviously that many cells will go bad, and overwhelm the capacity of the SSD to provide replacements. The advantage with ZFS is that, in the event that data is actually lost, ZFS's scrubbing/checksumming will detect it (and correct it). All the more reason to stay up on your SMART data.

Obviously, if money is not a concern, I would also recommend MLC. For that matter, you might as well go all the way to SLC. However, my overall argument is that the benefit of MLC, especially with ZFS, is not worth the premium over TLC. This is primarily because the reliability of a TLC drive over the course of a reasonable use life is no worse than an MLC drive.

 

[Hobbel]

Age alone has very little impact on silicon

Normally, the older the drive the more is written on it ;)

@Nick2253
I understand what you are thinking about MLC and TLC regarding FreeNAS - again, thanks for your opinion. My concerns are overall performance over a long time of use. What I've read so far is, that the more space is used, the slower the TLC-SSD gets. Don't know if it's the decreasing amount of free space, controller thing, cache or something else...

With all your input it's 50:50 on the Samsung 860 Pro or Evo. There are some other SSD on the market, on which I could save another €30... but I have good experiences with Samsung.

I'm super pessimistic when it comes to MLC and TLC in this build. It will become a reference system which also can be suitable for my customers. Therefore, saving money is the 2nd argument.

 

[MatthewSteinhoff]

If money is no object, feel free to buy yourself peace of mind. The Samsung 960 EVO and PRO drives are very nice.

For me, I still buy low-end SSDs. Our ADATA 960G drives went into production first quarter 2016. SMART says they have 96% of their lifespan left. Even if wear accelerates with age to 10% a year, we've got more than six years left on the drives. I can't predict the future, but I'd bet this week's salary that we will replace those drives as well as the server before the SSDs wear out.

Many SSD truths seem only apply to first generation drives with early firmware. It's like everyone heavily investigated them, chiseled facts in stone and not much has been done since. SSD endurance warnings were first aired before wear leveling, before trim and before native (hidden) over provisioning.

Cheers,
Matt

 

[Hobbel]

For me, I still buy low-end SSDs

the ADATA costs the same as the 860 EVO. so both are "low-end" ;)

the order will be placed next week... Will report back, which SSDs will replace the WD Reds and how they will behave

 

[Nick2253]

Normally, the older the drive the more is written on it ;)

As an extremely general rule, yes :D. However, workload variation is so high as to make these kind of generalities useless for planning a system. If your workload demands lots and lots of writes, then you might need even SLC to meet your longevity requirements.

My concerns are overall performance over a long time of use. What I've read so far is, that the more space is used, the slower the TLC-SSD gets. Don't know if it's the decreasing amount of free space, controller thing, cache or something else...

The speed issue is a reasonable concern for TLC for two reasons:

First case is that writing a TLC cell is relatively costly, time-wise, so SSDs typically use a cache (or use TLC cells as SLC cells temporarily) during write operations. When this cache is exhausted, writes slow down. This is usually very pronounced on smaller drives around 256GB or less (factors of 3 or so for ~100GB drives), but not so much on large drives around 512GB or more (usually only 10-20%).

Second case is that TLC cells are much more susceptible to voltage drift. This means that reading data that was written a while ago can be more time consuming, since the controller has to do more complicated analysis to determine the cell's contents. The solution to this is to rewrite the data over time. As the drive gets full, more data needs to be rewritten, and less data can be rewritten at a time. What's nice is that, since ZFS is a copy on write filesystem, that feature actually works in tandem with the limitation of a TLC SSD. If you keep your TLC to reasonable fullness (for the same reasons you don't want to go over about 85% on a ZFS drive), you might notice a slight slowdown, but it shouldn't be significant.

 

[Hobbel]

When this cache is exhausted, writes slow down

exactly this is one of my concerns. I've no experience how much this "slow down" can be in about 1-2 years. This is why I'm looking at MLC.

If it was just to save money: Mirroring a 860 Evo with a 860 Pro (when the Evo will fail or extremely slow down, replace it with a Pro)?
(saving €150 on each mirror)

 

[Nick2253]

exactly this is one of my concerns. I've no experience how much this "slow down" can be in about 1-2 years.

The write cache is not something that permanently fills up. Just like the cache in a HDD or on your CPU, it is used transactionally. The only time you'd exhaust your write cache is if you tried to write more than that amount to the SSD in one transaction. The cache in most SSDs (even MLC) is some amount of SLC flash. MLC SSDs suffer this exact same problem, except that the difference between the SLC and MLC write speeds is much smaller, so is less noticeable. For that reason, MLC SSDs typically have a much smaller cache to help accelerate small file transfers (typical random I/O workload).

If it was just to save money: Mirroring a 860 Evo with a 860 Pro (when the Evo will fail or extremely slow down, replace it with a Pro)?

Based on actual, real world testing, your Evo and Pro probably have about the same lifespan. This stress test compared the 840 to the 840 Pro (TLC and MLC, respectively): https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/3

If spending the extra €150 to put a Pro in each mirror makes you feel better, then I would do it.

 

[toadman]

The right answer is characterize your workload and do the reliability calculations. Unless you have a *very* write heavy workload I think you'll find the numbers suggest TLC is fine. Most of the OEM storage vendors I know in the industry did their supplier deals based almost entirely on $/GB from flash suppliers for their servers/storage arrays vs. endurance specs or anything of that kind. It just doesn't matter for normal workloads as long as the devices themselves are large enough capacity. And you can always increase endurance (If that's your worry) for the same workload just by buying larger SSDs right?

 

[Chris Moore]

exactly this is one of my concerns. I've no experience how much this "slow down" can be in about 1-2 years. This is why I'm looking at MLC.

You have a couple of real experts trying to answer your question and you sound like you are ignoring their voice of experience.

Here is a YouTube video that explains it, perhaps easier to understand:
https://www.youtube.com/watch?v=-XZNr7mS0iw

You keep saying things like, "long term". How long are you thinking you will be able to use this configuration?

 

[Hobbel]

You have a couple of real experts trying to answer your question and you sound like you are ignoring their voice of experience.

that wasn't my intention. I'm very happy that there are the possibilities of discussing such topics with other people around the world. The last thing I do is ignoring the answers.

How long are you thinking you will be able to use this configuration?

Good question. This is my first RAID/ZFS build with SSDs. I just don't want to run in problem after some months. Simply said: I don't have any experience how this will behave. Specs on paper are nice, but not always guaranteed in reality. If I need to replce my HDD in my laptop, I would go with TLC without having bad feeling.
Let's say, "long term" are 2 years...?

 

[Chris Moore]

that wasn't my intention. I'm very happy that there are the possibilities of discussing such topics with other people around the world. The last thing I do is ignoring the answers.

Good question. This is my first RAID/ZFS build with SSDs. I just don't want to run in problem after some months. Simply said: I don't have any experience how this will behave. Specs on paper are nice, but not always guaranteed in reality. If I need to replce my HDD in my laptop, I would go with TLC without having bad feeling.
Let's say, "long term" are 2 years...?

If long term is only 2 years, I suggest that you look back at the post above:
https://forums.freenas.org/index.ph...d-red-with-samsung-860-pro.61352/#post-436583
Those drives have been in service just shy of two years and are still going strong. There is your answer, don't agonize over it. Keep an eye on them and if you see some early failure, as any drive can do, replace the drive with a spare and get the failed unit replaced under warranty. If I am not mistaken, all those Samsung drives carry at least a 3 year warranty. Done deal.

 

[nightshade00013]

I agree If you have two years of intended use a drive with a three year warranty should not have a problem.

It's a little older generation but some interesting information about some real life testing of some SSD's have been done. Take from it what you will but I think it does give some information about the drive speed fall off to expect as well as how long they can last. https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead