Broke yet another SSD - 5 and counting.

[Crispin]

Hi folks,

I'm curious as to how people have faired long term with COTS SSDs as cache drives in a pool.

I have just bust my 5th (might be 6th) SSD in 4-5 years with ZFS.
I always have them as cache devices in a RAIDZ1 or RAIDZ2 pool. Mostly single device but have run two in a pool.
They normally last about a year before checksums start creeping up and finally they let go. I've only lost a pool once when I had one as a ZIL which was previously a cache drive.

I mostly get them RMAd so it's more hassle than cost. The disks range from OCZ, Crucial, SANDforce controllers etc. It's across the board.

See this thread for one I've just lost: https://forums.freenas.org/index.ph...t-showing-odd-numbers-for-cache-device.37102/

It's never a wear issue either: 231 SSD_Life_Left 0x0013 091 091 010 Pre-fail Always - 0

Anyone else see this type of thing?
NAS usage is a home server, downloads about 20GB backups a night and compresses it to tar but other than that, quite a simple life.

Cheers,
Crispin

 

[boris_lee]

When talking about SSD reliability,I usually trust Intel...

 

[Crispin]

I've not tried them. Usually Samsung, Crucial, Corsair and Sandisk are the ones I've had.
I'll be replacing this one with Intel and see how that goes.

I find it odd though that I break them in ZFS. I have them in my laptop and desktop and run pretty intensive DB stuff on them and they just keep churning.

 

[jgreco]

Even if you're running "pretty intensive DB stuff" on them, a typical SSD for laptop or workstation with an OS installed probably has a fair number of mostly-static pages, holding the OS, unused disk space (free SSD flash pages), etc. You probably aren't churning more than 50% of the pages.

With ZFS, it is constantly shoveling out data to the L2ARC as ARC contents are evicted from the host memory. Depending on the traffic into the server, it may be nearly constant, and it is going to tend to be writing to the whole SSD. It's virtually certain that all areas of the SSD will be written to every few {days, hours, etc}. This is kinda stressy for a SSD.

 

[Crispin]

If what you saying in the second paragraph is true then why would SMART say "SSD_Life_Left 0x0013 091 091 010 Pre-fail Always - 0" which I see as 91% free. This has been the case with previous SSDs too (not the exact same number but also pretty high)

As for the DB stuff - I'm generating roughly 200GB of data before truncating it and starting again. That's writing most of the drive each time (256Gb disk) In fairness, this has only happened about 20 times so it's still early days.

The only real difference is that the cache SSDs are 64GB while the desktop has 256GB disks in them.

Still confused by the SMART report...

 

[jgreco]

Writing most of a drive is different than writing all of a drive. We're having a lively discussion of underprovisioning of devices in several other threads right now.

91 likely indicates that a fair number of blocks have already failed. Might be interested to know the value of Retired_Block_Count (ideally the RAW value). Guessing it's gotta be in the hundreds, likely thousands.

The only real difference is that the cache SSDs are 64GB while the desktop has 256GB disks in them.

"The only real difference"...? No.

Have you actually written to each and every sector of that 256GB SSD, forcing the controller to allocate and map a page for each sector? Probably not. You have, however, for the 64GB SSD's. This presents different operational challenges to the SSD controller. The SandForce based consumer SSD's are really not optimized towards that kind of use, as they're using a variety of tricks to reduce write amplification. A lot of that technology is optimized towards typical consumer-PC style use. So your 64GB SSD's are being stressed a lot more (and a lot more often) than your 256's.

 

[jgreco]

Just spent a little time looking around the OCZ website.

http://ocz.com/consumer/ssd

I notice that they've rated a lot of their SSD's as endurance of 20-50GB/day. An L2ARC can easily see ten times that with the default FreeNAS configuration. 8MB/sec -> 700GB/day. Not likely to be maxxing that out 24/7, but I could easily see a moderately busy NAS pushing 300GB+/day out to L2ARC. My gut feeling is that you're stressing things in the SSD design that they aren't expecting to be stressed.

 

[Crispin]

"The only real difference"...? No.

91 likely indicates that a fair number of blocks have already failed. Might be interested to know the value of Retired_Block_Count (ideally the RAW value). Guessing it's gotta be in the hundreds, likely thousands.

Here's the whole report.

Code:

########## SMART status report for ada1 drive (SandForce Driven SSDs: OCZ-B592KW38CR0R1K0D) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 085 085 050 Pre-fail Always - 85/214020693
5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
9 Power_On_Hours_and_Msec 0x0032 099 099 000 Old_age Always - 1217h+04m+11.329s
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 360
171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 215
177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 1
181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 0
182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 128 129 000 Old_age Always - 128 (0 127 0 129 0)
195 ECC_Uncorr_Error_Count 0x001c 120 120 000 Old_age Offline - 85/214020693
196 Reallocated_Event_Count 0x0033 100 100 003 Pre-fail Always - 0
201 Unc_Soft_Read_Err_Rate 0x001c 120 120 000 Old_age Offline - 85/214020693
204 Soft_ECC_Correct_Rate 0x001c 120 120 000 Old_age Offline - 85/214020693
230 Life_Curve_Status 0x0013 100 100 000 Pre-fail Always - 77395310674020
231 SSD_Life_Left 0x0013 091 091 010 Pre-fail Always - 0
233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 23796
234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 22502
241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 22502
242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 9607

Interesting - The retired block count is zero, SSD_Life_Left - raw zero is apparently normal and meaningless, but then looking at the Lifetime_Writes_GiB, if this is a 64Gb disk and it's written 22.5TB to it - that's 351 "fulls". Is that that considered a lot?

Your second post -
If this is the case then
a) does your theory conflict with what SMART is showing? Can you tell from the above results that the disk is having a hard life? To me it looks ok (but I'm a noob)

b) would a bigger, say 265 or 512GB disk last longer or should I just bite the bullet and get an enterprise disk? 200GB Intel disk is £300 while a 240GB Crucial is £70. At a 1:4 ratio I don't mind if they last 2 years and throw them away...

 

[jgreco]

Interesting - The retired block count is zero, SSD_Life_Left - raw zero is apparently normal and meaningless,

I'd guess it's not counting them, so, yes, meaningless. Bah.

but then looking at the Lifetime_Writes_GiB, if this is a 64Gb disk and it's written 22.5TB to it - that's 351 "fulls". Is that that considered a lot?

Depends on your point of view.

I would expect contemporary MLC to be able to sustain an order of magnitude more writes than that (3000? 5000? Depends on whose MLC, etc).

But I notice that if we look at your power-on hours, divide by 24, that gives us appx. 50 days of operational time. Then if we look at lifetime writes and divide that by days, to get GB/day, we end up with 450GB/day written.

I have no idea what specific model of SSD that is. Go check to see what they rate it for in GB/day. Then multiply it by the warranty period. So let's say you got a 50GB/day 3 year warranty. The manufacturer therefore feels you can write about 55TB to the drive without them being all that likely to have to service the drive under warranty. Geeks love to talk endurance in terms of what things SHOULD be, but warranty periods are more likely to have statistical failure rates as a major component to the numbers. I don't know if OCZ specifies a separate TB limit to writes under warranty... something I don't have time to figure out even if I knew your SSD model, but I will note that if you've got a 50GB/day 3 year drive yours is already halfway to that 55TB number!

I solidly feel that your drives SHOULDN'T be failing that fast, but my confidence in consumer grade SSD's is such that I find myself able to say "that figures" and not even feel horribly cynical saying so. I know that's not the answer you wanted to hear, but, on the flip side, you've got the practical experience of 5 drives from multiple vendors failing quickly. You may well be the person most qualified to be our resident expert on the (un)reliability of consumer grade SSD drives.

Your second post -
If this is the case then
a) does your theory conflict with what SMART is showing? Can you tell from the above results that the disk is having a hard life? To me it looks ok (but I'm a noob)

Someone with more experience with SandForce controllers would be better positioned to give a more nuanced opinion, sorry.

b) would a bigger, say 265 or 512GB disk last longer or should I just bite the bullet and get an enterprise disk? 200GB Intel disk is £300 while a 240GB Crucial is £70. At a 1:4 ratio I don't mind if they last 2 years and throw them away...

Well, right. What's your RAM size, by the way? Do bear in mind that L2ARC tends to thrash a little bit much on systems with less than 64GB RAM. Some people here will say absolutely no L2ARC on systems less than 64. There's merit to that, but it can be used on somewhat smaller systems in certain cases. Either way, the question is how large an L2ARC is a good idea. Usually a 4:1 or 5:1 L2ARC:RAM is a safe thing to do.

 

[Crispin]

But I notice that if we look at your power-on hours, divide by 24, that gives us appx. 50 days of operational time. Then if we look at lifetime writes and divide that by days, to get GB/day, we end up with 450GB/day written.

This might make sense. Disk in question was in previous oSol NAS with ZFS. had an easy life there. I took it out and shoved it in a drawer for a while.
When I built the new NAS, this time with FreeNAS, I did an rsync from old to new. This shifted around 9-10TB of data around.
Once done I shuffled the data around again from CLI. I then did plenty of checks, scrubs, disk failure tests (pulled disks out while running) etc. I had to do 3 resilvers before I let it settle down and turn the old one off.

That would add to the average?

I'll blindly send this in for an RMA. If they do, cool, if not - bla. It's been fun while it lasted.

My current NAS, N54L microserver) has 16GB in it now, running a new jails, nothing serious.

Resident SSD breaker? I'll help if I can. I'll probably get another SSD, SCAN.co.uk has the Sandisk 250GB ones for £50 which is a crazy price.
Happy to help if I can.


Thanks for the feedback and help.

 

[cyberjock]

I personally have stuck with Intel exclusively until last year (bought one Crucial because it was a great price) because of all of the problems that other brands have had. Intel had problems, but they fixed their major ones before I started buying them. I've had zero failures out of 8 or 9 drives I've owned and probably a dozen others in friend's systems that I manage.

 

[jgreco]

The main problem with Intel is the cost. We've found it cheaper to pair up two SSD's of different types using hardware RAID1 for ESXi datastores, it gives suitable reliability characteristics at a lower cost. This is similar to the reason that many ZFS advocates will use inexpensive consumer-grade SATA drives instead of the pricey enterprise-grade nearline SATA. Given sufficient redundancy, the lower cost solution is a winner. We lose a drive now and then and just replace it from spares on hand.

 

[tvsjr]

I'm just a n00b (at least with FreeNAS), but I have to wonder... for your use case, is an L2ARC even providing any value?

 

[jgreco]

It could be, really depends on the workload and stats. It is also quite possible, on a 16GB system, that it is useless or actively detrimental...

 

[Robert Trevellyan]

What's up with #174 and #194?

174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 215

194 Temperature_Celsius 0x0022 128 129 000 Old_age Always - 128 (0 127 0 129 0)

 

[CaptainSensible]

I would not use SSDs as any part of a NAS.

My advise is to only use your NAS as storage, forget messing about cache drives, jails and all that rubbish.



Have a good main PC and just treat your NAS as a useful and stable storage device.

 

[jgreco]

I would not use SSDs as any part of a NAS.

My advise is to only use your NAS as storage, forget messing about cache drives, jails and all that rubbish.

Have a good main PC and just treat your NAS as a useful and stable storage device.

Nobody's forcing you to do anything.

However, network attached storage is used in a lot of contexts. "Have a good main PC"? Wow. We have dozens of physical and virtual machines here, all of which share NAS storage. It's very pleasant to have snappy and responsive network storage. What would you suggest instead? Having a slow filer and then upgrading all of those machines to be able to support local copies of the dozens of TB of data we keep on the filer? That'd be crazy expensive.

Maybe you meant that we shouldn't use NAS for VM virtual disk storage. Local datastores on the hypervisors do work, after all. However, that then means that you can't do live migrations of your virtual machines between hypervisors. That'd suck! And buying sufficient RAM to hold the working set in ARC would be prohibitively expensive. We currently have more than sixty VM's sitting over on the new filer, each with a working set of at least several gig. Each 128GB of RAM costs $1200 (almost $10/GB). Each 400GB NVMe SSD costs $350 (less than $1/GB). The math is obvious. Fast shared storage wins. L2ARC caching gets you there cheaply.

Of course, our needs are different than your typical prosumer, home hobbyist, or media junkie. For those people, they are already committing to running the NAS 24/7. You really want them to run another PC 24/7, in addition, just so they don't have that "jail" "rubbish"? In addition to their NAS, do you mean to suggest that you want them to have a dedicated PC to run Plex on, eating another 40-80 watts, when it is so simple to provision your FreeNAS box with a slightly larger CPU with a few extra cores and some extra memory, which only takes a handful of extra watts to do?

I don't think your advice is very wise.

 

[cyberjock]

I would not use SSDs as any part of a NAS.

My advise is to only use your NAS as storage, forget messing about cache drives, jails and all that rubbish.

Well crap. Good luck getting good performance while properly honoring sync writes.

 

[jgreco]

Well crap. Good luck getting good performance while properly honoring sync writes.

Or building a super-fast filer entirely out of SSD's... or... or ...

 

[HoneyBadger]

I would not use SSDs as any part of a NAS.

My advise is to only use your NAS as storage, forget messing about cache drives, jails and all that rubbish.

Have a good main PC and just treat your NAS as a useful and stable storage device.

You do realize that there's people here who store more than just their downloaded 0day war3z torrent rips on their FreeNAS machines, right?