RAIDZ1 resilvering on SSD.

[scotch_tape]

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut at rutrum mauris, a bibendum nisl. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Duis gravida odio in eros porta porttitor. Proin ornare tincidunt mattis. Vestibulum vel nisl dignissim, consectetur felis sed, tristique augue. Nulla semper elit nec sem pulvinar, non convallis velit tincidunt. Nam egestas accumsan quam. Mauris efficitur nisi eget tellus suscipit, vitae posuere lacus suscipit.

 

[cyberjock]

Well, according to some research on cost comparision you won't see the cost per GB of SSDs hit the same cost until almost 2020. So I'm thinking "no time in the near future".

SSDs are their own beast and have their own failure modes. With SSDs it seems to be a "all or none" scenario. Either they work properly or they poop all over the floor.

 

[cyberjock]

But not with a bunch of 2TB drives at home! There's a difference when enterprises want a 24TB all SSD pool and when your average forum user wants it. Anybody that has the financial backing to go to a 24TB all SSD pool will NOT be coming to the forums. They'll call iX and get a ready-made system.

You want a forum drag, how about you think about what the target audience is that shows up in the forum.... you seem to have a serious disconnect between what the average person wants to do here and what the average enterprise with 6-figure pockets do.

 

[cyberjock]

BTW: Here's your reference. http://www.networkcomputing.com/sto...cheaper-than-hard-drives-not-in-thi/240164894

Keep in mind that companies don't go "all SSD" because they can. The go "all SSD" for performance reasons. And generally speaking, its likely to be far cheaper to go with platter based pools with ZILs and L2ARCs as necessary than to jump out and buy 2TB SSDs for 4-digits each.

Also, (and you may be surprised by this), but SSDs lose some of their umph going to ZFS. They still do smoking fast random writes and reads compared to platter disks. But they have their own limitations that become more apparent with ZFS from what I've read about them. Obviously I haven't tested this myself as I don't have that kind of cash to find out for myself. ;)

 

[cyberjock]

Oh, and to answer your question about resilvering SSDs. The big "question mark" with SSDs on ZFS is that they will theoretically all wear out at about the same time since data is striped across them. So unless one fails due to a hardware issue or otherwise fails prematurely, all of the SSDs are likely to wear out at almost the exact same time. Not exactly a good thing as you'll have to pre-emptively resilver your pools as lifespan gets low to make sure you don't end up with multiple failed disks at the same time causing a loss of the pool.

One company I talked to has literally gone to a schedule where they determine the estimated lifespan of the drives and replace one every 2-3 months, good or not. This causes all the drives in the pool to have staggered lifespans and you'll avoid the potential situation of multiple failures in quick succession.

 

[cyberjock]

Technically, I'm running an "all" SSD pool. I have 2x32GB SSDs in a pool. It goes pretty fast, it does what I need it to do "test the heck out of beta versions of FreeNAS" and such. I'd never consider my whopping 64GB pool to be useful though. Nobody goes with ZFS and then does just 64GB.

The whole point and design of ZFS is hell-f*cking-huge and ultra-f*cking-reliable.

 

[cyberjock]

ZOMG... so really? You double the storage capacity and you almost double the disk writable capacity. You can pretty much extrapolate up from there. How hard was that to extrapolate out... and why do you want to make the argument that it matters? If you have 24TB of drives you're going to write tons of data! PERIOD. Far more than a 100GB drive because you have more space! Even more if you do mirroring or redundancy since you'll be adding mirrors, redundancy(and don't forget all the checksumming and transaction stuff).

Not sure what the point of your argument is yet... But my point is setting yourself up for all of your drives to fail at the same time is a recipe for disaster. And anyone that does enterprise class servers knows this very simple fact. They also know that they cannot easily predict how much writing the pool will need for any given timeframe, so the drive could last years or could fail in 6 months. And at the ultra-high cost of SSD versus platter based drives, guess which looks better until it's absolutely blatantly obvious to the casual computer user that the money is well spent.

You also have lots of companies that will not go to all SSD because they won't deem it a "reliable" technology yet. Many industries won't touch a new technology for at least 10 years. ZFS has been out for 9 years now, and many companies are just starting to touch ZFS. And not surprisingly they are doing everything that we tell people not to do. They do non-ECC RAM, they do ZFS RAID on a hardware RAID, etc. Many of our recommendations apply to ZFS as a technology completely aside from FreeNAS. And companies are doing it so wrong its almost entertaining. Been talking to 2 companies that have actual ZFS deployed out in the field, are complaining about everything there is to do with ZFS and hate it with a passion, and its because they didn't deploy it properly. Just spent an hour yesterday talking to a FreeBSD developer about this whole ZFS thing and he agrees that ZFS has major problems ahead because people really don't know how to do ZFS properly and try to use incorrect assumptions and do things horribly wrong. I'm really curious to see how ZFS adoption is going to be in 3 and 5 years out because companies are getting ZFS wrong and then running from it thinking its a total piece of crap. Some after spending 5-figures on hardware!

 

[Yatti420]

@ scotch_tape these aren't your typical windows boxes.. They require planning and commitment.. The forum guard cat has been clear on an all ssd pool.. Not advisable from a cost-performance-reliability perspective.. Just throw some spinning rust in and if you really need an SSD (I'd say unlikely for the majority of users) add it at a later time..

 

[cyberjock]

I do expect that at some point in the future I will (and many people will have solid state pools). The question is if that will be in 2015 or 2025. I'm not expecting even rich geeks to be able to afford reasonably sized all solid state pools for home use until at least 2020. But, there are some of these new technologies(memristors, racetrack memory, etc.) that if pans out could make that number a year or two sooner(at best). New technologies take years to get into high production volume, so we have that wait before costs can even begin to come down.

 

[warri]

Depends very much on the use case I guess. Unfortunately I don't think SMART or any other means provide that information. I'm basically just using my NAS for storage of mostly static data, so I guess the writes amount to a very small number (probably < 20 TB per disk). On the other hand, if you run I/O intensive tasks (VMs, big backups) this number should increase significantly. A quick search on Google didn't reveal any actual real life data, but I only looked very briefly.

 

[cyberjock]

Your questions is basically NaN(Not a number).

Platter based drives aren't limited to a total number of writes. It's basically whenever the drive has a mechanical or electrical problem. You don't wear out platter drives by writing to them like you do SSDs.

SSDs have a fairly well known lifecycle based on the design of the memory cells. To boot, the manufacturers deliberately lower the bar on total writes so that you don't see disks failing before you hit 100% writes. The latest Samsung EVO drives are rated for 1000 writes(which you can hit in just a 2 weeks or so if I remember correctly), but errors didn't occur until over 3000 writes had occurred. To know what the expected lifespan of any given SSD you'd have to consult with the manufacturer.

 

[cyberjock]

Which is why I am wondering if 36.5 petabytes of writes on a 2 terabyte ssd drive would be pretty reasonable compared to an actual hard disk.

And I answered that above. The answer is NaN. It's apples to oranges. SSDs are limited to finite writes(ignoring manufacturer's defect or firmware defect). Platter drives do not have such a limitation. But, if you took my drive(42k lifetime power-on hours) and I wrote at its limit(about 110MB/sec) nonstop for those 42000 hours you'd get a total number of 15.8Petabytes written so far.

Now this is all theoretical and "apples to oranges" and here's why.

If you write a single byte to an SSD, it's technically one byte, but it counts as 4k written since that's the smallest block of data you can write for all of the drive I've been able to get data on. That cell is considered to have been "written to". To add to that, if you wrote a 1 byte file to an NTFS partition the file would be stored in the MFT record itself. For larger files(say a 1GB file written) you will have that 1GB of file data plus the accompanying NTFS metadata. So you've got abstraction due to the physical characteristics of writes, abstraction due to the file system chosen(ext does some cool tricks too), then "really big hairball in your vomit" abstraction because the SSD abstracts the entire "media".

Of course, SSDs can often do 450MB/sec or more so they could reach their limits potentially MUCH faster. But if their write endurance is high enough then they could beat out most platter drives. And instead of looking at total Petabytes written it could be broken down into individual writes performed. The numbers that these manufacturers come up with are both conservative and liberal. And since you get a dose of both the manufacturer pretty much gets to add as much of each as they want to come up with numbers to make their drives look better than the competition. No competitor seems to make their numbers look like they are blatant BS(which is good for consumers).

So how do you figure out how long an SSD will last if you install it in a new server? Nobody really knows until you install it and start using it. Which is why I said above too many enterprises won't do SSD because they can't provide those numbers with any kind of reliability. Enterprises aren't going to pay BIG money to put a server together to find out that in 5 months your disks are going to wear out.

And to finally close my post I'll say this. I've been buying Intel SSDs since the week after G2s hit the market. I've never had one fail and all of my G2s are still in use. I've had friends buy Intel SSDs for their machines. They all show 99% drive life remaining. I'm sorry, but that is fscking amazing! Before I went all SSD on my desktops/laptops I could expect to have to do at least 1 RMA a year between all of my systems. Now I haven't done a drive RMA except for my server. So my personal take is that SSDs are freakin' amazing if you buy a quality product. And I think any enterprise that is concerned that their SSD is going to wear out prematurely needs to do more research. SSDs are amazing devices. They can have amazing lifespans and can be amazingly fast. Their density and cost leave something to be desired, but technology is always marching along.

All of my Intel SSDs are currently estimated to reach end of like in the year 2178 or something.

 

[jgreco]

And I answered that above. The answer is NaN. It's apples to oranges. SSDs are limited to finite writes(ignoring manufacturer's defect or firmware defect). Platter drives do not have such a limitation. But, if you took my drive(42k lifetime power-on hours) and I wrote at its limit(about 110MB/sec) nonstop for those 42000 hours you'd get a total number of 15.8Petabytes written so far.

So how do you figure out how long an SSD will last if you install it in a new server? Nobody really knows until you install it and start using it. Which is why I said above too many enterprises won't do SSD because they can't provide those numbers with any kind of reliability. Enterprises aren't going to pay BIG money to put a server together to find out that in 5 months your disks are going to wear out.

Well in a way I think you guys are talking past each other here; you're both really saying the same sorts of things.

Any idea how many writes your average 2tb hard disk does? (in its lifetime)

This is a perfectly valid question for purposes of estimating cyberjock's second quoted paragraph. For example, the 30TB filer here is intended to be archival in nature, so if we look strictly at production rather than burn-in traffic, it is entirely possible that the average sector will see a single write over its entire service life. It doesn't really matter if it is maybe two or three or four; order-of-magnitude estimates are sufficient to see that that this is trivially within an SSD's endurance.

The math for 36.5PB on a 2TB drive is about 18K write cycles. Generally a competent admin is going to be able to determine whether that is inside or outside the endurance requirements for a storage subsystem. For a system with loads of constantly-changing data, it is probably insufficient. For a system where all the data changes once a day, it is absolutely sufficient (assuming a 10 year service life, only 3650 writes per sector).

Now, here's the thing. I'm going to just come on out and say cyberjock's wrong, not in theory but rather in practice. If you have a 2TB SATA3 SSD, it is able to write at a maximum of about 600MB/sec. To write the entire unit at peak speed would require about 3500 seconds. You can do that about 24 times in a day. So given that sort of treatment, an 18K write cycle drive would last 750 days. But bear in mind that the drive is exclusively busy doing writes. While there are a few applications I can picture with such abuse, a typical ZFS pool is not one of them, so if we allocate the drive as 100% busy with 33% writes and 67% read, the endurance figure stretches out to 2250 days, or 6.1 years, which is greater than the average life expectancy of critical storage systems in an enterprise. At the point where we have 4TB SATA3 SSD's, it seems like endurance would become nearly irrelevant as long as service life was bounded to five years.

And then you have to remember that with a NAS, you're both limited to the speed of the network interfaces and you're spreading that data across multiple platters, which reduces the load.

As the size of the device grows dramatically but other characteristics do not, it is worth re-evaluating your preconceptions about longevity.

 

[fracai]

Now, here's the thing. I'm going to just come on out and say cyberjock's wrong, not in theory but rather in practice. If you have a 2TB SATA3 SSD, it is able to write at a maximum of about 600MB/sec. To write the entire unit at peak speed would require about 3500 seconds. You can do that about 24 times in a day. So given that sort of treatment, an 18K write cycle drive would last 750 days. But bear in mind that the drive is exclusively busy doing writes. While there are a few applications I can picture with such abuse, a typical ZFS pool is not one of them, so if we allocate the drive as 100% busy with 33% writes and 67% read, the endurance figure stretches out to 2250 days, or 6.1 years, which is greater than the average life expectancy of critical storage systems in an enterprise. At the point where we have 4TB SATA3 SSD's, it seems like endurance would become nearly irrelevant as long as service life was bounded to five years.

I was going to make a similar point that given cyberjock's statement regarding 42k hours (4.8 years) and 15.8 PB, you can extrapolate that an SSD with 36.8 PB could be expected to go more than 10 years. It's not going to be that simple, but it seems like the numbers at least don't preclude an SSD pool. The drives all failing at once would definitely be a concern, but that should be able to be handled by monitoring drive life and early replacement. Plus, if the lifetime really is 6-10 years, how likely is it that the drives wouldn't have already been replaced for upgrading capacity?

 

[jgreco]

bingo, which is why you need to truly grok the totality of the problem.

 

[cyberjock]

Yeah, the problem is there's very little solid info between practice and theory. Remember that TRIM does actually eat up write cycles too. After all, TRIM does move the data around at the physical level so it can erase pages that are empty. The question is whether TRIM will eat up a small number of write cycles or extremely large number of write cycles. That's hard to guesstimate even if you assume a single manufacturer. And each one has their own level of "aggressiveness" for TRIM providing optimal performance. The more aggressive the TRIMming the more write cycles lost. So if you want an aggressive TRIM system on your disk you are also claiming you want shorter drive lifespans. That's one of many things that the manufacturer's have to juggle and decide to do the way they want.

I don't consider me to be wrong in practice either. I made it very clear that the comparison was apples-to-oranges. I even went so far to explain that you've got abstraction of "disk lifespan" at multiple levels. If you want to elaborate I'm all ears though.

 

[Dusan]

After all, TRIM does move the data around at the physical level so it can erase pages that are empty.

As far as I know TRIM doesn't move any data around. It's the garbage collection that moves data around. TRIM only marks blocks as unused which results in GC actually needing to move less data around (it decreases the write amplification). It is explained here: http://www.thessdreview.com/daily-n...ion-and-trim-in-ssds-explained-an-ssd-primer/

 

[cyberjock]

I stand corrected on the choice of my words.

TRIM doesn't move the data, it tells the SSD controller that those areas are "available" on the disk and the actual GC moves the data. The fact remains that the data is moving around all the time with and without your writes. Of course, if you do no writing at all eventually the disk will reach a steady-state.

 

[cyberjock]

Not in the fashion you are describing.

There's other more significant factors such as all of the disks hitting no life remaining at the same time. The fact that all of the disks will hit EOL at about the same time is far more serious than being able to rebuild without an error.

 

[diskdiddler]

Ok wtf is going on, I was curious about SSD's with FreeNAS (I'm planning my 2017 build..... god help me can I hold out that long...?)

The OP's posts have turned to Lorem ipsum copy / paste with.. an unpleasant thumbnail image?