Hard drive failure = bad things happen

[SubnetMask]

I have been playing with FreeNAS, testing it and such. In a previous thread, I voiced my feelings about the lack of redundant controllers, and that feeling still stands, but despite that, I have had some thoughts of possibly 'throwing caution to the wind' and making a move to FreeNAS. Overall, it's been very stable, performance really has been great, and it supports VMWare related features that my old Promise doesn't even know exist. Plus I'm sure FreeNAS is a bit less picky about what drives over 2TB I use - With the Promise, due to its age, I'm limited to a few SAS models over 2TB. Granted, I shouldn't need any more 4TB drives any time soon, and most of the drives are smaller 1 or 2 TB drives for the VMWare VMs, but the idea of not being cornered into a few options is nice. To be fair, one of my biggest reservations to using something like FreeNAS, which isn't 'AS purpose built' as something like an EqualLogic, Compellent, Drobo, Promise, etc (we can argue semantics all day long - yes, the FreeNAS SOFTWARE is truly purpose built, but the hardware it runs on generally isn't quite so much - although truthfully, the Compellents change that thought process), stems from way back, I had a NAS type machine that worked well for quite some time, then all of a sudden, just 'went stupid'. The floor fell out from under it performance-wise (file transfers fell to a few KB/sec at best), and NOTHING that I did fixed it. RAM, NICs, controller, OS - tried changing everything (pretty much except the motherboard), nothing fixed it. Only thing I can figure is it was something went wrong with the motherboard. That's when I switched to a Drobo and haven't used anything remotely similar to FreeNAS since, and typically shy away from it.

I'll admit, the ZFS benefits like data integrity are attractive, and that as long as I have my boot device (Sandisk USB), the drives and a way to put it all together, the data can be brought back (such as my move from the 1U Supermicro box using a Supermicro UIO HBA to a 2U 12-bay box using a LSI 9217-8i HBA).

That being said, I did some more testing today to see how it handles a disk failure. Well, I'd have to say it didn't handle it well. Not well at all. When I pulled a drive from the RAIDZ1 volume, the system totally freaked, crashed and rebooted. That's BAD. VERY, VERY BAD. In doing some searching and reading, it appears the reason it did this was probably because there was swap space in use on the drive, and apparently, swap space is non-redundant. Again, that's BAD. VERY, VERY BAD. Unless I'm missing something, essentially, this means that no matter what you do, what kind of drives you use, how you configure them, etc, a single drive failure can or will bring down the entire system. Do I need to say it again? The only possible way around that I can see to that is if you set up a small, dedicated hardware RAID and dedicate that for swap space. Or possibly configure the system with no swap space. But I can't imagine that would be all that great either.

So is there something that I missed in the setup that would prevent such an occurrence, or is that 'just the way it is'? A Single drive crashing the system is BAD. It kinda defeats the purpose of redundancy from a reliability/uptime perspective.

 

[Redcoat]

Hey @Stux, this one has your name on it.

 

[Stux]

Hey @Stux, this one has your name on it.

To the OP, check out this post:
https://forums.freenas.org/index.php?threads/umass-storage-detached-swap-errors.59600/#post-422223

 

[Chris Moore]

You missed the latest version of FreeNAS.
Swap space is mirrored now.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[SubnetMask]

I'll take a look at the links, and Chris, if Swap is mirrored in 11.1 to prevent this very issue, then that's GOOD. VERY, VERY GOOD. :)

 

[Chris Moore]

I'll take a look at the links, and Chris, if Swap is mirrored in 11.1 to prevent this very issue, then that's GOOD. VERY, VERY GOOD. :)

It was a known issue that we have been working around, I have a system at work where swap was moved to a dedicated SSD mirror, but it is fixed now.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[SubnetMask]

Once the re-silvering is done with the drives in the unit now, What would happen if I put the drive I pulled out back in but in another slot (Obviously, this is not something that would happen with a failed drive)? Would it totally freak out because the failed drive was rebuilt onto another drive but now the original is back, or would it consider the drive defunct and give me an option to blank it and treat it as new? If it can't be put back without blowing something up and I need to destroy the volume or something before putting it back, that's fine too as right now I'm mainly testing it, and frankly, I probably could set it up better for iSCSI than I have.

 

[Chris Moore]

Once the re-silvering is done with the drives in the unit now, What would happen if I put the drive I pulled out back in but in another slot (Obviously, this is not something that would happen with a failed drive)?

Nothing. The system doesn't do anything to a drive until you tell it to do something. Even when you designate a hot-spare, it isn't so much doing it on it's own as it is following the instruction you previously gave.

Obviously, this is not something that would happen with a failed drive

I suppose that depends on why you considered the drive failed. I remove a drive when it even shows a 'pending' sector and put a fresh drive in. The removed drive gets run through a battery of tests in another system, and if it passes, it goes back in the pool of spare drives. I have removed drives, tested them, and later returned them to the pool in a different slot. More often than not, the testing I do exposes hundreds or thousands of additional bad sectors and the drive is sent for destruction. The testing I do is very similar to the burn-in testing that is done by the drive burn-in scripts referenced here:
Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Would it totally freak out because the failed drive was rebuilt onto another drive but now the original is back, or would it consider the drive defunct and give me an option to blank it and treat it as new?

It wouldn't do anything, including to pop-up any option to clean the drive. You can tell the system to clean the drive. That would be a good thing if the drive is still partitioned. The testing I do completely erases the drive, so it isn't needed, but if you don't clean the drive in any way, it will just sit there waiting for you to do something. If you try to use it as a replacement without cleaning it first, you will get an error. The nature of the error depends on the exact circumstances.

If it can't be put back without blowing something up and I need to destroy the volume or something before putting it back, that's fine too as right now I'm mainly testing it, and frankly, I probably could set it up better for iSCSI than I have.

You have options. I usually clean the drive in another system before reinserting it in the storage system. You can do a full wipe of a drive that is in the NAS.

 

[Chris Moore]

stems from way back, I had a NAS type machine that worked well for quite some time, then all of a sudden, just 'went stupid'. The floor fell out from under it performance-wise (file transfers fell to a few KB/sec at best), and NOTHING that I did fixed it. RAM, NICs, controller, OS - tried changing everything (pretty much except the motherboard), nothing fixed it.

Was that a FreeNAS system, or something else?
A thing to keep in mind is that performance with ZFS drops off significantly when the pool gets 90% filled. It begins warning you at 80% so that you have an opportunity to do something about it.

I'll admit, the ZFS benefits like data integrity are attractive

ZFS also has snapshots that are very low overhead and a big benefit of the filesystem.

 

[Chris Moore]

Because I know that you are very interested in high reliability. I thought you might want to look at these informative videos:
TrueNASX10 Hardware & TrueNAS Software Review
https://youtu.be/TgyYU6wuIgk

IXsystems TrueNAS X10 Torture Test & Fail Over Systems In Action with the ZFS File System
https://youtu.be/GG_NvKuh530

 

[SubnetMask]

Was that a FreeNAS system, or something else?
A thing to keep in mind is that performance with ZFS drops off significantly when the pool gets 90% filled. It begins warning you at 80% so that you have an opportunity to do something about it.


ZFS also has snapshots that are very low overhead and a big benefit of the filesystem.

To be honest, it's been so long, I can't say for 100% sure the specifics. I know at one point I had it running Windows server. I may have had it running a MUCH older version of FreeNAS, or perhaps it could have been something else. I really don't remember at this point. Probably the main thing that has 'scarred' me about it is that it was a single MB/controller setup, similar to what I have with FreeNAS (and how FreeNAS is typically set up), that went totally stupid, and nothing I did fixed it - not even a total OS reload (which for all I recall might have been a backup from an old version of FreeNAS to restore the data to Windows or vice versa) or swap of probably 75% of the hardware resolved the total performance tank. It's been a while, but I think it scarred me. What I saw with that is something I've never seen with any of the other 'typical commercial' options such as those from EqualLogic, EMC, etc - Compellent is a grey area as it's a lot like FreeNAS in that it's based on commodity hardware and purpose built software, but in reality, more like TrueNAS in that it has dual (or more) controllers, so if one takes a walk, the other (or another) can kick it out (if it hasn't outright failed) and take over. The Compellents I've been working with have been flat-out awesome.

Now, when I moved from the 4-bay 1U unit to the 12-bay 2U unit, I left some data I didn't care about on it as a test, and aside from the on board NICs being different on the 2U unit vs the 1U unit, requiring that I re-configure those to re-establish management access, since the 10Gb NIC was moved over, that came up right away, all data on the drives was A-OK and immediately accessible by VMWare. So that's something positive.

There are probably one or two people here that, based on my previous thread, would think that I hate FreeNAS - I don't hate FreeNAS - my hangups probably stem from the issues I just mentioned, and that I'm 'a fan of' availability (probably as a result of those issues). I'd be equally critical of a single controller system - Compellent, EqualLogic, EMC, whatever. Of the systems I work with, none have had an availability issue due to a controller issue (or any issue, really). I had a Compellent controller failure. The system didn't skip a beat. Nobody knew about the controller failure as everything kept going, the failed controller was repaired and reinstated, and everything continued on like nothing ever happened. My Promise had a controller panic (or something) and reboot - the VMs never noticed and didn't skip a beat. My EqualLogics at work? Not even a hiccup (other than a drive failure or two) in almost seven years. The only times they've gone down at all (other than me shutting them down) since I installed them almost seven years ago was due to environmental power issues that I couldn't control and the battery backups couldn't bridge. When a FreeNAS system freaks and reboots (or any single controller system really), everything notices and resets (based on my experience - obviously if the only controller goes down, everything's going to notice).

Obviously, FreeNAS has come a long way from its beginnings, and based on what I've seen so far, aside from the single disk failure crashing it which is apparently fixed in 11.1, it seems to be a really good setup - I'm just working to try to overcome my personal issues with single controller setups. After all, 10Gb iSCSI > 4Gb FC :).

 

[Chris Moore]

Obviously, FreeNAS has come a long way from its beginnings, and based on what I've seen so far, aside from the single disk failure crashing it which is apparently fixed in 11.1, it seems to be a really good setup

Even before the fix, there were some good alterations that could be made to make it more stable. For example, if you have a lot of RAM, the system is not likely to use swap. Also, there is a script that can be run to flush the swap and some people run that hourly using cron. Also, there is the trick of moving swap to a SSD that is dedicated to that purpose as an SSD is much more reliable and won't be used often for swap anyhow, it will last a very long time. I have only been running FreeNAS since 2011, but I have found it very reliable and only one controller has failed in that time. I had a spare on hand, so the down time was less than an hour and no data was lost, but I wasn't running iSCSI. When the controller failed on that system, it took half the drives offline instantly and the system freaked out and rebooted, so that wasn't good, but it doesn't happen often.
If you want the utmost in reliability, there is a cost. Just one of the controllers for our NetApp controllers at work cost as much as a TrueNAS with 77 x 10TB hard drives.

 

[SweetAndLow]

It was a known issue that we have been working around, I have a system at work where swap was moved to a dedicated SSD mirror, but it is fixed now.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

You sure about that mirrored swap statement? I can't find any info about it, do you have a link?

 

[Chris Moore]

You sure about that mirrored swap statement? I can't find any info about it, do you have a link?

No, but I have this:

Code:

> em0: link state changed to UP
> ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
> ipmi0: KCS mode found at io 0xca2 on acpi
> ipmi0: IPMI device rev. 1, firmware rev. 3.52, version 2.0
> ipmi0: Number of channels 2
> ipmi0: Attached watchdog
> GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
> GEOM_MIRROR: Device mirror/swap0 launched (2/2).
> GEOM_MIRROR: Device mirror/swap1 launched (2/2).
> GEOM_MIRROR: Device mirror/swap2 launched (2/2).
> GEOM_MIRROR: Device mirror/swap3 launched (2/2).
> GEOM_MIRROR: Device mirror/swap4 launched (2/2).
> GEOM_ELI: Device mirror/swap0.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap1.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap2.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap3.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware
> GEOM_ELI: Device mirror/swap4.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI:	 Crypto: hardware

 

[SubnetMask]

The resilver finished and all wasn't really good (NOT FreeNAS's fault though - Apparently the drive I put in has SMART errors and it's problems were dragging everything down - Needless to say that drive was removed and has now been marked as failed) - after the resilver, I upgraded to 11.1, which after that, performance sucked and I was thinking something was 'goofy' with 11.1 or something until I noticed the SMART error warning (the SMART error didn't show up - or at least I didn't notice it - until after the upgrade). Put that drive offline and performance went back up. After that and after the second resilver, the performance copying from one location to another on the same volume on the freeNAS connected via 10Gb iSCSI wasn't exactly what I expected (copying from my FC array to freeNAS pretty much netted 200MB/s or so, but FreeNAS to FreeNAS was around 100MB/s). But while not what I would have expected, it's not exactly horrible either.

But on to my actual question. Is the 'Resilver Priority' new in 11.1? I don't recall seeing it in 11.0U2. While I suppose one could set the resilver priority begin and end to something like 6:00pm and 5:45pm respectively, might it be possible to add a checkbox in a future release that sets resilver priority to high all the time? While the 18 or so hours to resilver the first time vs the 30 minutes or so the second could have been related to the problems with the other drive (before the second resilver I had found the resilver priority and set it to high during the default hours which the time it started fell within), at the same time, personally, I'd rather have the resilver priority high all the time with some performance degradation but completing it quickly, vs maintaining performance and prolonging the resilver process.

 

[Chris Moore]

Resilver Priority has been added to provide the ability to run resilvering at a higher priority. It is probably better to link to the doc as there are a lot of links embedded in it:
http://doc.freenas.org/11/intro.html?highlight=resilver priority#new-features-in-release

10Gb iSCSI wasn't exactly what I expected (copying from my FC array to freeNAS pretty much netted 200MB/s or so, but FreeNAS to FreeNAS was around 100MB/s).

This is most likely down to a combination of the number of disks you are using and the number of vdevs you have your pool split into. There are a large number of factors involved in making a pool perform at a high rate. If you will give information on the hardware you intend to use and what you intend to use it for, recommendations can be given.

While the 18 or so hours to resilver the first time vs the 30 minutes or so the second could have been related to the problems with the other drive

Yes, if you had a bad drive, that would have slowed it down, AND the way resilver used to work all the time is, it was low priority to allow the network users to still get as much performance as possible. At the same time, the duration of a reliver is dictated by the amount of data on the drive. For example, I have a pool at work with 218TB of data and it takes about 3 days to resilver a drive in that system.

Apparently the drive I put in has SMART errors and it's problems were dragging everything down

This is the reason that we usually suggest doing a burn-in on all drives before they are used:

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[SubnetMask]

I added my basic setup to my signature (it was elsewhere in my profile, but it didn't show up). Right now, aside from the core listed in my sig, there are four 3TB 7.2K SATA disks involved in the volume for my testing. I've got 8 HGST 7K3000 2TB SAS drives coming that I'll attach to FreeNAS for further testing, that should for some reason I decide not to continue with FreeNAS, are in the HCL for the Promise, so all is not lost if I should decide not to migrate over to FreeNAS. Should I make the move to FreeNAS, these would be my VMWar volumes that make up the storage for my VMs primary VMDK storage. Again, should I make the move, there are another six 4TB SAS disks that hold the bulk of my filestore data that I'd migrate over.

3 days to resilver a drive that's part of 218TB? The last time I added a 4TB drive to the Logical Disk on my Promise that consisted of other 4TB drives, it took over a month to rebuild (*LOL* *CRY*)!

This is the reason that we usually suggest doing a burn-in on all drives before they are used:

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Again, this round was testing, and to be honest, I didn't expect issues with the drive, but with the eight I have coming, I fully plan to do so as they are not 'new'. That was an issue I had with two of the 4TB SAS drives I added to my Promise - I added one to replace a failed SATA disk, and during the rebuild, it got kicked out due to errors. In the end, I had two disks kicked out and had to force the volume online/degraded to recover the data. The exact kind of testing that the 'BadBlocks' procedure can do is actually exactly what I'd like to do prior to adding one of these 'reman' disks to an array to do my best to be sure that there are no serious issues before I add them. Destructive or not, and time consuming or not, prior to adding a disk to a volume, I'd like to be able to 'stress' the drive to best ensure there are no problems with it.

 

[Chris Moore]

Between work and home I have a decent amount of experience with ZFS and I hope to offer some good advice. I will try to link to resources when I can to backup what I say with documentation as well, so you know it is not just my opinion.
If you want to use ZFS to store VMs, especially for iSCSI, you would want to have speed (glorious speed) and the best way to get speed is with a lot of vdevs (virtual devices) and the 'easiest' way to get there is with mirrored vdevs. That said, if you take those HGST 2TB drives are supposed to have 157MB/s sustained transfer speed, so 8 of them in mirrors should give you (minus some for overhead) give you around 628MB/s of speed. For iSCSI, you can also obtain higher speed by using a SLOG (Separate LOG) device, but it needs to be fast with high wear endurance. Something like this: Intel P3700 400GB HHHL AIC PCI NVMe SSD.

Here is a link that gives some good examples:
https://forums.freenas.org/index.ph...sdv-tln4f-esxi-freenas-aio.57116/#post-401275

 

[SubnetMask]

Between work and home I have a decent amount of experience with ZFS and I hope to offer some good advice. I will try to link to resources when I can to backup what I say with documentation as well, so you know it is not just my opinion.
If you want to use ZFS to store VMs, especially for iSCSI, you would want to have speed (glorious speed) and the best way to get speed is with a lot of vdevs (virtual devices) and the 'easiest' way to get there is with mirrored vdevs. That said, if you take those HGST 2TB drives are supposed to have 157MB/s sustained transfer speed, so 8 of them in mirrors should give you (minus some for overhead) give you around 628MB/s of speed. For iSCSI, you can also obtain higher speed by using a SLOG (Separate LOG) device, but it needs to be fast with high wear endurance. Something like this: Intel P3700 400GB HHHL AIC PCI NVMe SSD.

Truth be told, while 600+/-MB/s would not be heartbreaking (heck, even 300MB/s is in excess of what the Promise can do, I think), considering outside of my ESXi<> FreeNAS setup, everything is 1Gbps, so crazy performance, while welcome, isn't the end all be all. I've looked into the SLOG a bit, but holy hell - the one you referenced is hardly inexpensive lol. Not going to lie - assuming I make the jump to FreeNAS, performance equal to what I'm familiar with (at least 1Gbps) will be acceptable - anything over that would be great.