Resource icon

"Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Yes the fix isn't really a real fix, that has to come from vmware. My original problem was that 9.1 beta(& FreeBSD stable) disabled msi & msix interrupts across the board which caused interrupt storms and just flat out killed my vSANs. In the end the FreeBSD guys rewrote some code to make it smarter and got the quirks table tweaked so now FreeBSD & FreeNAS just work out of the box without any kind of tunables applied at all.

I was told that the linux kernel code vmware is using had a bug that has since been fixed in the linux kernel tree and if I can get my source to point me to the exact code I'm going to run it up the pole with vmware.
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
Well, I'm going on the crazy train! I'm going to try a virtualized FreeNAS once again, though this time with a lot more precautions. I do have two questions about you post and I could use your opinion on my plans.

1. What exectly do you mean when you say put FreeNAS on the bare metal before loading ESXi? I don't see why you couldn't just create a VM and install FreeNAS immediately in that.
2. Upgrading at 'SSD datastore'-speed seems kinda cool, but wouldn't your data be at risk all the same? Maybe I'm completely missing the point, but when you upgrade FreeNAS in the 2nd VM and it screws up your data I don't see why the 1st VM would then be able to see the data. (Sorry for my English, I find it hard to explain this particular bit)

My plan is to have 4x 1TB disks in Raid5 + 2x 1TB disks in mirror. The Raidz array will be my main pool and the mirror array my backup pool. In the main pool there will be a mix of important and entertainment data. I'll backup the important data regularly to the mirror in case something goes wrong. I will ALSO backup the important data to an external drive to be sure (yes, I've learned my lesson), although not as regularly. By using incremental backups the 1TB of storage should be plenty for long enough. It won't be an actual production server, but it will be used more serious then just experimenting. That's why I want to be sure I have the important data on three locations, he movies etc. I can download again. (I don't have the means at the moment to backup everything).
I have the VM configured with the onboard AHCI controller passthrough'ed, 12GB RAM and 2 vCPU's. The ethernetcontroller is virtual, although I might change that later because I suspect Jails do NOT like virtual interfaces. Furthermore I took heed of all your advice in your post (Reserve memory etc.), at least as much as I can (I can't afford ECC at the moment for example). I'm using a Intel DQ67OWB3 with a Core i5 2400s, which should support VT-d well enough as far as I can see.

Well, I'm curious as to what you have to say. Please keep in mind though that I have got to keep it low-budget. If that means I absolutely shouldn't put important data on it, so be it. But I think it's possible with enough safety measures installed, right?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
1. What exectly do you mean when you say put FreeNAS on the bare metal before loading ESXi? I don't see why you couldn't just create a VM and install FreeNAS immediately in that.

He means exactly that. Your pool should be available on a bare metal setup as well as the VM, if you use VT-d. If you don't use VT-d, then you are already in choppy waters because all of the other configuration options aren't reliable.

2. Upgrading at 'SSD datastore'-speed seems kinda cool, but wouldn't your data be at risk all the same? Maybe I'm completely missing the point, but when you upgrade FreeNAS in the 2nd VM and it screws up your data I don't see why the 1st VM would then be able to see the data. (Sorry for my English, I find it hard to explain this particular bit)

I'm not really sure what you mean by this. You should only have 1 VM for FreeNAS. I will say that bootup times, normal operation, and upgrades aren't significantly faster on an SSD than a USB stick. There are times when a config file change is faster, but once you setup FreeNAS I can't imagine you will be making so many changes so frequently to make the SSD worth the investment.

I have the VM configured with the onboard AHCI controller passthrough'ed, 12GB RAM and 2 vCPU's. The ethernetcontroller is virtual, although I might change that later because I suspect Jails do NOT like virtual interfaces. Furthermore I took heed of all your advice in your post (Reserve memory etc.), at least as much as I can (I can't afford ECC at the moment for example). I'm using a Intel DQ67OWB3 with a Core i5 2400s, which should support VT-d well enough as far as I can see.

If other's experience is any example, any board that isn't a server board generally doesn't have VT-d technology that works as well(if at all). I have a prosumer board that claimed VT-d, and even when enabled didn't really work. Frankly, I wouldn't recommend VT-d without server hardware after my own personal experience with it as well as the experience of others I've seen in the forum.

Well, I'm curious as to what you have to say. Please keep in mind though that I have got to keep it low-budget. If that means I absolutely shouldn't put important data on it, so be it. But I think it's possible with enough safety measures installed, right?

If cost is your primary controlling factor you should probably abandon FreeNAS in a VM or never ever trust it with data. Generally everything works until it doesn't. For most people everything is all fine and dandy and then one day they wake up and everything is gone or they reboot and now the pool is gone.

Also risky is using non-ECC RAM with ZFS. It'll work great until it doesn't. And if your situation is like most other people's, you'll have little or no warning that your RAM is bad until you've trashed your pool, the corrupted data has been backed up to the backup locations, and you have no good copies of your data anywhere.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
1. What exectly do you mean when you say put FreeNAS on the bare metal before loading ESXi? I don't see why you couldn't just create a VM and install FreeNAS immediately in that.
2. Upgrading at 'SSD datastore'-speed seems kinda cool, but wouldn't your data be at risk all the same? Maybe I'm completely missing the point, but when you upgrade FreeNAS in the 2nd VM and it screws up your data I don't see why the 1st VM would then be able to see the data. (Sorry for my English, I find it hard to explain this particular bit)

For a bare metal install, pull the drive you have install ESXi on, if you don't have ESXi installed yet you could use the drive to put your test FreeNAS install on. The point of this step is to validate that your hardware works before you introduce ESXi into your setup. You can put FreeNAS on a USB stick if you need to for testing.

Generally there is no real difference in risk between running FreeNAS standalong vs in a VM. The issue is that putting it in a VM allows people to make some generally really bad choices(like not enough ram, use virtual drives for your ZFS pool storage, etc).

Make sure you test your RAM with Memtest86 for a day or 2 to make sure your ram is good, though ECC is best but many budgets don't allow for this.

Finally like CJ said you should be using vt-d to pass in your disk controller for your drives, otherwise you will have a problem getting the raw disks into the VM. It's critical that FreeNAS has direct access to the disks so it can detect drive failures. In other ESXi this task is typically done by the RAID controller, ESXi itself won't detect your drives going bad until the party is all over and your data is gone.
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
He means exactly that. Your pool should be available on a bare metal setup as well as the VM, if you use VT-d. If you don't use VT-d, then you are already in choppy waters because all of the other configuration options aren't reliable.
I'm sorry, I've ahd a long night, but I still don't really get it I think.. This is what I think he means: Install FreeNAS on bare metal, install ESXi alongside it, install FreeNAS in a VM and copy the config from bare-metal FreeNAS (after everything works) and run it in VM with VT-d. Correct?

I'm not really sure what you mean by this. You should only have 1 VM for FreeNAS. I will say that bootup times, normal operation, and upgrades aren't significantly faster on an SSD than a USB stick. There are times when a config file change is faster, but once you setup FreeNAS I can't imagine you will be making so many changes so frequently to make the SSD worth the investment.
No, I don't mean I want a SSD, nevermind that. I was just wondering why you would have two VM's as jgreco suggests, because I don't think it protects against dataloss. Is it merely for the ease of upgrading?

If other's experience is any example, any board that isn't a server board generally doesn't have VT-d technology that works as well(if at all). I have a prosumer board that claimed VT-d, and even when enabled didn't really work. Frankly, I wouldn't recommend VT-d without server hardware after my own personal experience with it as well as the experience of others I've seen in the forum.
I admit that I'm probably a bit naive in this, but wouldn't Intel support their own technology?

If cost is your primary controlling factor you should probably abandon FreeNAS in a VM or never ever trust it with data. Generally everything works until it doesn't. For most people everything is all fine and dandy and then one day they wake up and everything is gone or they reboot and now the pool is gone.

Also risky is using non-ECC RAM with ZFS. It'll work great until it doesn't. And if your situation is like most other people's, you'll have little or no warning that your RAM is bad until you've trashed your pool, the corrupted data has been backed up to the backup locations, and you have no good copies of your data anywhere.
Well, thats a risk I'm willing to take for now given that I can have proper backups of the important data. And wouldn't a regular scrub detect the corrupted data? Thus enabling me to repair whatever's wrong and put back a healthy backup?
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
For a bare metal install, pull the drive you have install ESXi on, if you don't have ESXi installed yet you could use the drive to put your test FreeNAS install on. The point of this step is to validate that your hardware works before you introduce ESXi into your setup. You can put FreeNAS on a USB stick if you need to for testing.

Generally there is no real difference in risk between running FreeNAS standalong vs in a VM. The issue is that putting it in a VM allows people to make some generally really bad choices(like not enough ram, use virtual drives for your ZFS pool storage, etc).

Make sure you test your RAM with Memtest86 for a day or 2 to make sure your ram is good, though ECC is best but many budgets don't allow for this.

Finally like CJ said you should be using vt-d to pass in your disk controller for your drives, otherwise you will have a problem getting the raw disks into the VM. It's critical that FreeNAS has direct access to the disks so it can detect drive failures. In other ESXi this task is typically done by the RAID controller, ESXi itself won't detect your drives going bad until the party is all over and your data is gone.

I will test the memory. And VT-d works fine so far, is it possible that it will stop working fine over time?
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
I will test the memory. And VT-d works fine so far, is it possible that it will stop working fine over time?

Not likely, it's more of an issue that your system supports it and the disk controller your using works with it. Speaking of which what your are using for your disk controller?

FYI: I'd be prepared for a lot of learning in the near future, esp if you want to data to be safe. ZFS will give you data protection like no other system, but it will take's a bit of knowledge to keep from shooting yourself in the foot. Sure FreeNAS makes it really easy to get a up and running system, but many folks skip the reading step and end up with disaster down the road. Read as much as you can and tinker with it and try stuff before you commit data to it.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'm sorry, I've ahd a long night, but I still don't really get it I think.. This is what I think he means: Install FreeNAS on bare metal, install ESXi alongside it, install FreeNAS in a VM and copy the config from bare-metal FreeNAS (after everything works) and run it in VM with VT-d. Correct?

No, you will only have 1 installed copy of FreeNAS. It will run in the VM.


No, I don't mean I want a SSD, nevermind that. I was just wondering why you would have two VM's as jgreco suggests, because I don't think it protects against dataloss. Is it merely for the ease of upgrading?

You'll have at least 2 VMs because you'll have FreeNAS plus at least 1 other. If you don't have another VM you shouldn't be running ESXi.


I admit that I'm probably a bit naive in this, but wouldn't Intel support their own technology?

Some hardware supports it, some doesn't. Pretty much only the server grade stuff should be expected to work with VT-d.


Well, thats a risk I'm willing to take for now given that I can have proper backups of the important data. And wouldn't a regular scrub detect the corrupted data? Thus enabling me to repair whatever's wrong and put back a healthy backup?

No, you are missing the point. Your original copy of the data will be trashed, permanently. Then that corruption will propagate to your backups. The system will not be aware that it corrupted your files and there will not be an uncorrupted copy anywhere. Scrubs will actually make it MUCH worse because it will potentially corrupt the entire pool. Remember, a scrub verifies that all data is good. But if the data is good(but thanks to bad RAM) it thinks its bad it'll try to fix it. But it won't actually fix it. It'll corrupt it! Without ECC you will be unable to determine you have corruption from RAM nor will you be able to fix it, ever. Not even with backups.

Having bad RAM is just like asking someone with Alzheimer's what they have forgotten. If they could tell you then they wouldn't have forgotten it, would they? The bottom line is they have no clue what they forgot and you have no way of knowing either.

It's like this. If you have non-ECC RAM and a stick goes bad, kiss your pool goodbye. For everyone except 1 person they had massive data loss. That one person got very very lucky. They built the system and did some things and realized that the system had bad RAM before they did anything with it. You won't be so lucky. The system will run great for a while(and might not even give you a hint that RAM is bad). But you'll eventually figure it out, and at that point it will be too late.
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
Not likely, it's more of an issue that your system supports it and the disk controller your using works with it. Speaking of which what your are using for your disk controller?

FYI: I'd be prepared for a lot of learning in the near future, esp if you want to data to be safe. ZFS will give you data protection like no other system, but it will take's a bit of knowledge to keep from shooting yourself in the foot. Sure FreeNAS makes it really easy to get a up and running system, but many folks skip the reading step and end up with disaster down the road. Read as much as you can and tinker with it and try stuff before you commit data to it.
That it exactly what I was going to do. :) Oh and the diskcontroller I'm using for FreeNAS is the onboard controller of the Intel DQ67OWB3 board. It's apparantly called 'Cougar 6 Point Controller' or something in ESXi.

No, you will only have 1 installed copy of FreeNAS. It will run in the VM.
...
You'll have at least 2 VMs because you'll have FreeNAS plus at least 1 other. If you don't have another VM you shouldn't be running ESXi.
Nevermind, I can't seem to make my point clear.

No, you are missing the point. Your original copy of the data will be trashed, permanently. Then that corruption will propagate to your backups. The system will not be aware that it corrupted your files and there will not be an uncorrupted copy anywhere. Scrubs will actually make it MUCH worse because it will potentially corrupt the entire pool. Remember, a scrub verifies that all data is good. But if the data is good(but thanks to bad RAM) it thinks its bad it'll try to fix it. But it won't actually fix it. It'll corrupt it! Without ECC you will be unable to determine you have corruption from RAM nor will you be able to fix it, ever. Not even with backups.

Having bad RAM is just like asking someone with Alzheimer's what they have forgotten. If they could tell you then they wouldn't have forgotten it, would they? The bottom line is they have no clue what they forgot and you have no way of knowing either.

It's like this. If you have non-ECC RAM and a stick goes bad, kiss your pool goodbye. For everyone except 1 person they had massive data loss. That one person got very very lucky. They built the system and did some things and realized that the system had bad RAM before they did anything with it. You won't be so lucky. The system will run great for a while(and might not even give you a hint that RAM is bad). But you'll eventually figure it out, and at that point it will be too late.
Thanks, that actually sounds like something I should consider. But then why isn't this an issue when you're not virtualizing?

And maybe this is a bit off-topic, but could hashing help detect these kind of errors? Example: I have the data on my PC, say in a zipfile. I have a MD5-hash of that file. I copy the data to the server, it becomes corrupted without my knowledge and some time later I copy it to another PC. I'll then create another hash and compare the newer and original to see if the file is the same or corrupted. And actually now that I'm typing this I'm thinking I could just Google this too probably. Don't bother answering if you don't want to ;)
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
That it exactly what I was going to do. :) Oh and the diskcontroller I'm using for FreeNAS is the onboard controller of the Intel DQ67OWB3 board. It's apparantly called 'Cougar 6 Point Controller' or something in ESXi.

You will need a controller to boot off of and a controller to hang your FreeNAS disks off of. If you lucky your board has multiple controllers on it, google on PCI pass through for ESXi and see if you have a disk controller to set aside. Be careful not to set aside your boot controller because ESXi doesn't work without a disk too well(the latest version I think warns you if you try this).
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
You will need a controller to boot off of and a controller to hang your FreeNAS disks off of. If you lucky your board has multiple controllers on it, google on PCI pass through for ESXi and see if you have a disk controller to set aside. Be careful not to set aside your boot controller because ESXi doesn't work without a disk too well(the latest version I think warns you if you try this).
I use an Adaptec 1210SA for ESXi to hold the datastore, and FreeNAS boots off of the datastore. The pools will be on the Cougar controller.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Thanks, that actually sounds like something I should consider. But then why isn't this an issue when you're not virtualizing?

And maybe this is a bit off-topic, but could hashing help detect these kind of errors? Example: I have the data on my PC, say in a zipfile. I have a MD5-hash of that file. I copy the data to the server, it becomes corrupted without my knowledge and some time later I copy it to another PC. I'll then create another hash and compare the newer and original to see if the file is the same or corrupted. And actually now that I'm typing this I'm thinking I could just Google this too probably. Don't bother answering if you don't want to ;)

It is a problem to consider when virtualizing. But you'd be a complete and total idiot if you were going to virtualize and not use VT-d technology. If you plan to use VT-d, you are literally in server hardware, which uses ECC RAM. So if you are doing one thing you are pretty much doing the other.

Yes, you could hash every single file you own, and you could rehash it later to see if its changed. But unless you plan to keep a backup copy of every single file so you could restore said file, hash every single file before you use it, etc you are not going to benefit. Remember, ZFS hashes its data too, and it can't find the corruption because the saved hash is based on the corrupted data. But then you are talking about keeping a separate backup(and it cannot ever be sourced from the pool). So what you are really saying is "I want to save some money and I plan to inconvenience myself...forever; and I plan to make religious backups of my data BEFORE it ever makes it to the zpool..forever". Does that sound like a cost effective solution in the long term? My logic says "no freakin' way". It sounds like its just simpler, faster, and cheaper to simply use ECC RAM.
 

Thomas

Dabbler
Joined
Jun 30, 2013
Messages
29
It is a problem to consider when virtualizing. But you'd be a complete and total idiot if you were going to virtualize and not use VT-d technology. If you plan to use VT-d, you are literally in server hardware, which uses ECC RAM. So if you are doing one thing you are pretty much doing the other.

Yes, you could hash every single file you own, and you could rehash it later to see if its changed. But unless you plan to keep a backup copy of every single file so you could restore said file, hash every single file before you use it, etc you are not going to benefit. Remember, ZFS hashes its data too, and it can't find the corruption because the saved hash is based on the corrupted data. But then you are talking about keeping a separate backup(and it cannot ever be sourced from the pool). So what you are really saying is "I want to save some money and I plan to inconvenience myself...forever; and I plan to make religious backups of my data BEFORE it ever makes it to the zpool..forever". Does that sound like a cost effective solution in the long term? My logic says "no freakin' way". It sounds like its just simpler, faster, and cheaper to simply use ECC RAM.
I see what you mean. For the moment I'm going to keep backups of everything tha absolutely cannot be lost. For the rest I'll take my chances with normal RAM. I'm not talking about any long-term solutions here. If I ever decide to build a NAS that will survive World War Three I'll be sure to use ECC RAM. Thanks for your input! :)
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
I see what you mean. For the moment I'm going to keep backups of everything tha absolutely cannot be lost. For the rest I'll take my chances with normal RAM. I'm not talking about any long-term solutions here. If I ever decide to build a NAS that will survive World War Three I'll be sure to use ECC RAM. Thanks for your input! :)
Doesn't sound you understood what he told you.
He's telling you that if your FreeNAS pools are the primary source of your backups, your backups will essentially be useless, also.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Doesn't sound you understood what he told you.
He's telling you that if your FreeNAS pools are the primary source of your backups, your backups will essentially be useless, also.

Exactly. Your backups will get trashed if your ram in the server has problems. This is because the original copy is trashed.

Now if you think that nonecc is reliable enough for you that's your choice. But that's something I'd never advocate based on forum feedback from users that have lost everything despite having a solid backup schedule and doing everything else right. Its a single point of failure you cannot plan around, predict, or identify before its too late.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
But that's something I'd never advocate based on forum feedback from users that have lost everything despite having a solid backup schedule and doing everything else right. Its a single point of failure you cannot plan around, predict, or identify before its too late.

I agreed it's a rather nasty single point of failure, but if several people have lost their pools due to bad RAM then there has been a rash of folks building boxes out of left over parts(which is the all to common approach to building one's house server- though a very wrong one). But let's look at reality, most NAS systems for home use dont' have ECC ram and every single OS/file system combo will corrupt your files with bad ram and no fsck tool will fix it it either. All files must go through RAM before they get to the disk of any NAS system and if that RAM has bad bits well then you data is toast. What I think we need to preach is to burn in your system and memtest your ram ECC or not. Bad ECC ram causing random reboots will soon do some damage to ones pool also. Too many people just start copying data onto the box they just built without any idea if it's reliable or not. Just because it's not crashing doesn't mean it's not flipping bits like crazy where your file data cached in ram.

In summary bad ram will take anything down and most NAS systems sold for home use don't use ECC. I don't think that's a good thing, but it's reality.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Right, but some bad RAM won't potentially trash every single file on the entire server array. Typically that corruption is limited to files that are created/modified after a certain date. So what if the system is trashed, you mount the array with a fresh copy of Windows and 80%+ of the stored files will be fine. Most people will live with a few corrupt files. Just look at how much data corruption occurs daily around the world because of UREs or silent corruption on hard drives that you have no clue about. That's just a reality today, unfortunately.

With ZFS, it can and pretty much is an expected side effect of bad RAM to eat every single file on your pool, then belch it out all over your screen with a "zpool unmountable" equivalent error. And then you think that you are awesome and have backups that are made every night and you'll just recover from backups. But then you find out that those religious scrubs you've been doing actually trashed virtually ALL of your files. You'll be less than thrilled with the outcome. We(the FreeNAS community) aren't privy to the information Sun knew when they first created ZFS, but we are definitely seeing firsthand what Sun probably knew all along. The consequences for bad RAM with ZFS cannot be overstated. It can cause data corruption that can relicate through all of your files, through all of your backups, and you aren't likely to know about the issue until its too late to fix it.

Before Apple everything was ECC. Apple wanted to save some money back in the late 70s and early 80s and also realized that while it would suck to have some stuff get corrupted because of RAM, it wasn't necessarily an "end of the world" scenario. For ZFS though, it pretty much is an "end of the world" scenario. But apple realized they could save a boatload on RAM costs if 1/9th of the RAM cost wasn't necessary. So non-ECC was born, and has stick around because its cheap.

Wanna know something cool about RAM? It was predicted that error rates would increase(due to background radiation) at a rate that would be about double the rate as you double the density. Makes sense, right? You have 4GB of RAM and you'd expect twice as many errors as 2GB of RAM. Except it's not double the error rate. In fact, it's not even close. It's far less than expected and nobody really understands why. It was also expected that as transistor size shrank that error rates would increase(due to background radiation). As the transistor gets small more high energy radiation would affect a larger % of the total electrons in the transistor, leading to more bitflips etc. Except, again, the error rates would increase, but at a far slower rate than expected. Nobody really understands this effect. Just that its been documented.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Oh heavens, I've got to buy cyberjock (and probably pbucher) a beer for shouldering the load here. I'm on a bit of a hectic schedule right now. Looks like a perfect storm of new vSphere, new E5, and FreeBSD 9.2 coming out right around the same time. Heh.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Looking forward to the new E5 & FreeBSD 9.2, hoping the new vSphere is a bit more polished then 5.1 initial was. I'm hearing that 5.5 should be the pinnacle of the 5.x series that we can run for a few years(or at least until we cave into what every new goodies they cook up in 6.0 next year).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It becomes ever more challenging to figure out the whole VMware ecosystem as they keep adding and changing various features and modules. Has there been any estimate as to when 5.5 will actually hit FCS?
 
Top