RAID Level Oopsie - how to recover best?

[LIGISTX]

How you ask, I have no idea.

I somehow chose the wrong RAID Z level when I initially set up my pool. The math worked out perfectly with 1024 math so I didn't even think about it. But looks like I have a RAID Z1 instead of Z2 like I had planned since we all know Z1 is dead and gone at this point.

So........... I know "how" this has to be rectified, but I am wondering if anyone has ran into this a month into their server being deployed, and how they handled it. Thankfully a buddy of mine also has a large freenas setup so I can easily offload a lot of my data to him, and with all my random assortment of drives my data used to live on, I can figure it out. Will it be painful, yea, it will, but maybe with some thoughts from here it can be made a little bit easier.

Offloading all my directories that are shared via SMB shouldn't be that hard, but how should I go about backing up my jails data and root user data and VM's. I run a windows network, I have no real linux machines (I could throw fedora on my test bench and fiddle around with that if need be, but I am no linux expert so if this can be kept within windows ecosystem that would be helpful AF), so is there any advice how to do this?

Thankfully everything mission critical is backed up multiple places, so even if this goes FUBAR, its not like I am totally SOL.

But, hey, to living and learning I suppose. This is clearly my first "server", and I have learned a lot thus far, and what would a first build be without a monumental F up like this?

 

[artlessknave]

how many drives in the raidz1? size of drives?
do you have lots of open HDD ports?
you say you have a random assortment of drives, could you build a new raidz2 of the intended number of disks, with enough space to move everything from the old z1 to the z2, and then replace out all the random disks with the desired disks from the destroyed rz1?
perhaps you could post
zpool status -v
and
zpool list -v
for the relevant pool? (either from GUI shell or ssh)


moving jails and VM's would probably be easiest with zfs replication. if you can make a new disk pool big enough to move everything too, you could replicate the entire oldpool to newpool right from the GUI. if you need to split things up, you can also replicate to a file(or files) and then restore from the replication file to the replcatement pool

personally, the ease of management with mirrors (add/remove mirror disks, add new mirror pairs to pool), in addition to the faster resilver times, makes any of the raidz's less appealing, even at the cost of more drives

 

[m0nkey_]

To go from RAIDZ1 to RAIDZ2, you will need to destroy and re-create the pool.

 

[LIGISTX]

how many drives in the raidz1? size of drives?
do you have lots of open HDD ports?
you say you have a random assortment of drives, could you build a new raidz2 of the intended number of disks, with enough space to move everything from the old z1 to the z2, and then replace out all the random disks with the desired disks from the destroyed rz1?
perhaps you could post
zpool status -v
and
zpool list -v
for the relevant pool? (either from GUI shell or ssh)


moving jails and VM's would probably be easiest with zfs replication. if you can make a new disk pool big enough to move everything too, you could replicate the entire oldpool to newpool right from the GUI. if you need to split things up, you can also replicate to a file(or files) and then restore from the replication file to the replcatement pool

personally, the ease of management with mirrors (add/remove mirror disks, add new mirror pairs to pool), in addition to the faster resilver times, makes any of the raidz's less appealing, even at the cost of more drives

I don’t have enough drives to build a interim setup in that regard, although I may be able to build a large enough array to at least back up the most important data, VM’s and jails (since I don’t have those backed up anywhere). What exactly is ZFS replication, that sounds right up my ally here.


Sent from my iPhone using Tapatalk

 

[LIGISTX]

 

[danb35]

What exactly is ZFS replication

http://doc.freenas.org/11/storage.html#replication-tasks
The manual speaks in terms of replicating to another system, but it can go to the same system as well--just enter "localhost" as the remote hostname.

 

[LIGISTX]

http://doc.freenas.org/11/storage.html#replication-tasks
The manual speaks in terms of replicating to another system, but it can go to the same system as well--just enter "localhost" as the remote hostname.

I think I have determined my solution. I have a spare old PC, random core 2 duo setup. I’ll have at least 8 GB of ram, non ECC, and throw 4 3TB drives in RAID Z2.

Question I have is, I currently have really poor vdev and file naming structures since I am a noob and never bothered googling standards. I plan to correct this since I have this chance, but if I change the vdev name, will that utterly destroy all of my jails and VM’s since they will refer to directories that don’t exist? I am not sure how FreeNAS deals with this.

For instance, I have a windows 10 VM and a plex jail, the plex jail has mounted storage, the windows VM just has SMB shares. Will the only issues I have is change the mounted directory and change the SMB path and everything will work? Or will fundamental code break? Sorry, but I am clearly still learning a lot as I go...

I will replicate my critical data to this new FreeNAS hardware and in the future I will use this as a stagnant “critical backup” once this debacle is complete. I plan to keep this second server at my parents house and only plug it in every few months as a snapshot backup device. Separate physical location so hopefully acts of god type situations will not possibly affect both...


Sent from my iPhone using Tapatalk

 

[rs225]

Raidz1 is only dead and gone if you don't have a backup. Everyone without backups uses raidz2/3. Later on, they discover raidz isn't a backup. Everyone should use raidz1, and then they would have a backup when the inevitable occurs.

Raidz1 covers 99% of drive failure scenarios (with scrubs, tests, and notices); raidz2 gets at most 1% additional. The purpose is to accommodate long lead times for drive replacement, not to guarantee you never lose data.

So your real problem isn't what raidz level you have, it is the lack of backups. Fix that first. Murphy's Law will never be repealed.
On the day your data is lost, you'll be much better off with one hard disk filled with the most critical data, than a 2nd parity disk.

 

[LIGISTX]

Raidz1 is only dead and gone if you don't have a backup. Everyone without backups uses raidz2/3. Later on, they discover raidz isn't a backup. Everyone should use raidz1, and then they would have a backup when the inevitable occurs.

Raidz1 covers 99% of drive failure scenarios (with scrubs, tests, and notices); raidz2 gets at most 1% additional. The purpose is to accommodate long lead times for drive replacement, not to guarantee you never lose data.

So your real problem isn't what raidz level you have, it is the lack of backups. Fix that first. Murphy's Law will never be repealed.

I have cold backups, but the reason for Z2 in my eyes is so I don’t get rebuild errors upon recovering from a bad drive. Being a ~40TB vdev, math says I will have a read error upon rebuilding a drive in the event a drive dies. Z2 just elevates that risk, it also mitigates the risk of having a second drive die during the rebuild process. Not things you don’t know in sure, but these are the issues I am trying to mitigate. Nothing on here isn’t backed up in one way or another, but once it’s all set up and running it sure would be a pain to have to start over if I could have just thrown another drive for redundancy at it in the first place.


Sent from my iPhone using Tapatalk

 

[rs225]

I know only that most of the epic data loss I see on the forums is to people with raidz2 or better without backups, and isn't caused by having a first, second or third drive failure.

A single read error during a rebuild won't destroy a pool, just the data contained in that block, and ZFS will tell you exactly what file it was. Failures during resilvers are rare, because failures during scrubs are rare. They are the same thing.

 

[LIGISTX]

I know only that most of the epic data loss I see on the forums is to people with raidz2 or better without backups, and isn't caused by having a first, second or third drive failure.

A single read error during a rebuild won't destroy a pool, just the data contained in that block, and ZFS will tell you exactly what file it was. Failures during resilvers are rare, because failures during scrubs are rare. They are the same thing.

That’s fair. Personally I would feel more comfortable with Z2, and knowing it’s limitations and accounting for them via backups is still part of my strategy.

Thanks for the heads up.


Sent from my iPhone using Tapatalk

 

[danb35]

Raidz1 covers 99% of drive failure scenarios (with scrubs, tests, and notices);

I don't know that I'd agree with the exact numbers you're using, but there is way too much fearmongering (even by otherwise-very-knowledgeable folks here) going on about RAIDZ1. Certainly it carries a higher risk of data loss than RAIDZ2 (or RAIDZ3), but it's still much less than many here would have you believe.

 

[LIGISTX]

I don't know that I'd agree with the exact numbers you're using, but there is way too much fearmongering (even by otherwise-very-knowledgeable folks here) going on about RAIDZ1. Certainly it carries a higher risk of data loss than RAIDZ2 (or RAIDZ3), but it's still much less than many here would have you believe.

I think it’s safe to say in a production environment with critical data and downtime frowned upon, RAIDZ1 is dead. But, not being a sysadmin, I would go as far as to say so is Z2 and Z3. If your data is that critical and downtime is just not an option, I think most of our home brew servers are really not what you are after anyways. You need physical replication of servers, which I know you and other more experienced people on here know WAY more then myself about.

But I guess the point is, to my eyes, I think it’s worth the extra few bucks up front for a system with a lesser chance of encountering a fatal error, such as upon resilvering, a second drive dies.

But back to the main question 2-3 posts ago, what will happen when I change the vdev’s name?

I do plan to set up my secondary nas tomorrow if I have time to start fooling around with ZFS replication. Like I said, it won’t be a production box, just a offsite backup that will only be turned on occasionally. 8GB ram, an older core i3, and 4 3TB seagate (the model number that failed like 15-25%, thus the 50% parity....) should suffice for the purpose it will server.


Sent from my iPhone using Tapatalk

 

[danb35]

what will happen when I change the vdev’s name?

vdevs don't have names. Do you mean the dataset?

 

[LIGISTX]

vdevs don't have names. Do you mean the dataset?

Yes, sorry about that.

The highest level directory you see when in storage view in the webui. I thought that was the vdevs name, but I suppose I am incorrect. I thought I gave it that name when I created the vdev...


Sent from my iPhone using Tapatalk

 

[danb35]

The highest level directory you see when in storage view in the webui.

That would be the pool name.

 

[LIGISTX]

That would be the pool name.

Ok, clearly my ignorance is showing.

Still, question still stands. Do VM’s and Jails have not easily fixable underlying file structure or code that will break from changing the pool name, or are they robust enough to not be affected? I know any mounted directory in the jails will break, but I have no idea if they will still run, and have even less idea about VM’s.


Sent from my iPhone using Tapatalk