Adding a 6TB drive to a 3x2TB pool

[MrMe01]

As the title says, I am trying to add the disk and I get this error message;

Code:

You are trying to add a virtual device of type 'stripe' in a pool that has a virtual device of type 'raidz

How do I add this disk, or what's the best/safest way of using it, if I can't add it to the pool?

If it matters, it is a VM, running under Proxmox. The system is running from a disk image file, the storage disks are attached via PCIe passthrough to a Dell PERC H200 HBA card, in JBOD.

 

[Inxsible]

You can degrade your pool and then replace the drive that you offlined by the new 6TB drive. That won't give you anything though, since all 6TB won't be used if the other drives are of different sizes.

What is your end goal? Do you want to use the entire 6TB space?

 

[MrMe01]

Ideally I would like to use it as an additional 6TB minus whatever the redundancy data will be, I didn't add it when I set it up as it contained the data I have since put on the array.

 

[Inxsible]

You obviously cannot change the existing vdev which is set up as RaidZ1. If you would get another 6TB, you can create a 2nd vdev in a mirrored configuration. That will give you your 6TB space to use. And since you have RaidZ1, you have only 1 drive that can fail without data loss anyway and using a 2-way mirror won't change that.

If more than 1 drives fail in any vdev, your entire pool will be toast however. But you probably knew that when you chose RaidZ1.

If you use only 1x6TB as stripe, you lose the redundancy completely because if the single striped drive fails, you lose your entire pool. No point in having the RaidZ1 then.

Having said that, mixing Raidz1 with mirrors is something I don't do personally.

Make sure to burn in the 6TB drive before adding it to the pool. Same with the new one if you get another 6TB drive.

 

[MrMe01]

I don't have the spare cash for the extra 6TB drive at the moment, or for the foreseeable future, I am, however going to rebuild with 4 2TB drives instead of 3. I'll use the 6TB drive as offline/cold storage.

It's mostly films, ISO's and music, but there are other, more important things I should keep offline, like backups.

 

[Inxsible]

Using it as a spare/backup would be the best bet then. Is there any particular reason why you are choosing RaidZ1 ?

So with those 4x2TB drives

RaidZ2 - 2 drives for parity = 4TB total data space
RaidZ1 - 1 drive for parity = 6TB total data space
Mirror vdevs - 2x2TB + 2x2TB = 4TB total data space


Why not use mirrors? It gives you same amount of redundancy as RaidZ1 with 3 advantages

1) You could lose a drive in both mirrors and still be ok
2) Much faster resilvering compared to any RaidZ configuration
3) Ability to add another mirrored vdev later with a different sized drive.

 

[MrMe01]

You make a good argument for mirror vdevs, and I get that I am trading disk space for much better redundancy, but losing that space doesn't seem to be the best usage of all that capacity. I know, I'll lose my data and I'll be cursing this very thread, it doesn't seem logical. But I'm gonna do it, as it's not just my data that I am going to be storing.

 

[Inxsible]

You make a good argument for mirror vdevs, and I get that I am trading disk space for much better redundancy, but losing that space doesn't seem to be the best usage of all that capacity. I know, I'll lose my data and I'll be cursing this very thread, it doesn't seem logical. But I'm gonna do it, as it's not just my data that I am going to be storing.

Good luck.

 

[Inxsible]

If you really wanted to add the 6TB drive in your NAS, you could do one thing that I didn't think about earlier.

You could create a 2nd pool and add this 6TB in there. Since you only have 1 drive, you wouldn't get any redundancy, but you will still be able to use all 6TB of space.

The good thing about using a separate pool, is that if the 6TB drive fails for any reason, your first pool with all the data is not affected. You would only lose data which is on the 6TB drive. Later, if and when you get another 6TB, you can convert the striped drive to a mirror.

Note however that you will not be able to transfer data between the 2 pools.

 

[MrMe01]

Good luck.

Thank you :)

If you really wanted to add the 6TB drive in your NAS, you could do one thing that I didn't think about earlier.

You could create a 2nd pool and add this 6TB in there. Since you only have 1 drive, you wouldn't get any redundancy, but you will still be able to use all 6TB of space.

The good thing about using a separate pool, is that if the 6TB drive fails for any reason, your first pool with all the data is not affected. You would only lose data which is on the 6TB drive. Later, if and when you get another 6TB, you can convert the striped drive to a mirror.

Note however that you will not be able to transfer data between the 2 pools.

I am almost done shuffling data around now, only 500GB to go, or so. I did think of that myself, using it as a normal share, but I think with this setup, I actually keep my backups clean, my data safe and I have a real, but not offsite backup solution. What's even better is I am giving up all of my usable with my USB caddy drives, as the 6TB drive won't work with it as the controller is old. What this means is I am no longer going to do the whole backup and then leave accessible and then move it off when I need the space on the drive routine. I make a lot of disk images, keeping them on a USB 2TB drive is not a good idea..

 

[Inxsible]

Thank you :)
I am almost done shuffling data around now, only 500GB to go, or so. I did think of that myself, using it as a normal share, but I think with this setup, I actually keep my backups clean, my data safe and I have a real, but not offsite backup solution. What's even better is I am giving up all of my usable with my USB caddy drives, as the 6TB drive won't work with it as the controller is old. What this means is I am no longer going to do the whole backup and then leave accessible and then move it off when I need the space on the drive routine. I make a lot of disk images, keeping them on a USB 2TB drive is not a good idea..

You lost me there. Now I am not sure if you ARE creating a 2nd pool or not

 

[MrMe01]

No, just the 4x2TB drives, to give me 4TB, and the 6TB as cold storage. I have 500GB or so of data to move back onto the 6TB drive, then I can rebuild. With the 2TB drives occupied in this arrangement, I don't have the opportunity to keep large amounts of data that is USB accessible (I have a caddy that I use bare drives with, well, up until now).

I hope that clears things up?

 

[Inxsible]

Cool. So you are going with Mirrors or RAIDZ2 ?

 

[MrMe01]

I was going to go with mirrors, but I've not considered RAIDZ2. What are the the notable differences? Sorry to ask, but this is very new ground for me. Both hardware and software. I'm not new to Linux, but I am to PCIe passthrough virtualisation, SFF connectors on HBA cards, not much experience with BSD, but it's close enough to Linux to get by. This is my first real homelab server, so it's a bit of an experiment as well for me.

I managed to get a Dell T110-ii, Xeon E31220 with 32GB of RAM with 2x2TB drives recently.

 

[Inxsible]

First off, are those brand new drives or used? Either way, I hope you burned them in.

I have already pointed the difference between RAIDZ1 and mirrors. When it comes to RAIDZ2 and 2-way mirror :

• RAIDZ2 will offer a redundancy of 2 drives whereas a 2-way mirror only provides 1 drive redundancy. (3-way mirrors provide 2 drive redundancy, n-way mirrors provide (n-1) drive redundancy). In other words, your data will be safe even if 2 drives fail in RAIDZ2.
• RAIDZ2 resilvering is slower than 2-way mirrors because of the parity calculations and such.
• You need all your drives beforehand in order to configure RAIDZ2. You cannot add new drives to an existing RAIDZ configuration
  
• In a 2-way mirror, you can keep adding vdevs of 2 disks each and they can be of different sizes across vdevs. Eg. 2x6TB mirror 1 and 2x500GB mirror 2 (like I have in my NAS)

This is true for both RAIDZ2 and 2-way mirrors :

• You cannot use different size drives. Well you can, but you will only get the space equivalent to the smallest size disk

Hope that gives you some idea. For detailed information, you should research the RAIDZ configurations as well as mirror configurations.

Start with these 2 threads :

https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/

http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/


Please don't think that I am trying to dissuade you from RAIDZ2 (from RAIDZ1, yes. But not RAIDZ2). These are just 2 threads which give you some understanding as to what exactly RAIDZ2 and mirrors give you. You should research more and see what's best for you.

Please stay away from RAIDZ1 ;)

 

[MrMe01]

I keep an eye on SMART data, they're not new, but are Dell/WD enterprise grade drives. Then I have a WD Purple 2TB drive and a Hitachi 2TB drive. The 6TB drive is also a WD purple drive, they've all been put through their paces recently, a few TB of writes to them all, and none are showing any sign that there's a problem, so far.

Now that I've read that, I think I'm going to go with RAIDZ2.
I don't mind waiting for the array to be rebuilt when it comes to reslivering, even if it takes a few days, I can live with that, the two drive failure makes up for the wait.

 

[Stux]

You should look into hard drive burn in testing too.

Be worthwhile burning the 2TB disks in (even though they're not used) before you create the new RaidZ2 pool.

https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/

Backing up to a 6TB disks is a good plan.

In future, if you want to rebuild as a 6-way Raidz2, just refresh your backup, recreate your array with 2 more disks, and restore.

 

[MrMe01]

I've slept, read some more and had a think about this, I'm going to go with mirrors instead of RAIDZ2, it just makes better sense, I still get the redundancy, but I also get the speed. And by the looks of it, easier to expand.

Thank you for showing me the right way of using this software.

 

[MrMe01]

Welp, I got to the point with my data management where I could add the disks, go through a load of data and sort through things properly.

Reinstalled everything from scratch, Debian, PVE, set up PCIe passthrough, made sure all was good and ready and installed FreeNAS.

Some form of disk errors on two of the four disks when starting to the build the array. Not experienced enough to diagnose. I'll try a more familiar environment.

I tried straight Debian in the VM with mdadm, errors. Same disks. Did a full SMART test on all the disks, all come back fine.

So I decided to go nuclear again, this time using the PERC H200 in my box controlling the disks in a RAID 10. Built my stack of software as described above, running on this array. I install the Dell OMSA stuff and I see there are errors on the array. Same disks. Alarmed, I go through the menu to see what they are. Non-critical. A Google search reveals that because they're not certified drives, some Dell firmware somewhere in my system will throw up an error that somehow causes an issue with both FreeBSD and Linux, even when all the card is doing is passing through the disks to the OS.

For me to be able to use FreeNAS, all of the disks need to be certified with either this motherboard or the firmware Dell has flashed to the PERC H200. I have no idea if I can put LSI firmware on it or not, and I have only just started to look at doing such things. I post this here before I search for it or create a new post as the folks above may already be able to help, and as a followup.

 

[MrMe01]

It looks like I need to flash my card, https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/