Migrating To New Drives, Ironwolf HDD Pro? Other?

[isopropyl]

So, your pool's total space is now 25.4 TB. usable space is 20.3 TB. If we add a single n-way mirror vdev (it's not important how wide the mirror is, the space increase is the same) made of 20 TB drives, we have to add 20 TB * 0.8 = 16 TB of usable space.

So that means 20.3 + 16 = 36.3 TB of usable space with ALL your 7 vdevs of 4TB drives + the new mirror of 20TB drives; the spares do never count because they don't add any space.

Now, of that 36.3 TB you would be using 19.3 TB for your current data: 36.3 - 19.3 = 17 TB of free, usable space.

Each 4 TB mirror vdev adds to the pool 4 * 0.8 = 3.2 TB of usable space, as such: 4 (vdevs) * 3.2 = 12.8 TB

If we remove those 12.8 TB from the 17 TB of free, usable space we get 4.2 TB of free, usable space with the single n-wide mirror vdev of 20 TB drives and 3 2-way mirror vdevs of 4TB drives, plus the couple of spares.​

It took me a couple reads, but I think I understand. I kept losing you at "Now, of that 36.3 TB you would be using 19.3 TB for your current data: 36.3 - 19.3 = 17 TB of free, usable space.", however I see why we went further than that. I didn't realize you were calculating all my current drives plus the new 20tb drives. So you calculated it as a whole, then after you removed the 4 vdevs and got 12.8TB. Makes sense.

So simply put, after adding the 20tb drives, and removing the 4 devs, I will be left with 4.2tb of usable space.

However, currently I have 5.9TB of UNUSED free space in my pool. So I can safely add on 4.2TB to that correct? And I would have 10.1TB of UNUSED free space, correct?
By unused, I mean no data on it. Free. I can put stuff on it. Keyword, unused.

Or would that 5.9TB actually lower itself to 4.2TB, because of how you did the math?

P.S.: both of your spares are currently in use inside different vdevs.​

Are 2 of My Drives Failed? (See Edit: Moving Data Onto To New Vdev, To Remove Old)

Normally I wouldn't suggest weekly scrubs, but we are outside of standard procedures. You could try to reach to HGST (or WD?) in order to see if they have a way to fix this but I wouldn't be very optimistic. I suggest you to update the firmware of your HBA anyway. Edit: found this. It has a...

www.truenas.com

Yeah I know, you commented on it remember :p
I never really figured it out, I was waiting for two replacement drives to come in the mail just in case.

This does scare me a little though, because they are in use how would I go about choosing which hard drives to remove?
Worst case scenario, I take them out, and it copies the data but if something fucks up, I just put the 4tb drives back in no issues, no?

When "lose a drive" is stated when it comes to mirrors or raid, that implies the drive actually dying/dies correct? If I simply unmounted/disconnected 2 healthy drives in an entire vdev, nothing bad would actually happen, the pool would just be dead, but when I plug them back in, the pool would return to normal, correct?

Like say I disconnect both of my two drives in mirror 1.
Nothing bad actually happens correct? As long as the data on the two drives stays in tact, I can simply plug that mirror back in and all is good.

 

[Davvo]

So simply put, after adding the 20tb drives, and removing the 4 vdevs, I will be left with 4.2tb of usable space.

Correct.

However, currently I have 5.9TB of UNUSED free space in my pool. So I can safely add on 4.2TB to that correct? And I would have 10.1TB of UNUSED free space, correct?
By unused, I mean no data on it. Free. I can put stuff on it. Keyword, unused.

Nope, that is already factored in the calculations. We have added to the total usable space (which is 80% of the total space due to how ZFS, being a COW, handles space) of your current pool (so all your vdevs) the usable space of the new vdev, then we have subtracted the space you are currently using for your data and that's how we get to that final number of 4.2 TB.

Or would that 5.9TB actually lower itself to 4.2TB, because of how you did the math?

Exactly. If you want more space, remove less vdevs: each one adds or removes 3.2 TB of usable (and free, because you wouldn't be able to remove the vdev otherwise) space.

Yeah I know, you commented on it remember :p

Not after 4 am

This does scare me a little though, because they are in use how would I go about choosing which hard drives to remove?
Worst case scenario, I take them out, and it copies the data but if something fucks up, I just put the 4tb drives back in no issues, no?

Generally, yes. But there are proper procedures in order to detach a mirror vdev or replace a drive, you can't just offline it and take it out of the pool.

As long as the data remains on the drives, there is hope.

When "lose a drive" is stated when it comes to mirrors or raid, that implies the drive actually dying/dies correct? If I simply unmounted/disconnected 2 healthy drives in an entire vdev, nothing bad would actually happen, the pool would just be dead, but when I plug them back in, the pool would return to normal, correct?

If you don't sofware-detach the 2-way mirror vdev but instead just offline/physically detatch two drives of the same vdev from the system, your entire pool goes offline and stays so until the drives are made available to the pool.

Like say I disconnect both of my two drives in mirror 1.
Nothing bad actually happens correct? As long as the data on the two drives stays in tact, I can simply plug that mirror back in and all is good.

You pool goes offline, your nas gives you warnings. The moment you put those drives back everything should (hopefully) return to normality.

 

[Etorix]

Little confused by that last statement. Are you saying you would prefer not to do a 3 way mirror with 16 tb drive?

I said I personally would NOT use a 2-way mirror with 16 TB drives, and thus recommend 3-way mirrors with drives that large.

And I can eventually expand then to be 4-way mirrors by just adding on an extra drive to the vdevs in the future if I felt the need I guess.

Indeed. But at that point you really need a good reason to go for mirrors rather than raidz3 (+spares) because the cost by usable TB of storage is staggering.

But I don't understand how you state that it uses more drives than raidz2, because raidz2 as Davvo stated requires 4 drives, whereas a 3 way mirror only requires 3 drives in each vdev. So for raidz2 I'd need 1 more drive.

Indeed, but I wrote "for the same capacity". For the (raw) capacity of 3 drives, you need 9 drives (3 vdevs) with 3-way mirrors but only 5 drives with raidz2 (1 vdev). Mirrors win hand down for IOPS (x3) and storing little blocks; raidz2 wind hands down for cost.

 

[Davvo]

Indeed. But at that point you really need a good reason to go for mirrors rather than raidz3 (+spares) because the cost by usable TB of storage is staggering.

Which usually can only be performance.

 

[isopropyl]

Ok coming back to this because something needs to be done with my pool here. It's starting to concern me.
So, 20TB Exos drives. I can go with 6gb/s SATA drives, or 12gb/s SAS drives.
I plan to run 10g in the near future. But also my work load is not crazy. I'm not really doing any video editing, and if I was wait time wouldn't be a big factor. I do store all my files on it though. I stream games off of it, but I was considering in the future, maybe setting up a small SSD pool specifically for games. So beyond that, I do use Jellyfin, and will have a Proxmox server reading data off of it.

I'm trying to weigh if going SAS for an extra $100 per drive is worth it. Because that comes out to be a lot of money. If I eventually end up with 9 drives, that's almost another grand before tax and shipping.
I'm leaning towards just going with the 6gb/s SATA drives.
Also with multiple mirror vdevs it should be pretty damn fast still.

Are the SAS drives significantly more reliable too though? Or are they pretty close.
Both Exos drives, just SAS vs SATA. They will be plugging into my SAS backplane. SATA being more universal does give me some more options in the future as well to move them to a non SAS machine if ever needed, or reuse them elsewhere.



I will buy 3 of the SATA drives on the spot so I can get things started.
Just would appreciate a final input before I throw $1200 to them.

Will I run into any issues seeing my current pool is all SAS, and the new drives would be SATA?

As for adding them into my current pool, how would that work? Because I'm not replacing the entire pool, but only the few vdevs.
Would I replace a drive in use with the 20tb, let it resilver, then attach another 20tb as a 3rd drive in that mirror, and then replace the other drive that was in that mirror with the 3rd 20tb?
I haven't had a chance to setup the other HBA so Idk if it's working. So if I have to simply put in all 3 drives at once, while all the other drives remain in the system, that may be a problem because I have no more spare bays.
I may be able to run a sata cable to the motherboard, not sure.

 

[isopropyl]

Actually, really thinking about it.. I do think I will just go with the SATA drives.
Once this pool is converted to the new drives completely, I can look into building a backup system. And determine if I want to upgrade to SAS then.
And I can utilize the SATA drives in the backup system if desired.
This seems like the best route to go.

Now, my sanity questions at the bottom of the last post are still relevant though:
>Will I run into any issues seeing my current pool is all SAS, and the new drives would be SATA?
>Any issues with the mismatched capacities in the same pool?
From my past research, and understanding, both should be fine.

What I am lost on is the proper way to add them into my current system.

 

[Davvo]

I'm trying to weigh if going SAS for an extra $100 per drive is worth it.

In your situation, it's not worth going from SATA to SAS drives.
Also do note that SATA drive are SAS compatible, but not the other way around: that can become pretty significative when you have to troubleshoot an issue.

 

[isopropyl]

Also do note that SATA drive are SAS compatible, but not the other way around: that can become pretty significative when you have to troubleshoot an issue.

Yeah ik I stated that. I actually realize a benefit in my last post, and thats in the future I can reuse the sata drives without worrying about them needing to go in a system that supports sas.

Also again, trying to understand the proper way to actually replace the vdevs. Seeing I'm not replacing the entire pool upfront, but doing it slowly?