How to evolve old storage system

[fierarul]

Hello,

I have a HP Microserver with FreeNAS 9.1.1 that started with a 2x2TB mirror and then got another 2x3TB mirror for a total of 5TB usable.

I'm getting close to 80% on each pool so I have some time to think how to evolve this.

I only have 4 bays so there's no extra room, but I can remove the smaller pool if necessary as I just add data, almost never read (basically automatic laptop backups).

So, one idea to get a 6TB drive and add it to the 3TB mirror and make it a 3-way mirror. In order to make room I can remove one 2TB drive (and leave that mirror in a degraded state) or temporarily remove both drives from the 2x2TB mirror (will FreeNAS forget this pool?).

Then I can get a 2nd 6TB drive at some point, add it to the 3TB mirror, make it a 4-way mirror and remove the older 3TB drives. I expect that here if I am careful with autoexpand=on (which I have to read more about) I will get a new 6TB mirror with the same data. Now I can add back the 2TB mirror and perhaps copy all the data over.

Of course, going from 5TB to 6TB isn't very good but I still have the other drives to reuse. Or... I could go to 8TB directly.

What next though? Does a 3-way mirror make sense? Or should I aim for a RAIDZ2 with 4x6TB where I can use 12TB? But... how can I evolve the current mirrors in time to get there? Seems to me you only do a RAIDZ2 when you have all the disks at hand.

Note I don't care about performance as I'm the only 'user', all I care is data safety and space available.

 

[Arthur Hanlon]

You could offline one of the 3TB and replace with a 6TB, resilver shouldn't take too long depending on how full the mirror is. Once resilvered you could do the same with the remaining 3TB and then you will have yourself a 6TB mirror. This is what I've just done to replace 2TB WD Reds with 8TB Ironwolf NAS drives and it worked a treat but it's not without risk so make sure you have a backup.

I think how you have suggested doing it is the recommended way to prevent data loss during resilver but a mirror resilver doesn't tend to be too hard of the drives from what I hear.

RAIDZ2 will give as much usable storage as 2 mirrors but the mirrors will be faster so not sure if that would be the best choice and yes, I think you would have to have to destroy everything and recreate the pool with all drives to setup a RAIDZ2.

I must offer a disclaimer that I am still learning and there will undoubtedly be someone with a better grasp on what you are trying to do here that me.

 

[sretalla]

RAIDZ2 will give as much usable storage as 2 mirrors but the mirrors will be faster so not sure if that would be the best choice and yes, I think you would have to have to destroy everything and recreate the pool with all drives to setup a RAIDZ2.

This comment pretty much sums up that option...

You can expect more IO performance with mirrors, but if you have 2 drives fail, you have the risk that half the time, those 2 disks will be rom the same mirror... equals a dead mirror and an entirely dead pool (including the non-impacted mirror). RAIDZ2 allows any 2 drives to fail and you're still OK.

So with that in mind, balance your risk vs performance apetite and make the changes to suit.

 

[fierarul]

> You could offline one of the 3TB and replace with a 6TB, resilver shouldn't take too long depending on how full the mirror is.

But why offline anything when I could just make it a 3-drives mirror? (And I temporarily degrade the other 2x2TB mirror). Then, after the 6TB drive is part of the 3-drives mirror I could remove one of the older drives and make it a 2-drives mirror.

I just hope that auto-magically the mirror will grow to 6TB if all the drives are 6TB.

> RAIDZ2 will give as much usable storage as 2 mirrors but the mirrors will be faster so not sure if that would be the best choice and yes, I think you would have to have to destroy everything and recreate the pool with all drives to setup a RAIDZ2.
> RAIDZ2 allows any 2 drives to fail and you're still OK.

Yes, RAIDZ2 is much nicer than 2 mirror pairs but I can't afford to buy 4x6TB drives at once (and then copy the data somehow).

I'm thinking that for safety purposes a 3-drives mirror might actually be best. But then I need to go all the way to 8TB drives I guess.

I'm quite confused as it seems that the price/GB didn't go down since I bought the last drive years ago. I have WD Reds but I think I could buy cheaper stuff like a Seagate Barracuda.

 

[Arthur Hanlon]

But why offline anything when I could just make it a 3-drives mirror? (And I temporarily degrade the other 2x2TB mirror). Then, after the 6TB drive is part of the 3-drives mirror I could remove one of the older drives and make it a 2-drives mirror.

I just hope that auto-magically the mirror will grow to 6TB if all the drives are 6TB.

Didn't say you had to but it's an option. I mentioned it as it's exactly what I just did but yes, you could degrade your smaller array and add a 3rd drive in to safegaurd the data but it puts the smaller pool at risk, albeit minimal.

You are right that it should automatically grow, mine certainly did after the final resilver and I didn't have to change any settings.

As for price, you'll find that many people are shucking drives to get the best price but I didn't want to take the risk as the last WD external I bought had a WD blue in it so I just opted for a straight purchase of a NAS specific drive and get the benefit of the warranty too :)

 

[nikalai2]

Hi,

There is no WD Blue inside WD My Book 8TB (both revisions) from what i know. As you said, it would be better to go directly to 8TB.

Regards!

 

[fierarul]

> As for price, you'll find that many people are shucking drives to get the best price but I didn't want to take the risk as the last WD external I bought had a WD blue in it so I just opted for a straight purchase of a NAS specific drive and get the benefit of the warranty too :)

I heard of this, but didn't know it's called 'shucking' Indeed, a 'Seagate Backup Plus Hub 8TB' seems to be cheaper than the Seagate Barracuda even! But can one trust backups to these shucked drives?

> As you said, it would be better to go directly to 8TB.

Leaning towards this...

 

[Arthur Hanlon]

Hi,

There is no WD Blue inside WD My Book 8TB (both revisions) from what i know. As you said, it would be better to go directly to 8TB.

Regards!

Mine is a WD Elements 6TB not bought for the purpose of shucking, just bought to have but it definately has a Blue in it.

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Blue
Device Model:     WDC WD60EZRZ-00GZ5B1
Serial Number:    WD-WXXXXXXXXX
LU WWN Device Id: 5 0014ee 20f59fc01
Firmware Version: 80.00A80
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May  7 17:40:21 2019 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

 

[Arthur Hanlon]

> As for price, you'll find that many people are shucking drives to get the best price but I didn't want to take the risk as the last WD external I bought had a WD blue in it so I just opted for a straight purchase of a NAS specific drive and get the benefit of the warranty too :)

I heard of this, but didn't know it's called 'shucking' :) Indeed, a 'Seagate Backup Plus Hub 8TB' seems to be cheaper than the Seagate Barracuda even! But can one trust backups to these shucked drives?

> As you said, it would be better to go directly to 8TB.

Leaning towards this...

I think you have to watch for warranties etc. but I've seen nothing that would suggest that the drive quality is any lower than standard internal drive. I must admit it's super tempting.

 

[Arubial1229]

Mine is a WD Elements 6TB not bought for the purpose of shucking, just bought to have but it definately has a Blue in it.

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Blue
Device Model:     WDC WD60EZRZ-00GZ5B1
Serial Number:    WD-WXXXXXXXXX
LU WWN Device Id: 5 0014ee 20f59fc01
Firmware Version: 80.00A80
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue May  7 17:40:21 2019 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Element = Blue, EasyStore = Red

 

[fierarul]

What about WD My Book New 8TB USB 3.0? Seems to be a WD80EZZX, not intended for NAS. So many choices...

 

[Arthur Hanlon]

Element = Blue, EasyStore = Red

Good to know, thanks however, you don't get Easystore in the UK, that's a Best Buy only thing, right?

 

[HoneyBadger]

Careful with shucking external Seagates, that's a good way to end up with a shingled drive and tank your write performance horribly.

 

[fierarul]

> Careful with shucking external Seagates, that's a good way to end up with a shingled drive and tank your write performance horribly.

The shingled technology is something else I learned today. Everybody recommends avoiding it.

I don't mind write performance as I can wait for the backup to finish. All I care is that the data doesn't corrupt more easily in time due to the shingled tech (this might mean more often scrubbing perhaps?).

But isn't a normal Seagate Barracuda also shingled? In the end good old WD Red seems the simplest bet.

 

[nikalai2]

@fierarul I guess we live in the same country. :) I bought at least six WD My Book 8TB external drives and all of them are WD80EZAZ - white lable. A lot of peaple are saying that this is Ultrastar He10-8 SATA without TLER (you can use a script in freenas to enable it on startup) This model has 3.3 power reset pins but I have not encountered any problems because I am using a Supermicro chasis with backplane.

I had three 8TB Seagate Backup Plus. I shucked two of them and i returned the third one and bought a 8TB WD My Book from the same store (the biggest online resealer from our country, you can return orders in 30 days). I had the following reasons"

- you cannot shuck them without breaking tabs and this means that the warranty is lost
- those disks are slow, SMR technology, more suitable for backup / archives; you can find some detailes here

You can find tutorials on youtube for shucking WD external drives without bracking tabs. I am using stickers with serial numbers and purchase dates. I am sure that oneday it will be needed to use the warranty.

I am always looking for online deals. On holidays period you can find the WD My Book 8TB at a good price on Amazon DE or UK but you pay the shippment too. Maybe you have a friend with Amazon Prime.

I am said everytime i see that 130$ price on Amazon US / Bestbuy. I have no relatives unfortunatelly in US. You cand find good prices on 8TB / 10TB drives on auction sites but without warranty. :(

P.S. I don't recommend to shuck drives. I did it because i didn't want to pay more for internal drives and i took the risk. This was the cheapest option for me. I can only hope that in the future price for SSD's will go lower and we will afford to replace the spinners. :D

Cheers!


LE: I observed later HoneyBadger's post about SMR, sorry. I

I think write performance is important for resilvering process.

 

[HoneyBadger]

But isn't a normal Seagate Barracuda also shingled?

Seagate hasn't exactly been forthcoming about clearly labeling their drives as shingled or not. Their enterprise lines are safe, but almost all of their new consumer gear is SMR.

SMR drives are a bad combination for ZFS because ZFS is a "copy on write" style of filesystem that relies on being able to quickly write into free space, then go back and overwrite the now-outdated areas with other fresh data. This works fine on traditional PMR drives since there's no penalty to a rewrite, but SMR introduces the need to "re-shingle" the data that gets blown away by the larger write head.

 

[fierarul]

Thanks for the SMR technology link. Indeed, the cheapest Seagate Barracuda I see is ST8000DM004 which is the SMR super-slow drive mentioned in that post. Well, clearly Seagate is to be avoided.

I'll just stick to WD although they are pricy. The WD Purple WD81PURZ seems to be more reasonable.

All this drive market segmentation is getting on my nerves. Why don't they just publish read/write speeds and let me decide if it's good enough?

> SMR drives are a bad combination for ZFS because ZFS is a "copy on write" style of filesystem that relies on being able to quickly write into free space, then go back and overwrite the now-outdated areas with other fresh data. This works fine on traditional PMR drives since there's no penalty to a rewrite, but SMR introduces the need to "re-shingle" the data that gets blown away by the larger write head.

Thanks for the explanation. I think ZFS could be optimised here for SMR. Clearly during resilvering you could schedule the writes as to reduce "re-shingle" events. But considering these are consumer drives I doubt somebody is stressing out there how to write the ZFS/SMR planner.

 

[HoneyBadger]

Thanks for the explanation. I think ZFS could be optimised here for SMR. Clearly during resilvering you could schedule the writes as to reduce "re-shingle" events. But considering these are consumer drives I doubt somebody is stressing out there how to write the ZFS/SMR planner.

It's a bit of an odd situation certainly. SMR zones are currently 256MB, so trying to operate in smaller chunks means you'd need to do a lot of housekeeping and garbage collection. It's almost similar to how an SSD might program its NAND in 4K/8K blocks, but erase it in 1M - it needs spare area and time to tidy up the dirty pages before it can cleanly write again. ZFS might be a good candidate since it already has the "transaction group" concept, but it would require a fair bit of work and some pretty tight hardware requirements to say "okay, we have to write a minimium of 256MB to each member disk at a time here."

Compounding this issue is the fact that all consumer-available drives are DM-SMR (Drive-Managed) and hide their inner workings. You can get Host-Aware (HA) or Host-Managed (HM) firmware ... but all I can say about that is [PARAGRAPH REMOVED DUE TO NDA] ;)

Once filesystems have better support and vendors release HA-SMR/HM-SMR devices to the public, they might be more viable.

Right now, SMR has decent adoption in the enterprise space for BLOB (Binary Large OBject) storage - think archival-tier storage for medical imagery that gets written and has to be retained for seven years, but can't be spooled to tape because it needs to be available if a doctor wants to review an old CT scan. When you're handling petabytes of data that's mostly WORM (Write Once, Read Many) then you just want to buy 'em cheap and stack 'em deep - SMR and other recording tech like HAMR/MAMR are still going to keep spinning disks relevant in that space for some time.

Storage nerd, away!

 

[fierarul]

So it seems one of the 3TB drives is giving me ' 3 unreadable (pending) sectors'. Which as far as I can tell means bad sectors.

I'm thinking of replacing it with a Seagate IronWolf 6TB ST6000VN0033 (or a 4TB WD RED). Anything wrong with the Ironwolf? It's not SMR as far as I could find.

I've been reading this page about replacing a failing disk https://www.ixsystems.com/community/resources/replacing-a-failed-failing-disk.75/ but the advice seems wrong to me.

I don't want to resilver my new drive based on a single disk. Makes more sense to keep the 2-drives mirror as-is, extend it to the 3rd drive and resilver from both original disks, even if one of them has some bad sectors. Why would one degrade the pool to a single disk and then resilver from that?

 

[fierarul]

PS: SMR is an odd thing to look out for. Found this thread where multiple Seagate models are listed: https://forums.anandtech.com/threads/what-consumer-hard-drives-have-smr-platters.2525313/