Suggestions for expanding storage

[Mr. Bigglesworth]

Hello

I'm running a bit low on storage and need to expand with new disks.
At the moment I'm running 1 zpool consisting of 1 RAID-Z2 vdev with 6 x WD RED 4TB
My idea was to take the vdev apart and create a new one consisting of 9 x WD RED 4TB but in RAID-Z3 instead of RAID-Z2
I could also make mirror vdevs and just expand with 2 drives at a time perhaps in a new zpool?
The reason for creating a new zpool was to only lose some of the data in case 1 vdev fails - Restoring from backup is time consuming.

Any thoughts on this?

My hardware:

Supermicro X10SRI-F
E5-2675v3
128GB ECC RAM
2 x LSI SAS9220-8i flashed in IT mode
Mellanox ConnectX-3
The FreeNAS installation is running in ESXi with 64GB RAM reserved and passthrough on both of the storage controllers.
The network is 10gbit.

Thanks

 

[joeschmuck]

You should post how much added storage you are looking for, it will make a difference for us to give you sound advice.

Also, you are stating that you my like to add mirrored vdevs, this means that if both of those drives fail (infant mortality maybe) then all your data is lost. Also by adding three more drives and creating a RAIDZ3 at the same time, you have only added 8TB of storage (approx. value), is that really all you need? Also this means that you will need to destroy your old pool so do you have all your data backed up somewhere so it can be restored after you have recreated the pool? Lastly, you should be fine with a RAIDZ2 if you are sticking with the 4TB drives, it does not take a long time to resilver these. If you were looking at 6TB or larger then I would recommend RAIDZ3 for certain, but I will not talk you out of RAIDZ3, it's nice to have the extra safety of that hard drive.

 

[Mr. Bigglesworth]

Hello

To be honest my storage needs will probably increase over time, but so far my strategy is to buy extra when I need it.
Getting an extra 8TB would probably last me at least 3 years.
Right now I have an offsite backup, the speed is a bit slow but it works - I know that I should probably have an extra backup, but at the moment this is all I have.
If i was going to create a new RAIDZ3 pool with 9 drives I could borrow some drives that I would copy the current data to before taking it apart, so it wouldn't be a huge problem, just a bit time consuming.
3 years down the road it might be more cost efficient to make a new pool with maybe 6 TB drives when I run out of storage again
The reason i opted for RAIDZ3 was simply the amount of drives and not the time it takes to resilver them, I'd rather use an extra drive for parity than losing it all because of hardware failure.
I see your point with mirrors, which is why I would prefer RAIDZ2 or RAIDZ3 myself.

 

[joeschmuck]

Of course hard drives will become less expensive over time however they do not drop significantly over a 3 year period. I'm still waiting for the $160 (USD) 6TB drive and it's been close but not there yet. The holiday sales in the USA should be starting soon so I am crossing my fingers for luck that the drives drop in price.

There are two main reasons to use RAIDZ3, 1) Long resilver times, 2) Data is very important. If your goal is to jump up to 6TB drives or larger in the future then running RAIDZ3 is smart to setup early so all you have to do is replace the drives as they fail. When you start hitting the 6TB drive size then you are looking at long resilver times if you have a fairly full pool.

It sounds like you got this under control, good luck with your plans moving forward.

 

[Mr. Bigglesworth]

Thanks

It seems right now I should consider if I'm going to replace the 4TBs with 6TBs in the future.
I'm currently using WD REDs
The prices in Denmark are at the moment:

4 TB - 161,65USD (40,41$/TB)
6 TB - 254,53USD (42,45$/TB)
8 TB - 315,94USD (39,47$/TB)

So considering the price/TB I should start buying 8TB drives now, unfortunately seen from an economical point of view I can only afford 2 x 8TB now which means a mirror would be the only setup I could use.
I will however definately put some thought into the 4TB vs 6TB and RAIDZ2 vs RAIDZ3 setup, I'll need to figure out if I might do a 4 to 6TB disk swap down the road.

 

[danb35]

I'm still waiting for the $160 (USD) 6TB drive and it's been close but not there yet.

I got that (maybe $180) about a year and a half ago, but that was with "white label" drives. Not recommended.

 

[joeschmuck]

I got that (maybe $180) about a year and a half ago, but that was with "white label" drives. Not recommended.

NewEgg has the WD Ae 6TB drives at $160 each. Looking at the specs I'm going to pass on these drives. I need to wait a little longer. I'll hit my 5 year point on my WD Reds in 9 days. If I find the drives at $170 then I'll likely bite the bullet and buy them.

 

[Linkman]

NewEgg has the WD Ae 6TB drives at $160 each. Looking at the specs I'm going to pass on these drives. I need to wait a little longer. I'll hit my 5 year point on my WD Reds in 9 days. If I find the drives at $170 then I'll likely bite the bullet and buy them.

Currently the HGST 6TB 7200RPM NAS drives are close, with a NewEgg promo code they are $179 each, but not quite there yet.

 

[SweetAndLow]

NewEgg has the WD Ae 6TB drives at $160 each. Looking at the specs I'm going to pass on these drives. I need to wait a little longer. I'll hit my 5 year point on my WD Reds in 9 days. If I find the drives at $170 then I'll likely bite the bullet and buy them.

There is always the option to shuck the wd easy store 8tb for $160-$200 each.

 

[joeschmuck]

There is always the option to shuck the wd easy store 8tb for $160-$200 each.

I had no idea it contained a WD Red 8TB drive and a 3 year warranty. I suspect you would need to retain the external enclosure to place the drive back into if you needed to RMA the drive. $200 is still $80 cheaper than a single drive. This is some odd marketing. Now I'm actually thinking about doing this, it makes financial sense. Now to look for these on sale since I still only want to pay $160 per drive, it's not like I actually need an 8TB drive. Four of these drives would give me 4TB more than I'm looking for with no added cost. I can live with that.

 

[SweetAndLow]

I had no idea it contained a WD Red 8TB drive and a 3 year warranty. I suspect you would need to retain the external enclosure to place the drive back into if you needed to RMA the drive. $200 is still $80 cheaper than a single drive. This is some odd marketing. Now I'm actually thinking about doing this, it makes financial sense. Now to look for these on sale since I still only want to pay $160 per drive, it's not like I actually need an 8TB drive. Four of these drives would give me 4TB more than I'm looking for with no added cost. I can live with that.

Yeah my next purchase is going to be the 8tb easy store. I'll burn them in while in the enclosure so I can return them if needed. If they pass burn in they are probably just fine.

 

[Chris Moore]

I suspect you would need to retain the external enclosure to place the drive back into if you needed to RMA the drive.

I thought that also, it is true of the Seagate drives, but I was told that WD still honors the warranty even if you send the bare drive in.

 

[joeschmuck]

I thought that also, it is true of the Seagate drives, but I was told that WD still honors the warranty even if you send the bare drive in.

I think I would punch the drive serial number into the system and see what the warranty comes back as, if the drive alone is listed then I don't see an issue sending it in without the enclosure. One site (NewEgg) said there was a 3 year warranty, all other sites I looked at said it was 2 years for this specific external drive.

 

[Chris Moore]

Any thoughts on this?

How many drive bays / SAS ports do you have available? When I needed more space, I added a second RAID-z2 vdev to my system so I have 1 pool with 2 vdevs of 6 drives each. If you have the room for that number of drives, it improves the IOPS because of the additional vdev and more drives.

 

[Mr. Bigglesworth]

How many drive bays / SAS ports do you have available? When I needed more space, I added a second RAID-z2 vdev to my system so I have 1 pool with 2 vdevs of 6 drives each. If you have the room for that number of drives, it improves the IOPS because of the additional vdev and more drives.

I have room for a total of 16 drives.
I think right now I'm restrained by my finances, a new 6-disk vdev is a bit much for me .

I've already received 3 new disks, in the meantime one drive in my current array decided to fail on me so I replaced it with one of the new disks.
I'm probably going for a 8-disk RAIDZ2 array now.
I could theoretically in the future go for an extra 8-disk array, but I'll need to save up some money first.

 

[ethereal]

I think I would punch the drive serial number into the system and see what the warranty comes back as, if the drive alone is listed then I don't see an issue sending it in without the enclosure. One site (NewEgg) said there was a 3 year warranty, all other sites I looked at said it was 2 years for this specific external drive.

you don't need to send the enclosure back - wd honor the warranty of the hdd that is in there. i have done this 3 or 4 times over the years and never had any problems. i sell the cables and the usb board that comes with it to make them a little cheaper.

 

[Chris Moore]

I will however definately put some thought into the 4TB vs 6TB and RAIDZ2 vs RAIDZ3 setup, I'll need to figure out if I might do a 4 to 6TB disk swap down the road.

Something to think about:
I went through that a little more than a month ago. I swapped out one vdev worth of my older 2TB drives for new 4TB drives. I was able to do one drive a day for a week and be done with it. Because the resilver was done in under 6 hours, I might have been able to finish even more quickly if not for sleep and work. I have two vdevs in my pool so the other 6 dives are still 2TB drives. All are healthy and I am satisfied with the amount of storage for now.
The point is, moving up to a larger drive size was pretty painless and I think you might be pretty happy doing that later, but if you are going to change the configuration of your storage, it is best to do that when the volume of data is less than to do it later when you have more data to backup and restore.
Several years ago, I had built a RAIDz3 pool with 12 drives and was not satisfied with the performance of the system. I found read and write speed unacceptably slow. I tried several solutions to fix the speed but the thing I eventually did was to build another system with 12 drives in two RAIDz2 vdevs and I found the speed to be much better. Generally, the more vdevs you have, the better the performance of ZFS. Now, both of my storage systems are configured that way.
In your situation, even if you added a second vdev using 2TB drives, you would get more space to work with and you would get more performance from your hardware. I have a system with 24 drive bays and have considered adding 1TB drives in the 12 empty bays just to improve speed, not because I need the extra storage space.

 

[joeschmuck]

Those 8TB Easystore products were on sale at Best Buy for $127 each. Needless to say they sold out of the specific ones on sale virtually immediately but they are still selling the full priced ones at $200 each. I'm going to need to become more agressive in my holiday sales search.

I have room for a total of 16 drives.
I think right now I'm restrained by my finances, a new 6-disk vdev is a bit much for me .

I've already received 3 new disks, in the meantime one drive in my current array decided to fail on me so I replaced it with one of the new disks.
I'm probably going for a 8-disk RAIDZ2 array now.
I could theoretically in the future go for an extra 8-disk array, but I'll need to save up some money first.

I may have missed this but what are you using the storage for? Movies, Work Product, Photos? Do you need more speed than a single vdev provides? I ask because this is also something that needs to be considered when configuring your storage. In my situation I'm just using FreeNAS mainly as a backup for my computers and then it stores photos and some movies, basically nothing fast. I have tried to use is as an iSCSI device and it works pretty well however the pool is not optimized for that kind of data transfers so it's not as fast as it could be but that is okay with me for home use.

And I hear you when you start talking money and how we are all limited by the cost of hard drives, it would be nice to have the ability to purchase an entire vdev worth of drives and drop them into place, maybe my wife will win the lottery tonight ;). If I had a super fast internet connection then I could just use the cloud and the payment for a service like that is cheaper than buying hard drives right now, but I have a slow internet connection so it's not an option for me. This was somethign I seriously looked into a few years ago.

 

[Bob Paddock]

Apologies if this is not the correct place to ask, this thread is along the lines of my question.

I have a four bay TrueNAS with four 3T WD Red drives.
With the holiday sales I'm thinking of upgrading them to 8T WD Reds.

Anyone have experience with their reliability?
What would be the best way to burn them in before putting them in the TrueNAS and how long is a good burn in time?

 

[Arwen]

...
I have a four bay TrueNAS with four 3T WD Red drives.
With the holiday sales I'm thinking of upgrading them to 8T WD Reds.
...?

If you have a TrueNAS, that implies you have a support contract. FreeNAS is the, well free, version of the software. TrueNAS is the paid version that is probably only sold on iXsystems' hardware. If it really is a TrueNAS and you still have a support contract, then they would answer your question, (both which drives to buy, and how to maintain support).

That said, it's more likely you did not know the difference. (This is also to clarify the difference between TrueNAS and FreeNAS for other people reading the thread.)

Back to the question, sorry, my only 8TB disk is a Seagate Archive SMR. Rarely used, as it's a backup disk.