Advice on expanding 3TB drives to 6TB drives

[Matt Morgan]

Current setup

SUPERMICRO MBD-X10SL7-F-O uATX Server Motherboard LGA 1150 Intel C222 DDR3 1600
32 GB ECC RAM
Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
Vdev1 - 5 WD RED 3TB RaidZ2
Vdev2 - 5 WD RED 3TB RaidZ2

Total disk = 30 TB
Total capacity ~18 TB.

I'm over 80% capacity and have been deleting things to get back down to a safer level. I'm considering replacing the 3TB drives with 6TB drives. I would do this in 2 phases due to cost. When I setup the build originally, I tried to follow the 1 GB for each 1 TB of storage. The build has been rock solid for several years of home usage. It runs a Plex server that isn't heavily accessed.

Two main questions:

1. Is it safe to consider expanding from 3 TB drives to 6 TB drives given my hardware?
2. Is it safe to run Vdev1 with 6 TB drives and Vdev2 with 3 TB drives until I can afford the 2nd upgrade?

 

[Heracles]

Hey Matt,

Yes, it is possible to replace your 3 TB drives with 6 TB in a very safe way. Because you used RaidZ2, your pool will still benefit from redundancy during the re-silvering process.

So all you need to do is to get the new drives first. Once you have them, a burn-in period is always good (do some stress / load testing on the brand new drives to protect yourself against Defective-On-Arrival drives...).

Once you are confident your new drives are ready, you go in the WebUI and switch a first drive offline. Is your material hotplug capable ? If it is, you can pull out that drive and install the new one live. If your hardware is not hotplug, simply power off your FreeNAS before switching the drives.

Once the new drive is installed, go back in the WebUI and activate that drive as a replacement of the previous one. FreeNAS will start re-silvering the pool. Once done and your pool is back to Healthy, you are ready to do it again with the second drive.

Keep doing this, one drive at a time, until all 5 drives in your vDev are done.

Should you wish to replace the drives in both vDev, you can do the 2 vdevs in parallel : turn offline / replace one drive per vdev in each vdev at the same time. But considering your second question, I understand that this is not your plan.

Technically, because you use RaidZ2, it would be possible to do 2 drives at once in a single vDev, but I strongly advise against that. If you do it that way, your pool will ended up without ANY protection during the re-silvering process. Possible but really not smart nor recommended.

As for the amount of RAM, a FreeNAS server can never have too much RAM. Still, the 32G you have for what you described as regular home usage will do the job even after that increase in storage capacity.

 

[Jailer]

You've got 14 available SATA connections on that motherboard. Take advantage of that and connect your new drives and replace them without removing the old ones first. That way you won't lose any redundancy and you can replace a couple at a time and the resilver will take less time with the old drives still in place. Just make sure you burn in your new drives before you put them in to service.

 

[Bozon]

You've got 14 available SATA connections on that motherboard. Take advantage of that and connect your new drives and replace them without removing the old ones first. That way you won't lose any redundancy and you can replace a couple at a time and the resilver will take less time with the old drives still in place. Just make sure you burn in your new drives before you put them in to service.

Could you please detail the steps to do this for my edification? I looked around for a definitive way to do this, but I couldn't find anything. Do you mean create a new pool and move the data to the new pool? Is there a sequence of steps that allows you to mirror drives in a vdev and then remove the original drives after the mirroring process is complete? I seemed to remember something like this when I was playing around with my initial setup.

 

[Jailer]

You connect the new drives to the available SATA ports and follow the manual's procedure on replacing a drive.

https://www.ixsystems.com/documentation/freenas/11.2/storage.html#replacing-disks-to-grow-a-pool

 

[Matt Morgan]

@Heracles and @Jailer thank you so much for the great information. My main concern was the 32 GB of memory being enough for the expansion. Sounds like i should be fine there.

@Jailer I totally forgot that I have 4 more open ports for disks. My case (Node 804) only has room for 10 disks, so I think in my mind I had decided the MB only had 10 slots. Once I get the disks, I'll certainly try to add using some of the free ports to reduce risk of data loss. I've never done this process before, so I'm sure it will be terrifying until it is complete.

 

[drinking12many]

There isnt much difficult about the process. I have done it many times, just identify if you havent already which drives your replacing and do one at a time like stated above. I don't even bother to offline them. I just shut it off (non hot swap) pull the drive I am replacing put the new in and mark it as the replacement for the failed one in the GUI easy peasy. "glabel status" and "camcontrol devlist" can help you identify which device is which. As listed above though if you have extra ports the link Jailer provided is the best way.

 

[giveyouup]

As @Jailer said your motherboard got 14 sata ports. Use them first, if they doesn't fulfill your capacity either then replace those 3TB drives for 6TB

 

[Bozon]

There isnt much difficult about the process. I have done it many times, just identify if you havent already which drives your replacing and do one at a time like stated above. I don't even bother to offline them. I just shut it off (non hot swap) pull the drive I am replacing put the new in and mark it as the replacement for the failed one in the GUI easy peasy. "glabel status" and "camcontrol devlist" can help you identify which device is which. As listed above though if you have extra ports the link Jailer provided is the best way.

@Jailer was talking about doin the replacement before pulling the old drive which is a much better and safer way to do it. The pool is never in a degraded state. The manual even says you can do the replacement as an external if it doesn’t fit in the case and then when done shutdown and do the physical replacement changing the external to an internal drive.

 

[Matt Morgan]

So it sounds like moving from 10 3TB drives to 10 6TB should be fine with current hardware. What about switching to 10 TB drives? Would I be pushing it a bit too far at those storage levels?

Option 1

5 6TB drives (1 vDev) @ ~$200 each => $1,000 and would yield 27 (9 + 18) usable TB of storage in RaidZ2 setup I currently use.
~$37 per TB of usable storage

Option 2

10 6TB drives (both vDevs) @ ~$200 each => $2,000 and would yield 36 (18 + 18) usable TB of storage in RaidZ2 setup I currently use.
~$56 per TB of usable storage

Option 3

5 10TB drives (1 vDev) @ ~$300 each => $1,500 and would yield 39 (9 + 30) usable TB of storage in RaidZ2 setup I currently use.
~$39 per TB of usable storage

I'm split between Option 1 and Option 3. Option 2 seems silly at this point. It took me almost 3 years to fill up my 2nd vDev (9 TB). I'm thinking that might be a fairly consistent metric moving forward unless I switch to 4K as my primary video format (currently 1080P).

This brings me back to my question about whether my hardware would also handle moving to 10TB drives. If I kept the same hardware, I'm guessing the first vDev with 10TB drives would be fine, but what about upgrading the 2nd vDev down the road? Could I run 10 10TB drives in 2 vDevs running RaidZ2 like I currently do?

That would yield ~60TB of storage...which of course is completely stupid.

Still....could it?

 

[Heracles]

Hi again Matt,

A single vDev of 5 drives, 10TB, RaidZ2 is surely an option. From your posts, I think your main concern is about the ratio of 1 Gig RAM per TB of storage.

This is not a hard requirement without which everything will collapse. It is a rule of thumb and nothing more than that.

In general, when a bigger storage is needed, it often implies more activity in the server. It is because of this increased activity that an increase in RAM is important.

As an example, I have a DR server. That one is running only with the lowest RAM acceptable for FreeNAS to be stable. The only thing that server is doing all day long is receiving snapshots sent by ZFS from the main server. No concurrent connection, no share offered from there, no nothing. The server is basically sleeping all day long. For that, no matter how big is the storage to sync, the server is and will stay at very low activity.

At the other end, if you have a FreeNAS server used for hosting iSCSI datastore to a very busy ESXi server, doing deduplication and running some jails of its own, you may reach very quickly the point where more RAM will be required.

So for your home server, 5x 10TB RaidZ2 with 32Gig of RAM sounds a very good option.

 

[drinking12many]

@Jailer was talking about doin the replacement before pulling the old drive which is a much better and safer way to do it. The pool is never in a degraded state. The manual even says you can do the replacement as an external if it doesn’t fit in the case and then when done shutdown and do the physical replacement changing the external to an internal drive.

Oh its definitely the better way if you have the SATA ports. I know most of us probably dont though :(

 

[Matt Morgan]

Update, just completed the upgrade!

Ended up purchasing 5 new 10 TB disks and replacing one at a time using the open ports to avoid putting the system into a degraded state. Took a few days, but it is done and I have a bunch of free space to fill.

Thanks to everyone for the support and advice.

Next up, upgrading from 11.1 U7 to 11.2 U3 and then moving apps from warden jails to iocage jails.

Should be fun.