Accommodating extra drives during migration

[BitCollider]

I’m replacing drives on my TrueNAS machine. The drives are old, one is complaining about a bad sector, and I want more space in my pool anyway. I don’t have enough SATA ports for my old drives, my new drives, and my boot drives, and I’m hoping to get advice.

My motherboard has eight SATA ports. Four are in use for my storage drives, in a single RAID-Z2 pool. Two are in use for mirrored boot drives. I intend to swap out the four old storage drives for four bigger new drives. I thought I would simply create a new pool using the new drives and copy everything over.

But… I’m two ports short! I only need the ports during the migration. Afterwards I’m going to remove the old drives and probably recycle them. I’m pretty sure I should be giving my data drives the eight SATA ports. So what do I do with my boot drives? A few ideas from research.

Buy an HBA. This seems like tremendous overkill. I don’t want to run all eight drives permanently and I’m hoping to avoid the extra complexity of figuring out an HBA.

Mirror my boot drives onto USB sticks then remove the boot drives; boot and run from USB during the pool migration. This seems most straightforward but running from USB sounds risky.

Buy an external enclosure and move one of my boot drives into it. Plug in via USB and boot from the external drive during the pool migration.

Is there a better option? Is plugging in an external enclosure reasonable?

 

[samarium]

If you are to effectively keep the pool, then just replace the pool drives one by one, using the webui replace option, allowing the pool to resilver each drive, keeping individual drives being replaced until that individual has completed replacing, thus maintaining as much redundancy as possible. After all you drives have been replaced the pool will either automatically show the new space hopefully shown by the webui, depending upon the zpool get autoexpand poolName property, or you need to tell zfs to expand to use the space, zfs list poolName; zpool online -e poolName; zfs list poolName. There may also be a webui option to expand the pool if it doesn't autoexpand, which is the current TN default for new pools.

 

[unseen]

I’m replacing drives on my TrueNAS machine. The drives are old, one is complaining about a bad sector, and I want more space in my pool anyway. I don’t have enough SATA ports for my old drives, my new drives, and my boot drives, and I’m hoping to get advice.

My motherboard has eight SATA ports. Four are in use for my storage drives, in a single RAID-Z2 pool. Two are in use for mirrored boot drives. I intend to swap out the four old storage drives for four bigger new drives. I thought I would simply create a new pool using the new drives and copy everything over.

But… I’m two ports short! I only need the ports during the migration. Afterwards I’m going to remove the old drives and probably recycle them. I’m pretty sure I should be giving my data drives the eight SATA ports. So what do I do with my boot drives? A few ideas from research.

Buy an HBA. This seems like tremendous overkill. I don’t want to run all eight drives permanently and I’m hoping to avoid the extra complexity of figuring out an HBA.

Mirror my boot drives onto USB sticks then remove the boot drives; boot and run from USB during the pool migration. This seems most straightforward but running from USB sounds risky.

Buy an external enclosure and move one of my boot drives into it. Plug in via USB and boot from the external drive during the pool migration.

Is there a better option? Is plugging in an external enclosure reasonable?

If all you need is two more SATA 3 ports, you can buy a PCIe card which will give you that for $20 as long as you have a free PCIe slot on your motherboard. Then you can take your boot mirror out and run them off the add-in card while migrating.

 

[BitCollider]

If you are to effectively keep the pool, then just replace the pool drives one by one, using the webui replace option, allowing the pool to resilver each drive, keeping individual drives being replaced until that individual has completed replacing, thus maintaining as much redundancy as possible. After all you drives have been replaced the pool will either automatically show the new space hopefully shown by the webui, depending upon the zpool get autoexpand poolName property, or you need to tell zfs to expand to use the space, zfs list poolName; zpool online -e poolName; zfs list poolName. There may also be a webui option to expand the pool if it doesn't autoexpand, which is the current TN default for new pools.

This is definitely a nice, simple approach. I had gotten the impression that reslivering was hard on drives. I wasn’t sure whether it was a good idea to put my old drives through it. If that isn’t actually a concern, it sounds like one-by-one replacement is the easiest route.

 

[BitCollider]

If all you need is two more SATA 3 ports, you can buy a PCIe card which will give you that for $20 as long as you have a free PCIe slot on your motherboard. Then you can take your boot mirror out and run them off the add-in card while migrating.

There are some scary things written about PCIe SATA expansion cards for TrueNAS, usually suggesting that data will somehow end up corrupted. On the other hand, they’re just boot drives. In the absolute worst case, I guess I will reinstall TrueNAS and restore my config.

 

[Davvo]

I had gotten the impression that reslivering was hard on drives.

It is a stressful process, especially for RAIDZ VDEVs.

There are some scary things written about PCIe SATA expansion cards for TrueNAS, usually suggesting that data will somehow end up corrupted.

Because it will if you are not careful in what you choose.

If you are to effectively keep the pool, then just replace the pool drives one by one, using the webui replace option, allowing the pool to resilver each drive, keeping individual drives being replaced until that individual has completed replacing, thus maintaining as much redundancy as possible.

This is the proper way; the safest option is to temporarely offline one of the boot mirror drives and use that SATA port to plug the first new drive, then replace the old ones: this way you always have the full redundancy of your array.

 

[unseen]

There are some scary things written about PCIe SATA expansion cards for TrueNAS, usually suggesting that data will somehow end up corrupted. On the other hand, they’re just boot drives. In the absolute worst case, I guess I will reinstall TrueNAS and restore my config.

Yes, I wasn't suggesting you put your data disks on an add-in card. As these things are just some SATA interface chips that get connected to the PCIe bus, there's no badly implemented firmware to go wrong and mess up your data. I ran with my boot mirror on just such an add-in card for many years when I ran out of on-board SATA interfaces.

There's no reason to assume that a simple PCIe add-on card will be any less reliable than the integrated SATA interfaces on your motherboard - it will probably use exactly the same PCIe-SATA interface chips.

 

[samarium]

Well ... not sure I agree with you there. Most motherboard SATA ports are either CPU connected (AMD, maybe Intel now, not sure) or Southbrige chipset (Intel, and AMD) SATA support, and neither of these are offered on add-on cards. And I'm sure you are not talking about SAS HBA chips that are embedded on some server motherboards.

Addon cards are usually using SATA chips from ASMedia, JMicron, Marvell and others but not AMD or Intel. Some motherboards graft on some of those ASmedia etc chips for secondary SATA ports, but not many. I agree that they mostly work, and have been getting better, but they are not the same as you stated. Many of the SATA addon cards offer "firmware" RAID, but those can usually be avoided, usually simply on price.

Either way, the OP doesn't need extra ports, single disk replacement just the same as a faulty disk replacement will slowly replace the pool, and there will still be one free SATA port.

 

[Davvo]

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

 

[NugentS]

If all you need is two more SATA 3 ports, you can buy a PCIe card which will give you that for $20 as long as you have a free PCIe slot on your motherboard. Then you can take your boot mirror out and run them off the add-in card while migrating.

Be very careful about what you buy.
Here be Dragons

 

[NugentS]

Actually the correct process here, using a spare port is to insert a new disk and then replace 1 of the existing disks.
ZFS will resilver, whilst maintaining the original parity levels. Once resilvered then ZFS will kick out the replaced disk, which can then be physically removed. The process can then be repeated as many times as required.

Not quick - but reasonably safe

 

[unseen]

For goodness sake! All I did was propose that the original poster used a 2 port SATA add-on card temporarily for his boot mirror and everyone comes along with lots of hand waving objections.

If there are known, documented problem with specific cards for low throughput use like a boot mirror, name them. Hand waving is not useful.

 

[Davvo]

For goodness sake! All I did was propose that the original poster used a 2 port SATA add-on card temporarily for his boot mirror and everyone comes along with lots of hand waving objections.

If there are known, documented problem with specific cards for low throughput use like a boot mirror, name them. Hand waving is not useful.

I linked a resource that specifically explains why that might not be great. Look into it.