Triple mirrored my boot volume; new device not showing gptid?

[Kuyper]

In preparation for upgrading to FreeNAS 11, I am in the process of migrating from a 16GB RAID1 pair of USB boot "drives" to new 64GB USB drives. The plan was to clone my old 9.3 install onto the new drives, remove them, and then upgrade, so that if all goes horribly wrong :) I can just go back to my 16GB drives.


In order to keep it simple, I decided I would first add one of the 64GB drives to the pool, let it re-silver, add the 2nd, resilver, remove the original pair and then grow the root FS (autoexpand should take care of that, right?)

However, after the pool completed its work:

Code:

[kuyper@knox ~]$ zpool status freenas-boot
  pool: freenas-boot
 state: ONLINE
  scan: resilvered 9.70G in 2h3m with 0 errors on Sun Dec 17 12:54:01 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		freenas-boot									ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/43450352-723f-11e4-b1ab-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			gptid/436bc0c9-723f-11e4-b1ab-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			da9p2									   ONLINE	   0	 0	 0

errors: No known data errors

I noticed that the new drive was not showing a gpart ID.

Is that a problem?

I ask, because upon reboot, I nearly thought I was screwed, because the grub splash screen would display, but within one second, my host would reboot! I finally found that, provided I specifically selected the newly added 64GB (Samsung) USB drive (the other 2 were SanDisk, so easy to pick the new one) in my Boot BIOS, FreeNAS (phew!) booted as normal.

Now I'm worried; can I safely add the 2nd 64GB drive (da9), and remove the 2 16GB drives (da7 & da8)? It seems like they're no longer set up properly (I certainly could not boot without the new drive in place!)

Funny thing is, the resilvering definitely created a (GPT) partition table for the new drive (da9) that matches da7 & da8 in all regards, except that da9p2 correctly shows the larger partition:

Code:

[kuyper@knox ~]$ gpart show da7
=>	  34  31266749  da7  GPT  (14G)
		34	  1024	1  bios-boot  (512k)
	  1058		 6	   - free -  (3.0k)
	  1064  31265712	2  freebsd-zfs  (14G)
  31266776		 7	   - free -  (3.5k)

[kuyper@knox ~]$ gpart show da8
=>	  34  31266749  da8  GPT  (14G)
		34	  1024	1  bios-boot  (512k)
	  1058		 6	   - free -  (3.0k)
	  1064  31265712	2  freebsd-zfs  (14G)
  31266776		 7	   - free -  (3.5k)

[kuyper@knox ~]$
[kuyper@knox ~]$
[kuyper@knox ~]$ gpart show da9
=>	   34  125337533  da9  GPT  (59G)
		 34	   1024	1  bios-boot  (512k)
	   1058		  6	   - free -  (3.0k)
	   1064  125336496	2  freebsd-zfs  (59G)
  125337560		  7	   - free -  (3.5k)

Have I skipped a step?

Any thoughts appreciated.

Thanks
Kuyper

 

[joeschmuck]

The easy way to do this is to backup your config file, then remove your old USB drives, now install the new USB drives and perform the installation and select all three devices during the install, lastly restore your config file. You are not the only one who has issues with adding an extra device and this is the fast way to fix it.

 

[Kuyper]

The easy way to do this is to backup your config file, then remove your old USB drives, now install the new USB drives and perform the installation and select all three devices during the install, lastly restore your config file. You are not the only one who has issues with adding an extra device and this is the fast way to fix it.

Thanks Joe - so what you're describing, however, is a full re-installation from boot media, going directly to 11.1 on the new drives, and then just re-importing the full config? I assume all my installed apps (Plex etc) in the various Jails are transitioned in cleanly this way?

What about the fact that my old USB drives are not currently bootable - when I added the 3rd mirror, something happened and it now only boots with that drive in place - I am wary of shooting myself in the foot here, and don't want to overwrite that 64GB drive.

I guess I can start out with an unmirrored, single 64GB new drive; convince myself that all is well, and only then re-use the (1st) 64GB drive as the 2nd mirror.

I don't plan on running a triple mirror, BTW - I was just trying to do that to move (near) seamlessly over to the new drives.

Sigh :)

 

[joeschmuck]

I assume all my installed apps (Plex etc) in the various Jails are transitioned in cleanly this way?

Assuming your jails are installed on your pool like most people do, then yes.

If you have the option to use a single small SSD or even a laptop hard drive, you would be better off than using the USB Flash Drives. Also, you don't need mirrored USB Flash Drives, just keep a copy of your config file backed up and you can easily recreate the system again should all hell break loose. And if you lost your config file, you could pull it from the pool as well.

I am wary of shooting myself in the foot here, and don't want to overwrite that 64GB drive.

That is smart thinking, play it safe and use a different flash drive. But in reality, unless you have an encrypted pool the recovery is pretty simple, even if you do have to recreate a Plex jail. Now some other jails could be a pain to configure and get back to normal.

 

[Kuyper]

Assuming your jails are installed on your pool like most people do, then yes.

If you have the option to use a single small SSD or even a laptop hard drive, you would be better off than using the USB Flash Drives. Also, you don't need mirrored USB Flash Drives, just keep a copy of your config file backed up and you can easily recreate the system again should all hell break loose. And if you lost your config file, you could pull it from the pool as well.


That is smart thinking, play it safe and use a different flash drive. But in reality, unless you have an encrypted pool the recovery is pretty simple, even if you do have to recreate a Plex jail. Now some other jails could be a pain to configure and get back to normal.

Thanks again for the quick responses; I have plenty of spare small SSDs lying around nowadays :) Perhaps that's a better way to go. Pity, I just bought those mini Samsung USB sticks :)


Plex is pretty fickle; I've lost metadata before :( so I try to mess with it as little as possible. Even upgrading Plex is a little dangerous and I have to hold my breath, and sweat, each time :) It take forever to install because I have a deliberately down-rated Xeon (only 2 running cores for power/heat reasons) as I don't ordinarily need a lot of CPU power for what this box does. All my video streams work fine as I don't need to do much transcoding in Plex.

Next up, when my new 8TB drives arrive, I'll be growing my existing (totally full) pools, but will need to bring in 1 or 2 at a time, and retire the old 4TB drives, as I have no more case or SAS/SATA ports. Trying not to jam too much heat into the case...

This project started out 2 or 3 years ago trying to build the most power efficient NAS I could, and I had it all running with 5x4TB drives in around 50W (!!) but over time, added 4xNIC, SAS controller, very fast (Sun rebranded) ZILs (they run hot!) so I guess saving power or keeping it cool is no longer a priority :)

Cheers
Kuyper

 

[joeschmuck]

but will need to bring in 1 or 2 at a time,

1 at a time only!

Yes, use one of your SSDs, you will be much happier knowing your odds of a boot drive failure has just dropped significantly.

 

[Kuyper]

1 at a time only!

Yes, use one of your SSDs, you will be much happier knowing your odds of a boot drive failure has just dropped significantly.

Thanks for your help Joe - fresh install, with a recover from DB backup was, by far, the best way to go. I'm still on a USB thumb drive for now (a single, no mirror, because if it craps out, I can re-install so easily! I had no idea that the "Save/restore" config worked so well) - I noticed a lot of complaints about 9.3 -? 11.1 components being updated, but it all ran very smoothly.

Now that I've caught my breath, and when I get time to look inside my case and see if there's a spare place to squeeze in an SSD, I'll just re-do the exercise and install fresh on the SSD. The restore will be from today's 11.1 backup, so should go faster.

When I bring in the new 8TB spinning rust, I will replace one at a time - per your suggestion (I assume you mean zpool replace?), tho I was actually hoping I could *add* them, maybe 2 at a time if I have enough spare SAS ports, to my existing pools, then remove a 4TB one at a time from the old batch, so that I'm never vulnerable during the resilvering (or does zpool replace protect me from that?). I'm on raidz1-0, so only 1 parity drive:

Code:

  pool: tank01
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 16:43:53 with 0 errors on Sun Dec  3 16:43:56 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		tank01										  ONLINE	   0	 0	 0
		  raidz1-0									  ONLINE	   0	 0	 0
			gptid/07582aa3-7293-11e4-921a-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			gptid/07fdef2c-7293-11e4-921a-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			gptid/089c3601-7293-11e4-921a-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			gptid/09414ac2-7293-11e4-921a-bc5ff4c7d2fc  ONLINE	   0	 0	 0
			gptid/09e9d535-7293-11e4-921a-bc5ff4c7d2fc  ONLINE	   0	 0	 0
		logs
		  da5										   ONLINE	   0	 0	 0
		  da6										   ONLINE	   0	 0	 0

errors: No known data errors

Each of those are identical HGSTs:

Code:

[kuyper@knox ~]$ sudo camcontrol devlist
<ATA HGST HMS5C4040AL A580>		at scbus0 target 2 lun 0 (pass0,da0)
<ATA HGST HMS5C4040AL A580>		at scbus0 target 3 lun 0 (pass1,da1)
<ATA HGST HMS5C4040AL A580>		at scbus0 target 4 lun 0 (pass2,da2)
<ATA HGST HMS5C4040AL A580>		at scbus0 target 5 lun 0 (pass3,da3)
<ATA HGST HMS5C4040AL A580>		at scbus0 target 6 lun 0 (pass4,da4)
<STEC ZeusIOPs G3 E12B>			at scbus0 target 8 lun 0 (pass5,da5)
<STEC ZeusIOPs G3 E12B>			at scbus0 target 9 lun 0 (pass6,da6)
<Samsung Flash Drive FIT 1100>	 at scbus9 target 0 lun 0 (pass7,da7)

soon to be replaced by HGST He8s. I was planning on keeping 2 4TBs around and using them for extra parity. Can I do that with mismatched geometry? ZFS doesn't care, afaik, as long as there's space, right? Hence 2x4 to cover a single 8 failure.

Sorry, my questions are now officially into "scope creep" :)

Cheers
Kuyper

 

[joeschmuck]

I was planning on keeping 2 4TBs around and using them for extra parity. Can I do that with mismatched geometry?

No you can't do that. If you tried then all the drives would pretend they can only hold 4TB of data, not what I'd think you want.

Follow the user guide for how to replace a hard drive, the procedure it solid and works. Only replace one drive at a time to ensure you resilver a new drive as fast as possible. I suspect that if you tried to resilver two drives at a time the it would be a slow process. Also if something goes wrong then it could lead to complications you do not want. As that say K.I.S.S., I'm sure you know what that stands for, if not Google it.

 

[Kuyper]

No you can't do that. If you tried then all the drives would pretend they can only hold 4TB of data, not what I'd think you want.

Meh, in that case, when I add another SAS controller and get a bigger case some time, these guys come out of retirement and get to be in a new pool then :)

As that say K.I.S.S., I'm sure you know what that stands for, if not Google it.

intimately familiar :)

Thanks again; gonna leave you alone now!

Cheers
Kuyper