grub-install failures with new/replaced drives?

[drinny]

I recently upgraded from the latest stable release of FreeNAS (9.2.1.9) to the 9.3 beta (currently FreeNAS-9.3-BETA-201412050249). The upgrade went smoothly with no real issues to speak of. My boot drive (I'll call that disk "A") was converted from UFS to ZFS in the process, so I decided to take advantage of the ability to mirror the boot drive.

I added an additional USB flash drive (disk "B" - another 4 GB to match the 4 GB drive that was there) and things seemed to go OK. I was able to see the boot drive as a mirror when using 'zpool status', so I figured all was well.

Today, I decided to see if I could upgrade those 4 GB USB drives to 8 GB ones instead. I removed disk "A" and replaced it with the new 8 GB (disk "C"). I went through the GUI to replace the drive and seemingly it worked fine. The mirrored freenas-boot filesystem resilvered and everything came back intact. I decided to reboot the system to make sure things were still OK, and this is when I ran into issues.

Rebooting the system, I realized it wouldn't post. I tried changing the BIOS to boot from each of the USB sticks (drives "B" or "C") to no avail. I removed both disks and put drive "A" back into the system and the machine booted up fine.

Combing through the logs, it looks like the part of FreeNAS that handles writing out the grub information to the new drives was having a problem all along. The initial creation of the mirror to drive "B":

Dec 5 09:51:29 freenas notifier: 1+0 records in
Dec 5 09:51:29 freenas notifier: 1+0 records out
Dec 5 09:51:29 freenas notifier: 1048576 bytes transferred in 0.100428 secs (10441086 bytes/sec)
Dec 5 09:51:29 freenas notifier: dd: /dev/da8: end of device
Dec 5 09:51:29 freenas notifier: 5+0 records in
Dec 5 09:51:29 freenas notifier: 4+0 records out
Dec 5 09:51:29 freenas notifier: 4194304 bytes transferred in 0.573074 secs (7318956 bytes/sec)
Dec 5 09:52:19 freenas notifier: Installing for i386-pc platform.
Dec 5 09:54:25 freenas notifier: /usr/local/sbin/grub-install: error: cannot find a GRUB drive for da8. Check your device.map.

So grub never got installed on drive "B" when I added it to the system and would explain why that wouldn't boot from that drive.

Then the logs from when I tried to replace drive "A" with drive "C" today:

Dec 7 21:07:16 freenas notifier: 1+0 records in
Dec 7 21:07:16 freenas notifier: 1+0 records out
Dec 7 21:07:16 freenas notifier: 1048576 bytes transferred in 0.414338 secs (2530725 bytes/sec)
Dec 7 21:07:18 freenas notifier: dd: /dev/da10: short write on character device
Dec 7 21:07:18 freenas notifier: dd: /dev/da10: end of device
Dec 7 21:07:18 freenas notifier: 5+0 records in
Dec 7 21:07:18 freenas notifier: 4+1 records out
Dec 7 21:07:18 freenas notifier: 4718592 bytes transferred in 1.791809 secs (2633423 bytes/sec)
Dec 7 21:07:27 freenas notifier: Installing for i386-pc platform.
Dec 7 21:08:08 freenas notifier: /usr/local/sbin/grub-install: error: unknown filesystem.

So, it appears that installing grub to both a new mirrored drive as well as a replacement drive within the mirror bombs (although I'm not sure if the latter has anything to do with the fact that I was trying to resilver a new drive off of one that never had a working grub installation in the first place)?

Before I file a bug report, I just wanted to make sure that I did things correctly and that nobody sees anything immediately out of the ordinary?

 

[jlpellet]

From my prior questions during 9.3 beta testing, I undersatnd that 9.3 does not support converting an existing installation from a single to a mirror boot but that can only be done during a fresh install. I think this is because of the GRUB issue you raise. As I understand it, this is not a bug but rather a function that is not supported in 9.3. I decided not to try a boot mirror install myself until the relase version is available. Hope this helps.

 

[drinny]

There is an option within version of the beta that I'm using (FreeNAS-9.3-BETA-201412050249) to "attach" a drive to the boot volume from within the GUI. I would have figured that if it weren't possible to convert a single drive to a mirrored pair, that there wouldn't be any use of having that available as an option?

 

[sef]

From my prior questions during 9.3 beta testing, I undersatnd that 9.3 does not support converting an existing installation from a single to a mirror boot but that can only be done during a fresh install. I think this is because of the GRUB issue you raise. As I understand it, this is not a bug but rather a function that is not supported in 9.3. I decided not to try a boot mirror install myself until the relase version is available. Hope this helps.

It does support it. Ticket #6993 was filed about this.

 

[sef]

I just tried it (in a VM, admittedly); the first time seems to have failed. I detached it, and did it again, and it worked. I then told VMWare to use the new drive for booting, and it seemed to work.

So that seems to be a workaround for now.

 

[drinny]

It looks you may be running a different version which fixed the bug in #6955? It appears that adding a new drive to an existing stripe may be fixed in that version?

It appears that bug was solved by adding the full device path to the attached drive (which would explain why it succeeded when you detached the drive and attached it as new, vs replacing the drive)? I followed the same logic with the replacement. After the replacement drive resilvered, I ran grub-install on the full device path instead of just the device name that seems to have worked. I was able to boot off of the drive once grub-install completed successfully.

Dec 8 07:01:53 freenas notifier: 1+0 records in
Dec 8 07:01:53 freenas notifier: 1+0 records out
Dec 8 07:01:53 freenas notifier: 1048576 bytes transferred in 0.418211 secs (2507289 bytes/sec)
Dec 8 07:01:55 freenas notifier: dd: /dev/da10: short write on character device
Dec 8 07:01:55 freenas notifier: dd: /dev/da10: end of device
Dec 8 07:01:55 freenas notifier: 5+0 records in
Dec 8 07:01:55 freenas notifier: 4+1 records out
Dec 8 07:01:55 freenas notifier: 4718592 bytes transferred in 1.830900 secs (2577198 bytes/sec)
Dec 8 07:02:05 freenas notifier: Installing for i386-pc platform.
Dec 8 07:02:50 freenas notifier: /usr/local/sbin/grub-install: error: cannot find a GRUB drive for da10. Check your device.map.

freenas# /usr/local/sbin/grub-install /dev/da10
Installing for i386-pc platform.
Installation finished. No error reported.