Wierd problem-add SSD - cannot boot usb flash drive

[philhu]

I have a supermicro system, with 36 disk connected to a pci/express card (IT mode jbod). The motherboard also has 6 sata connectors not used.

I am adding 3 ssd drives, 1 for zil (16gb), and 2 for l2arc (2x160g). So I add these to 3 of the 6 internal connectors. Boot, system sees the SSD Disks during post and says 'the bios will be loaded'.

Booted, get message 'This is a NAS data disk - cannot boot'

So, thats fine, forgot to uncheck the HD boot, so do so
The make usb first boot, then get error that 'cannot find boot, hit any key to retry'.

So, nothing I do seems to make it boot as long as the SSD units are connected to the internal sata controller. If I unplug them, the usb boots fine, system comes up and sees the 36 disk slots on the IT mode pci/express card... But if anything is connected to the 6 internal (unused) sata ports, it doesn't seem to boot the usb stick.

Am I missing something wierd here? I set the bootstrap to just boot usb, period. Same problem. Acting like the bios loaded by internal sata port chipset is interfering with usb boot.

Any ideas?

 

[Mirfster]

Wipe the SSDs, seems like they may have some existing configurations on them.

 

[Spearfoot]

I know it sounds crazy... but sometimes it seems like BIOS settings just don't 'take' and have to be re-entered. So you might want to try double-checking them.

Can you post a screenshot of your BIOS boot settings? Like this:

 

[Robert Smith]

It looks like you ehter need to tweak the bootloader (the one on the USB) settings, or reinstall FreeNAS with SSDs attached.

 

[philhu]

I will send the bios screen later today

Robert, it looks like you are right. Is there instructions to 'tweak the bootloader'?

Or..I assume I can reinstall Freenas to new USB stick, while ssd drives attached. So save config(s)?, do reinstall, reload configs? Nothing lost right?

Thanks all for the help!

 

[Robert Smith]

Yes, save the config [you should be saving it routinely, as the usb stick can die at any time]. Reinstall, import the pool, restore config.

Nothing should be lost, but making a fresh backup before any kind of manipulation is a good idea.

Tweaking the bootloader, on the other hand, would require finding its config file, figuring out what is wrong, and fixing it. LOL. Sorry I could not be more specific.

 

[philhu]

ok, I will reinstall.

The one question, after it is back up, i can restore config, but how do I import the pool, or if I restore config first, and reboot, won't the pool come back automagically?

OK, I see it, storage, import volume

But again, if i reload config, and reboot, it will just import it and all the jails, won't it? See my volume list below

 

[Robert Smith]

Logically, you import the volume first, and then restore the config; otherwise I am not sure if FreeNAS has safeguards in place to make certain everything happens in the right order and nothing breaks due to missing mounting points.

You may want to actually investigate why FreeNAS bootloader favors SSD data disks over the correct USB boot drives; it would help other users in similar situations. It is my guess that reinstall will fix it, but we do not know for sure as a few times this problem was reported we never found out why it was happening in the first place.

I am not sure if your version of FreeNAS uses Grub or another bootloader. If it is Grub, start looking by in the /boot/grub/grub.cfg and see where the boot entries point.

 

[philhu]

well, it isn't working here. If I have these ssd drives plugged in and the bios sees them, then i cannot, under any circumstances, boot the usb flash drive

I just spent an hour, I cannot even boot the installer flash if the internal drives are plugged in

I am on Freenas 9.3.10

 

[Robert Smith]

I think this is what happens. Let us say your USB drive is da0. And the bootloader is configured to boot same partition on da0.

Now, you plug in an SSD. The SSD becomes da0, and the USB is pushed down the list and becomes da2, or something.. It does not matter what your BIOS sees. The boot loader is still configured to boot from da0. See what I mean?

 

[Robert Smith]

I cannot even boot the installer flash if the internal drives are plugged in.

The SSDs are wiped clean, correct? Try disconnecting all other drives, to make sure it is not trying to boot off those, and see what error message you get exactly.

 

[philhu]

SSD are wiped clean, they are new

The error is 'Cannot boot media, hit return to try rebooting' or something to that effect

I understand this. It shouldn't affect the boot/installer should it, as it would have to accept a generic boot, I would assume

well, how can that be fixed? I looked at the grug.cfg. Nothing on it points to any specific device

Here is the grub.cfg:
[philhu@freenas] /boot/grub# cat grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /usr/local/etc/grub.d and settings from /usr/local/etc/default/grub
#

### BEGIN /usr/local/etc/grub.d/00_header ###
insmod part_gpt
if [ -s $prefix/grubenv ]; then
load_env
fi
set default="0"

if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi

export menuentry_id_option

if [ x"${gelipassphrase}" != x ]; then
set pass="$gelipassphrase"
export pass
fi

if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi

function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}

function load_video {
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
}

terminal_output console
set timeout=5
### END /usr/local/etc/grub.d/00_header ###

### BEGIN /usr/local/etc/grub.d/10_kfreebsd ###
### END /usr/local/etc/grub.d/10_kfreebsd ###

### BEGIN /usr/local/etc/grub.d/10_ktrueos ###

submenu "FreeNAS (9.10-STABLE-201605240427) - 2016-06-01 18:55" {
menuentry "Normal Bootup" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/9.10-STABLE-201605240427/@/boot/kernel/kernel
kfreebsd_loadenv /ROOT/9.10-STABLE-201605240427@/boot/device.hints
kfreebsd_module /ROOT/9.10-STABLE-201605240427/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/9.10-STABLE-201605240427
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ng_ether.ko
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id="0"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic="1"
set kFreeBSD.debug.ddb.textdump.pending="1"
set kFreeBSD.hw.hptrr.attach_generic="0"
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role="2"
set kFreeBSD.hint.isp.1.role="2"
set kFreeBSD.hint.isp.2.role="2"
set kFreeBSD.hint.isp.3.role="2"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode="2"
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait="1"
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Single User Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/9.10-STABLE-201605240427/@/boot/kernel/kernel -s
kfreebsd_loadenv /ROOT/9.10-STABLE-201605240427@/boot/device.hints
kfreebsd_module /ROOT/9.10-STABLE-201605240427/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/9.10-STABLE-201605240427
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ng_ether.ko
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id="0"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic="1"
set kFreeBSD.debug.ddb.textdump.pending="1"
set kFreeBSD.hw.hptrr.attach_generic="0"
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role="2"
set kFreeBSD.hint.isp.1.role="2"
set kFreeBSD.hint.isp.2.role="2"
set kFreeBSD.hint.isp.3.role="2"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode="2"
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait="1"
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Verbose Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/9.10-STABLE-201605240427/@/boot/kernel/kernel -v
kfreebsd_loadenv /ROOT/9.10-STABLE-201605240427@/boot/device.hints
kfreebsd_module /ROOT/9.10-STABLE-201605240427/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/9.10-STABLE-201605240427
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ng_ether.ko
kfreebsd_module_elf /ROOT/9.10-STABLE-201605240427/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id="0"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic="1"
set kFreeBSD.debug.ddb.textdump.pending="1"
set kFreeBSD.hw.hptrr.attach_generic="0"
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role="2"
set kFreeBSD.hint.isp.1.role="2"
set kFreeBSD.hint.isp.2.role="2"
set kFreeBSD.hint.isp.3.role="2"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode="2"
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait="1"
set kFreeBSD.grub.platform="$grub_platform"
}
}

set default="FreeNAS (9.10-STABLE-201605240427) - 2016-06-01 18:55"
### END /usr/local/etc/grub.d/10_ktrueos ###

### BEGIN /usr/local/etc/grub.d/30_os-prober ###
### END /usr/local/etc/grub.d/30_os-prober ###

### BEGIN /usr/local/etc/grub.d/40_custom.dist ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /usr/local/etc/grub.d/40_custom.dist ###

### BEGIN /usr/local/etc/grub.d/41_custom ###
if [ -f ${config_directory}/custom.cfg ]; then
source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
### END /usr/local/etc/grub.d/41_custom ###

### BEGIN /usr/local/etc/grub.d/50_otherbe ###
submenu "Boot Environment Menu" {

submenu "FreeNAS (Initial-Install) - 2016-05-19 05:20" {
menuentry "Normal Bootup" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/Initial-Install/@/boot/kernel/kernel
kfreebsd_loadenv /ROOT/Initial-Install@/boot/device.hints
kfreebsd_module /ROOT/Initial-Install/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/Initial-Install
kfreebsd_module_elf /ROOT/Initial-Install/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Single User Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/Initial-Install/@/boot/kernel/kernel -s
kfreebsd_loadenv /ROOT/Initial-Install@/boot/device.hints
kfreebsd_module /ROOT/Initial-Install/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/Initial-Install
kfreebsd_module_elf /ROOT/Initial-Install/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Verbose Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/Initial-Install/@/boot/kernel/kernel -v
kfreebsd_loadenv /ROOT/Initial-Install@/boot/device.hints
kfreebsd_module /ROOT/Initial-Install/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/Initial-Install
kfreebsd_module_elf /ROOT/Initial-Install/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
}


submenu "FreeNAS (default) - 2016-05-19 05:03" {
menuentry "Normal Bootup" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/default/@/boot/kernel/kernel
kfreebsd_loadenv /ROOT/default@/boot/device.hints
kfreebsd_module /ROOT/default/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/default
kfreebsd_module_elf /ROOT/default/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id=0
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Single User Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/default/@/boot/kernel/kernel -s
kfreebsd_loadenv /ROOT/default@/boot/device.hints
kfreebsd_module /ROOT/default/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/default
kfreebsd_module_elf /ROOT/default/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id=0
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
menuentry "Verbose Mode" {
insmod zfs
search --no-floppy -s -l freenas-boot
kfreebsd /ROOT/default/@/boot/kernel/kernel -v
kfreebsd_loadenv /ROOT/default@/boot/device.hints
kfreebsd_module /ROOT/default/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/default
kfreebsd_module_elf /ROOT/default/@/boot/kernel/ispfw.ko
set kFreeBSD.bootfile="kernel"
set kFreeBSD.kernel="kernel"
set kFreeBSD.kernel_options=""
set kFreeBSD.kernelname="/boot/kernel/kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules"
set kFreeBSD.kernel="kernel"
set kFreeBSD.module_path="/boot/kernel;/boot/modules;/usr/local/modules"
set kFreeBSD.kern.cam.ctl.ha_id=0
set kFreeBSD.autoboot_delay="2"
set kFreeBSD.loader_logo="freenas"
set kFreeBSD.loader_menu_title="Welcome to FreeNAS"
set kFreeBSD.loader_brand="freenas-brand"
set kFreeBSD.loader_version=" "
set kFreeBSD.kern.cam.boot_delay="30000"
set kFreeBSD.debug.debugger_on_panic=1
set kFreeBSD.debug.ddb.textdump.pending=1
set kFreeBSD.hw.hptrr.attach_generic=0
set kFreeBSD.vfs.mountroot.timeout="30"
set kFreeBSD.hint.isp.0.role=2
set kFreeBSD.hint.isp.1.role=2
set kFreeBSD.hint.isp.2.role=2
set kFreeBSD.hint.isp.3.role=2
set kFreeBSD.net.inet6.ip6.auto_linklocal="0"
set kFreeBSD.vfs.zfs.vol.mode=2
set kFreeBSD.kern.geom.label.disk_ident.enable="0"
set kFreeBSD.hint.ahciem.0.disabled="1"
set kFreeBSD.hint.ahciem.1.disabled="1"
set kFreeBSD.kern.msgbufsize="524288"
set kFreeBSD.hw.usb.no_shutdown_wait=1
set kFreeBSD.grub.platform="$grub_platform"
}
}

}

### END /usr/local/etc/grub.d/50_otherbe ###

 

[Mirfster]

SSD are wiped clean, they are new

Booted, get message 'This is a NAS data disk - cannot boot'

If you are getting that message with the SSDs attached, then they have some NAS/Raid/Etc. configuration on them. Wipe them yourself to be sure.

P.S. Code tags hint; helps avoid walls of text...

 

[Robert Smith]

See where it says...

Code:

search --no-floppy -s -l freenas-boot

That means, Grub is searching for a device labeled "freenas-boot" and sets it as 'root'.

So your next step may be to enumerate all filesystem labels, and see if the expected one is "freenas-boot" and no wrong ones are named as such.

 

[philhu]

If you are getting that message with the SSDs attached, then they have some NAS/Raid/Etc. configuration on them. Wipe them yourself to be sure.

P.S. Code tags hint; helps avoid walls of text...

No, that was the first error when nothing was attached but the zfs data disks

 

[philhu]

See where it says...

Code:

search --no-floppy -s -l freenas-boot

That means, Grub is searching for a device labeled "freenas-boot" and sets it as 'root'.

So your next step may be to enumerate all filesystem labels, and see if the expected one is "freenas-boot" and no wrong ones are named as such.

that is the label of the 2 boot usb sticks, it was a mirroered install..

The ssd are new, never written, 3 of them
There are 11 data disks in a zfs raidz3, there is NOTHING ELSE

So it is looking at all devices, it should see that label, and boot it, most likely it isn't even reading the stick at all, because the bios is saying it is unable to boot device, return to retry.

 

[Robert Smith]

Yes, you can try disconnecting all drives, including the USB stick, to learn what message you get when BIOS is unable to boot anything.

The first one about "NAS data disk - cannot boot" is definitely not from BIOS.

 

[philhu]

Yes that one was from only data disks in the system.

This is with nothing installed or when the usb is installed and an ssd is on the internal controller

The exact error message on boot is:
Reboot and Select proper boot device
or Insert Boot media in selected boot device and press a key

I did notice that when i plug in one new ssd only, that part of the bios load an AHCI bios

I do not see how adding this deivce stops it from seeing the usb to boot.

During post it does say it found 2 usb mass storage devices

 

[Robert Smith]

I think you got it. The motherboard is unable to boot from USB in AHCI mode. You can probably force IDE mode; but that would not be nice for SSDs.

See if there is a firmware update available for the motherboard.

 

[BigDave]

At this point I would:

1. post model of motherboard and let's see if the manual can help with some boot settings
2. Board may be confused with mirrored boot device showing up as two devices, try booting from single device
3. As a last resort, try taking bios back to a default configuration, and then setting boot config again

EDIT:

1. You did say you had four SATA ports on mobo, so make new install to fourth SSD and boot from that...