Boot pool uncorrectable failure?

[khisanthax]

Morning,

I'm getting the dreaded "warning boot pool has encountered an uncorrectable io failure and has been suspended." I have truenas core installed on a VM in proxmox which is on zfs raidz1, I have a 512GByte NVME on PCI passed through to truenas. It seemed to have happened over night when the server was doing backups, may have been a heavy load. What I noticed this morning is that the pci NVME drive is no longer listed in 'Disks' in prox and lspci doesn't show the drive anymore. This is the 3rd time the drive seems to disappear. Would this error be caused by the drive failing and disappearing?

 

[Davvo]

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

This is the 3rd time the drive seems to disappear. Would this error be caused by the drive failing and disappearing?

I don't see your hardware list, but it could be.

 

[khisanthax]

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

I don't see your hardware list, but it could be.

Thanks for the second link and loved the first and sorry. The ZFS pool is RaidZ1 with three ssd drives. Proxmox and it's VM, including truenas core is installed on that. Truenas has a NVME on PCI passed to it by Proxmox.

In proxmox the NVME isn't showing up on the Disks section. This morning when I did lspci it didn't show up but now it's there, or I could have missed it. This my lspci just now. Unless my assumption is wrong that even when the pci is passed through I should still see it in the drives section. I'm wrong arent I?

edit: my apologies for not adding my info in my first post. A small part of me was happy that the drive might be dead, just so that I didn't have to investigate my 5th problem for the week. But I should have had a better attitude about it.

 

[Davvo]

he ZFS pool is RaidZ1 with three ssd drives. Proxmox and it's VM, including truenas core is installed on that.

So, are you telling me your three drives are both yout boot-pool and your data-pool? Partitioning the drives is not supported by TN, you should have a boot drive (with its own boot-pool) and your data drives (with their own data-pool). If things are like this, we might have found the issue.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Truenas has a NVME on PCI passed to it by Proxmox.

What is the exact model? We need you to list all your hardware, as per forum's rules, in order to help you.
In this case, I need to know at least:

• your motherboard model
• your drives model
• that PCIe to NVMe adapter you are using

 

[joeschmuck]

Two things:
1) Have you POWERED OFF the Server? And then Powered On? I am deliberate in my word choice.
2) If step 1 does not fix it, I would stop the TrueNAS VM and examine the VM configuration. You might even clone it to see if that helps and/or remove the NVMe drive from the VM, save it, then add the NVMe drive again.

I'm getting the dreaded "warning boot pool has encountered an uncorrectable io failure and has been suspended."

If you have created a backup of your TrueNAS configuration then you can recreate the boot drive and load the config file, should all return to normal.

Please stop providing links to your screen captures that are not on the TrueNAS forums, no one here likes them and are hesitant to click on the links, trojans and bads things all over the place. You can attach images directly to the forum and it is prefered. Thanks.

 

[khisanthax]

So, are you telling me your three drives are both yout boot-pool and your data-pool? Partitioning the drives is not supported by TN, you should have a boot drive (with its own boot-pool) and your data drives (with their own data-pool). If things are like this, we might have found the issue.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

What is the exact model? We need you to list all your hardware, as per forum's rules, in order to help you.
In this case, I need to know at least:

• your motherboard model
• your drives model
• that PCIe to NVMe adapter you are using

I don't believe that the three drives, which are one zfs pool, are both the boot-pool and the data-pool. When I installed proxmox it gave me the option to create a zfs pool, which I did, with the three SSD drives. After truenas core was installed as a VM within proxmox I passed the NVME on the pcie card to it and only used the NVME for data storage. Is that not a separate boot-pool and data-pool for Truenas Core? Proxmox itself has the boot-pool and data-pool on one pool, would that be a problem?

OS: Proxmox 7.4
PCIe to NVME adapter: MHQJRH (no name brand)
NVME: WD Black SN850
Drives: 2x TEAMGROUP AX2 1TB, 1x TEAMGROUP EX2 1TB
Motherboard: Dell PowerEdge C4Y3R (Dell R720xd server)
CPU: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz 2x12
Ram: 196GB ECC

Previously, I did power off the server and turn it back on, and it seemed to work. At that time I saw the NVM in the proxmox list of disks, attached it to the Truenas core VM, started the VM and it worked. But then this morning, I saw the error I mentioned earlier, the NVME was not in the list of disks on proxmox and I assumed the nvme failed but it might not be in the list of disks because of the pass through. When I power on the VM after re-passing through the NVME it fails to start, this is also what occured prior to powering off and on the server.

I just started using truenas so I don't have any configuration files backed up or that I need. I only enabled NFS and mapall to root so I can definitely do a fresh install. Being that truenas core is a VM, it's boot drive is a virtual drive. Would that be a problem?

No more external links.

 

[joeschmuck]

Proxmox itself has the boot-pool and data-pool on one pool, would that be a problem?

I don't think that is possible. The boot-pool is seperate.

When I installed proxmox it gave me the option to create a zfs pool, which I did, with the three SSD drives.

I don't know what this is, but it sounds very wrong. If you have no data on these drives, I would destroy everything and start all over. Make a virtual drive of 10GB and add that to the VM, this will be your boot pool. You do not need to pass through any other drives at this point, and then install TrueNAS to the only drive in the VM. Once that is done you can setup some basic setting in TrueNAS, and then power off, pass through the three 1TB drives, power on, and use the TrueNAS GUI to create a pool with the three drives.

Since I've never used Proxmox, I can't really help you any more.

 

[khisanthax]

I don't think that is possible. The boot-pool is seperate.

I don't know what this is, but it sounds very wrong. If you have no data on these drives, I would destroy everything and start all over. Make a virtual drive of 10GB and add that to the VM, this will be your boot pool. You do not need to pass through any other drives at this point, and then install TrueNAS to the only drive in the VM. Once that is done you can setup some basic setting in TrueNAS, and then power off, pass through the three 1TB drives, power on, and use the TrueNAS GUI to create a pool with the three drives.

Since I've never used Proxmox, I can't really help you any more.

I appreciate the advice and insight. The three drives are what proxmox are installed on, if I passed that through proxmox wouldn't see the drives it's on. So, yes, I have truenas already installed on a virtual drive, so I'll continue with the rest of your advice, thank you.

 

[Davvo]

I find the process of creating a zfs pool before installing TN to be worring.

 

[khisanthax]

I find the process of creating a zfs pool before installing TN to be worring.

With my limited knowledge I can see how that would be worrying. With truenas virtualized under proxmox the question becomes how to install proxmox such that it's own file system doesn't interfere with the virtualized truenas.

Right now the setup is messy. Once I get the rear bay drives working on a separate HBA controller and install proxmox on that I hope to clean things up a bit. Proxmox and truenas on the boot drives, the rest of the VMs on the SSD zfs pool and a separate zfs pool consisting of the sas drives. The main HBA controller will be passed to truenas.

I need to learn more.

 

[joeschmuck]

With my limited knowledge I can see how that would be worrying. With truenas virtualized under proxmox the question becomes how to install proxmox such that it's own file system doesn't interfere with the virtualized truenas.

Right now the setup is messy. Once I get the rear bay drives working on a separate HBA controller and install proxmox on that I hope to clean things up a bit. Proxmox and truenas on the boot drives, the rest of the VMs on the SSD zfs pool and a separate zfs pool consisting of the sas drives. The main HBA controller will be passed to truenas.

I need to learn more.

Basically you must setup your Proxmox software to boot from a dedicated drive just for Proxmox. Next you need to physically add the hard drives you want to be your storage for TrueNAS. Now you create a virtual machine for TrueNAS, you would create a virtual boot drive (~8GB), RAM (10+GB, more is better and since you have it I would say 16GB unless you plan to add VM's into TrueNAS, then add more), add a NIC, and pass through the physical storage drives (actually the drive controller), and add a virtual CD-ROM drive to mount the TrueNAS ISO to. If this were a new system then the storage drives should be blank/empty. Now you bootstrap the TrueNAS ISO and install TrueNAS on to the virtual 8GB drive. Once all done you end up rebooting TrueNAS and then you can log in to the GUI. Next you create a pool, you should see the drives you passed through and you can now create the pool.

What this type of setup affords you is that you can take your TrueNAS hard drives to any other computer and bootstrap TrueNAS and have access to your data. This is also the only safe way to implement TrueNAS, yo must pass the drives through and it's by far the best option to pass the entire drive controller through. This means that your Proxmox boot drive cannot be on the same controller but it's able to be done.

To learn more, go to a Proxmox site and figure out how to create a VM, how to pass through the hard drive controller, and play around with it. Take your time, you will learn it.

 

[Davvo]

Additionally if you really want to futureproof you can create a bigger OS drive (up to 25GB?) for the remote case you will need more space.

 

[danb35]

For the sake of general knowledge, Proxmox doesn't have the boot pool/data pool distinction TrueNAS does--if you install it using ZFS (which is one of several filesystem options which also include btrfs), it will create a pool of the type you specify (stripe (which it calls RAID0), mirror (which it calls RAID1), or RAIDZn), on whichever drives you specify, install the OS there, and allow use of the remaining space to store VM images, ISOs, templates, etc. You can create additional pools too, but it's a perfectly valid--indeed, common--configuration under Proxmox to have VM images on the boot pool.

Even with TrueNAS as a guest OS, I wouldn't see a problem using a virtual disk for the boot pool. But you'd definitely want to pass through the HBA to that guest VM for your data drives.

 

[khisanthax]

Thank you everyone for your replies and advice. I bought the rear flex bay kit for the r720xd, but it was missing a cable. Once that comes in I'll have proxmox and truenas on its own boot drives as I already have a second HBA controller so that proxmox can pass primary HBA controller with the drives to truenas. The os boot drives will be 2 1.2tb sas drives as a mirror, so 25GB for truenas is more than doable.

One question, you mentioned that 16GB of ram for truenas is enough. Was I wrong in understanding that truenas using zfs could use more ram than that? Or is that based on my current storage space?