Hyper-V Disk not showing up in pool, expected behavior?

[tony95]

So I just set up a 8x8GB z2 Pool and it is fast. I was concerned that it wasn't actually z2 because it is very fast with writes. I am copying 3 other 8GB disks to it at near full speed. So I removed two disks from Hyper-V and the pool went offline. When I looked at the pool I was only seeing 5 disks but I only remove 2 of 8 so I should have a working zpool of 6 disks. On my Hyper-V SCSCI controller I saw the missing disk was #8 and the removed disks were #6 and #7. So I moved disk #8 to to #6 on my SCSI controller and the disk came back online and so did my zpool. Is this expected behavior? Does TrueNAS stop looking for disks when it finds one SCSCI port not in use? I am guessing this is something that should be resolved in a future update, correct?

 

[sretalla]

So I removed two disks from Hyper-V

I'm not sure how this relates to your situation... are you running TrueNAS on top of Hvper-V? If that's true, how are you passing the disks through? and have you taken this into account? : https://www.truenas.com/community/t...ide-to-not-completely-losing-your-data.12714/

When I looked at the pool I was only seeing 5 disks but I only remove 2 of 8 so I should have a working zpool of 6 disks.

Could you share zpool status -v and zpool list -v to help us see what you're talking about?

 

[tony95]

I'm not sure how this relates to your situation... are you running TrueNAS on top of Hvper-V? If that's true, how are you passing the disks through? and have you taken this into account? : https://www.truenas.com/community/t...ide-to-not-completely-losing-your-data.12714/

Could you share zpool status -v and zpool list -v to help us see what you're talking about?

Yes, the disks are attached to Hyper-V as physical disks. Hyper-V has a SCSI controller and you can add physical disks to it, I am not using virtual disks. It might be Hyper-V dropping the disks, I am not sure. I would need to test in another OS to find out if it is the same behavior. Everything is in the zpool now so no point in showing it, just that the #8 controller disk was missing when I removed #6 and #7 and came back when I moved the #8 disk up in the controller list.

 

[sretalla]

Yes, the disks are attached to Hyper-V as physical disks. Hyper-V has a SCSI controller and you can add physical disks to it, I am not using virtual disks.

Just be aware that it's not the same as adding the disk controller to TrueNAS and would be considered as not production safe by this forum. Strange/unexpected/bad behavior is therefore no surprise to us.

In order to virtualize in a way that would be considered safe, you need to pass an entire physical controller into the VM and use only disks attached to it (except for the OS disk, which could be virtual if needed).

 

[Chris Moore]

Is this expected behavior? Does TrueNAS stop looking for disks when it finds one SCSCI port not in use?

You are using a virtualization layer. It is NOT how TrueNAS behaves on real hardware. I have two systems at work where I am going through replacing old drives with new drives (physical servers) and pulling drives from a pool does not take the pool offline. If you are using swap, it can cause some unexpected behavior, but that is another question.

 

[Chris Moore]

Yes, the disks are attached to Hyper-V as physical disks. Hyper-V has a SCSI controller and you can add physical disks to it, I am not using virtual disks. It might be Hyper-V dropping the disks, I am not sure. I would need to test in another OS to find out if it is the same behavior. Everything is in the zpool now so no point in showing it, just that the #8 controller disk was missing when I removed #6 and #7 and came back when I moved the #8 disk up in the controller list.

This is a dangerous method of configuring FreeNAS / TrueNAS and you stand a good chance of loosing any data you put in the ZFS pool by building it this way.

Is this for testing or something you plan to use operationally?

If you can give us some information to go on, we can make some suggestions.

 

[tony95]

You are using a virtualization layer. It is NOT how TrueNAS behaves on real hardware. I have two systems at work where I am going through replacing old drives with new drives (physical servers) and pulling drives from a pool does not take the pool offline. If you are using swap, it can cause some unexpected behavior, but that is another question.

Yeah, I need to look at Hyper-V, it maybe be happening there and not in TrueNAS.

 

[tony95]

This is a dangerous method of configuring FreeNAS / TrueNAS and you stand a good chance of loosing any data you put in the ZFS pool by building it this way.

Is this for testing or something you plan to use operationally?

If you can give us some information to go on, we can make some suggestions.

Why is this dangerous? TrueNAS should have almost native access to the drives.

 

[sretalla]

almost native access to the drives

There's the danger... anything other than direct access is no good.

 

[Patrick M. Hausen]

almost native access to the drives

See?

 

[tony95]

Just be aware that it's not the same as adding the disk controller to TrueNAS and would be considered as not production safe by this forum. Strange/unexpected/bad behavior is therefore no surprise to us.

In order to virtualize in a way that would be considered safe, you need to pass an entire physical controller into the VM and use only disks attached to it (except for the OS disk, which could be virtual if needed).

Yes, my OS and a LOG drive are virtualized, vhdx.

 

[tony95]

See?

No, I don't

 

[Patrick M. Hausen]

No, I don't

The problem is with "almost". That's why I marked that word bold and red. Anything but direct native disk access is a road to desaster.

 

[tony95]

The problem is with "almost". That's why I marked that word bold and red. Anything but direct native disk access is a road to desaster.

Even a controller would be almost as well, you have to attach something to the VM, you can't get rid of that layer without using an entire PC which I am not willing to do.

 

[Patrick M. Hausen]

An entire controller passed-through via PCIe gives you direct access to the controller and the drives which is what is needed if your virtualise.

 

[tony95]

An entire controller passed-through via PCIe gives you direct access to the controller and the drives which is what is needed if your virtualise.

Then passing the drive should give me direct access to the drive, correct?

 

[Chris Moore]

The basic principle is, FreeNAS / TrueNAS (lets just say 'the OS') needs direct access to the disk at the controller level, so the general guidance is to pass the entire controller through to the VM. Here are the recommendations for ESXi, which should be taken as true for any virtualization layer.

Some useful guidance to review:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Virtually FreeNAS ... an alternative for those seeking virtualization

[---- 2014/12/24: Note, there is another post describing how to create a full-sized FreeNAS VM directly managing a bunch of disks. THIS post is aimed at people wanting to use FreeNAS to provide a little basic file sharing off of virtual disks on shared VM storage. ----] There's been a lot of...

www.truenas.com

FreeNAS 9.10 on VMware ESXi 6.0 Guide
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]

The goal of this build was firstly to provide a strong 6-bay NAS, Plex server and offsite backup for my Primary NAS. Also, it needs to run ESXi in a performant manner, and will host Windows, OSX, Linux guests, as well as a pfSense gateway. The ESXi installation was the pilot project for an ESXi...

www.truenas.com

 

[Chris Moore]

Then passing the drive should give me direct access to the drive, correct?

No, and that is particularly true when talking about removing or replacing a drive and possibly pulling SMART data from the drive for health monitoring.

 

[tony95]

No, and that is particularly true when talking about removing or replacing a drive and possibly pulling SMART data from the drive for health monitoring.

So, I am losing the smart status. Didn't realize that. I guess I can monitor it at the host level or reconfigure with disk controller passthrough. I am not going to buy new hardware, so can a motherboard controller be passed through? I wouldn't think so since the host needs it. If that is the case, then I will certainly monitor the disks outside of TrueNAS.

 

[Chris Moore]

I would suggest that you find a way to pass a whole controller to the NAS OS. If you don't, I think you are setting yourself up for failure.

We have seen it happen before. It isn't a matter of if it will happen, just when...