SOLVED multiple "ada0" partitions on Xen PV w/passthrough

[leecro]

Hi,

I've been using FreeNAS for several months now, in a few different configurations - some production, some educational. As disclosed in the title, this particular issue was encountered on a Xen PV platform - nevertheless I believe this may be a storage issue rather than being a consequence of the virtualized setup (which I know is not recommended)

I had no difficulty with FreeNAS 9.0 release on XenServer 6.1, but I wanted to pursue tighter integration than could be done on an HVM system and am using godbolt's (9.1.1) VHD from http://forums.freenas.org/threads/how-i-got-a-xenhvm-kernal-and-xen-tools-working-in-freenas.15287/

After configuring the system for xenguest functionality and setting up host details etc, I added the drive controller in passthrough as I'd done in previous scenarios without difficulty:

Code:

xe vm-param-set other-config:pci=0/<PCI ID> uuid="<VM UUID>"

The drives had been used before as part of another ZFS volume, but I'd wiped them all and regenerated the GPT metadata before attempting to import.

Only 5 of the 6 drives appeared within the GUI. I've seen references to this being the case when drives are already partitioned (these are not) or when non-AHCI modes are set in the BIOS (AHCI remains set)

After various searching, I've gathered the output from several commands and I'm somewhat baffled by the results:

Firstly, camcontrol:

Code:

[root@fm ~]# camcontrol devlist                                     
<WDC WD20NPVT-00Z2TT0 01.01A01>    at scbus2 target 0 lun 0 (ada0,pass0)
<WDC WD20NPVT-00Z2TT0 01.01A01>    at scbus3 target 0 lun 0 (ada1,pass1)
<WDC WD20NPVT-00Z2TT0 01.01A01>    at scbus4 target 0 lun 0 (ada2,pass2)
<WDC WD20NPVT-00Z2TT0 01.01A01>    at scbus5 target 0 lun 0 (ada3,pass3)
<WDC WD20NPVX-00EA4T0 01.01A01>    at scbus6 target 0 lun 0 (ada4,pass4)
<WDC WD20NPVX-00EA4T0 01.01A01>    at scbus7 target 0 lun 0 (ada5,pass5)

So far so good, to my mind - the drives are all WD Greens, and the latter two belong to a newer generation as implied by their model numbers.

Next, gpart:

Code:

[root@fm ~]# gpart show                                             
=>      1  3907583  ada0  MBR  (1.9G)                               
        1      62        - free -  (31k)                           
      63  1930257    1  freebsd  [active]  (942M)                 
  1930320      63        - free -  (31k)                           
  1930383  1930257    2  freebsd  (942M)                           
  3860640    3024    3  freebsd  (1.5M)                           
  3863664    41328    4  freebsd  (20M)                             
  3904992    2592        - free -  (1.3M)                           
                                                                     
=>      0  1930257  ada0s1  BSD  (942M)                             
        0      16          - free -  (8.0k)                         
      16  1930241      1  !0  (942M)                               
                                                                     
=>        34  3907029101  ada0  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                     
                                                                     
=>        34  3907029101  ada1  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                     
                                                                     
=>        34  3907029101  ada2  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                     
                                                                     
=>        34  3907029101  ada3  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                     
                                                                     
=>        34  3907029101  ada4  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                     
                                                                     
=>        34  3907029101  ada5  GPT  (1.8T)                         
          34        2014        - free -  (1M)                       
        2048  3907026944    1  linux-data  (1.8T)                   
  3907028992        143        - free -  (71k)                        

I'm new to a lot of this, but there appear to be two 'ada0' drives involved here - one MBR formatted (the VHD with FreeNAS itself) and one GPT (my first passed-through SATA drive).

Code:

[root@fm ~]# sysctl kern.disks                                       
kern.disks: ada5 ada4 ada3 ada2 ada1 ada0 ada0

Possible corroboration.

Finally,

Code:

[root@fm ~]# dmesg | grep ada0                                       
xbd0: attaching as ada0                                             
ada0 at ahcich0 bus 0 scbus2 target 0 lun 0                         
ada0: <WDC WD20NPVT-00Z2TT0 01.01A01> ATA-8 SATA 2.x device         
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)         
ada0: Command Queueing enabled                                       
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)     
ada0: quirks=0x1<4K>                                                 
ada0: Previously was known as ad4                                   
g_dev_taste: make_dev_p() failed (gp->name=ada0, error=17)  

I assume that final line is a pertinent error, but I've found no intelligible references to any of the above phrases - I'm still getting to grips with the basics of *nix systems in many ways, and all I can make of these threads is the suggestion that the two drives have a similar GPT label. Since one of them is MBR and the other has been recently reset several times (from within various liveCD environments), I don't see how this could be addressed.

I've tried various searches, but found nothing of relevance (probably due to my uninspired search terms). If my understanding of this problem is correct, is there any way to override the labelling without compromising the FreeNAS GUI's ability to correctly manage the abstracted structure?

Please let me know if I appear to have misunderstood anything significant above - this is not crucial to me, but if everything is as it appears to be then this may be an interesting bug. I'm OK with moving this to the n00bs area if necessary.

Thanks in advance for any help or advice you may be able to provide.

 

[mav@]

I guess you've got naming conflict between normal disk names of passed through AHCI controller and fake disk with the same name created by XEN PV drivers (for "compatibility" reasons). FreeBSD GEOM does not deny having two providers with the same name, that is why you see both of them in `gpart show`, but that is definitely considered wrong practice. As a workaround you can make CAM subsystem to not use ada0 device name. Try to set via loader prompt or via GUI tunables interface something like: hint.ada.0.at="scbus100".

 

[cyberjock]

Oh my. That just scares the bejesus out of me.

Are you passing through the SATA/SAS controller to the VM with PCI-passthrough aka VT-d or doing disk passthrough?

 

[leecro]

The process suggested by "mav@" worked perfectly, thankyou!

Code:

sysctl kern.disks: ada6 ada5 ada4 ada3 ada2 ada1 ada0

lovely stuff.

I'd just finished making another HVM to prove beyond doubt that 9.1.1 can see these drives correctly (it did). That tuneable sorted everything right out.

cyberjock, I went with the former approach. Since "no access to low-level drive control" is one of the excellent reasons listed in Please do not run FreeNAS in production as a Virtual Machine!, I decided to take that step and assign the SATA controller as a vm-param-set. This experience has made me aware that I need to learn a LOT more about BSD and *nix in general before trying to get fancy with unsupported configurations. There'll be nothing of any real importance on this system but that doesn't lessen my appreciation for such excellent assistance.

One minor issue with this system (ever since updating to 6.2) is that the dom0 kernel panics during the FreeNAS domU's shutdown - the messages suggest this is also a storage error, possibly even the opposite problem to the one described here (forcibly mounting the 'new' (no longer passed-through) drives as "/".) This is not worth investigating since it's trivial to reset the machine and hopefully I won't need to restart the domU for testing any more, thanks again. :)

 

[Nick Jeppson]

One minor issue with this system (ever since updating to 6.2) is that the dom0 kernel panics during the FreeNAS domU's shutdown - the messages suggest this is also a storage error, possibly even the opposite problem to the one described here (forcibly mounting the 'new' (no longer passed-through) drives as "/".) This is not worth investigating since it's trivial to reset the machine and hopefully I won't need to restart the domU for testing any more, thanks again. :)

I realize that this is an old thread but I wanted to comment to help others who might happen upon this problem. The dom0 kernel panic is a result of the storage controller not being claimed by the 'pciback' driver at boot. Xenserver is neat in that it will dynamically allocate pci passthrough devices upon request, and then return them when the VM shuts down. In this case the storage controller is being sent back to the system and the kernel tries to mount the disks there as root, replacing the real root and kernel panicking.

The fix is to add pciback.hide=(bus:device.function of the device) in /boot/extlinux.conf right after the vga= 785 line. Once that has been edited, run extlinux -i /boot and reboot. That should solve the problem.