Truenas SCALE on XCP-ng / Xen and disk serial numbers

[redakula]

Hi

I am setting up SCALE on xcp-ng 8.2.1 (Xenserver). Xen guest tools are installed.
However i get the following error due to the xen disks not having a serial number i assume:

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 423, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 459, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 741, in do_create
    verrors.check()
  File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 62, in check
    raise self
middlewared.service_exception.ValidationErrors: [EINVAL] pool_create.topology: Disks have duplicate serial numbers: '' (xvda, xvde).

I was able to add the two xen disks (xvda and xvde) wihout issue as long as i checked the "show disks with non unique serial numbers". But after the initial setup i am unable to add any pools at all without encountering the above error.
Both additional xen disks and disks on a passed through HBA controller.

So basically i have SCALE up and running on a xen disk as boot drive and another xen disk for apps and misc. data. But i am unable to add the data disks that i planned on adding on the HBA controller.

Google does not have any hits that seem to match my error so i am interested to know if anyone has been able to run SCALE under XEN?

I hope i have all relevant data

Details on setup:
Truenas SCALE 22.02 release
On a supermicro x11scl-f, i3 9100 with 64GB ram. LSI 2308 passed through to the Truenas scale VM.

 

[Kris Moore]

I know others have run through Xen in the past, but this really should be a ticket on https://jira.ixsystems.com for us to investigate. We may need some software fixes to better accommodate this type of setup.

 

[sretalla]

I think you're ignoring all the popular wisdom on virtualizing TrueNAS...specifically you need to pass a SATA controller of some kind directly into the VM, not use Virtual Disks.

https://www.truenas.com/community/t...ide-to-not-completely-losing-your-data.12714/

 

[redakula]

I know others have run through Xen in the past, but this really should be a ticket on https://jira.ixsystems.com for us to investigate. We may need some software fixes to better accommodate this type of setup.

Hi

Ok i will create a ticket there - i'm new to truenas so tried the forums first

If i create pools manually and import them into truenas via the gui it seems to work fine so could be simply the check in the GUI...

Regardfs

 

[redakula]

I think you're ignoring all the popular wisdom on virtualizing TrueNAS...specifically you need to pass a SATA controller of some kind directly into the VM, not use Virtual Disks.

https://www.truenas.com/community/t...ide-to-not-completely-losing-your-data.12714/

Hi

The xen disks are just boot and some metadata... All important stuff is on the LSI controller available to truenas directly.
Using the xen disks mean i don't have to allocate a full physical disk for just a few GB.

Regards

 

[sretalla]

All important stuff is on the LSI controller available to truenas directly.

Good.

some metadata

Do you mean pool metadata? That's pretty important. (and it's pool integral, so will take down your pool with it if lost)

 

[MorMadness]

as @sretalla stated you need to pass the entire HBA device not just the disks. which means any disk that is on the HBA controller can not be used for any other function except to be passed to the TNS VM.

I had the same thing using VMWare. If you create each disk as a raid 0 or if you leave the controller in ACHI mood and only pass the raw disk to the VM of TNS then you get a disk serial duplication issue.

I circumvented this by booting VMware from a 250GB NVMe PCI drive and using the rest as a datastore. Then I created a VM for TNS on that NVMe datastore and passed the entire RAID controller using the passthrough function and gave the VM the PCI RAID device that was set to HBA. Once in TNS I created NFS and iSCSI folders and pass those back to VMware to use as storage. since it is all on the same physical server there is no lag on accessing the drives.

 

[MorMadness]

Thought pictures would help.
DL380 G9 24SFF with 20 SSD and 4 HDD ((10)1TB SSD (6) 480GB SSD (4) 2TB SSD (4) 2TB HDD) and 1 NVMe PCI Drive.
(2) E5-2680v4 Xeon
128GB RAM
This is my main VM server

did the same thing with another DL380 G9 12LFF Drives. This one is a media and backup server so less RAM and much larger Drives
(2) E5-2660v3 Xeon
64GB RAM

I HA the VMware installation and TNS VM amongst 3 nodes so I have replications of the configs, so if a server goes down I can be back up with little effort.

The best thing about running TNS on VM is that you can add and subtract anything you need. second Network adaptor, extra drive from itself for temp data. The 8GB VMware drive that is mounted is from truenas via a NFS attached storage to put the APPS store data on, this keeps it neat on the TNS server and didn't have to add physical drives.

 

[mroptman]

There's a work around that someone found. See the comments on this issue: https://ixsystems.atlassian.net/browse/NAS-118670

Looks like you will need to revert back to 12.0-x train to setup new pools. Boot back into 13.0-x and then import the new pools.