Recommended Memory to Boot Drive Ratio?

[MotorN]

I was running a TrueNAS Scale VM for about 6 months, basic SMB storage, 4C 16GB with no issues. After reading about ram and more is the better I re-installed Scale with 128GB. I was struggling to install on the same 120GB SSD and eventually gave up and used a 256GB NVME. System seemed fine for about 3 days and then the VM stopped. ESXi would not power the VM back on, started recreating copies of the the VM and creating orphans. ESXi event log says I ran out of storage space for the swap.

Is it true that you should have a boot drive twice as large as your memory? This would explain why I couldn't re-install on the 120GB and why the 256GB died after 3 days.

 

[NugentS]

Nope - I had a 128GB server on a 16GB Boot disk

 

[HoneyBadger]

ESXi would not power the VM back on

Can you post the details of your ESXi/vSphere configuration here?

What you've described here is ESXi refusing to power the VM on because of a lack of datastore space for the .vswp file - but when done following best practices, all of the memory assigned to the TrueNAS VM will be reserved (by design) because PCIe hardware passthrough will be in use for the HBA/storage controller, so there wouldn't be a .vswp file created at all.

So I'm concerned you have a situation where your data isn't as safe as it should be.

 

[MotorN]

Can you post the details of your ESXi/vSphere configuration here?

What you've described here is ESXi refusing to power the VM on because of a lack of datastore space for the .vswp file - but when done following best practices, all of the memory assigned to the TrueNAS VM will be reserved (by design) because PCIe hardware passthrough will be in use for the HBA/storage controller, so there wouldn't be a .vswp file created at all.

So I'm concerned you have a situation where your data isn't as safe as it should be.

.

 

[HoneyBadger]

Right now it would seem that you've exhausted the space on a datastore somewhere (or powering the machine on would result in it) so you're dealing with a VMware level issue first and foremost.

IBM Thinkstation P700
Dual E5-2660 v3
196GB
8-Port SATA/SAS HBA (https://www.aliexpress.com/item/4001039658674.html?spm=a2g0o.order_list.0.0.6b341802qXrSgO)
***My computer only has 4 bays with an allowed configuration of 4 x 3.5" + 4 x 2.5" drives
Asus Hyper 3x4 Card (500GB(VMs) + 256GB(Cache?) installed)
GT 710

I currently have ESXI installed on a 120GB SATA SSD connected to HBA, Vcenter VM installed on a 2TB HDD connected to HBA, TrueNAS VM on 120GB SSD to HBA (2C with 8GB) and 3 x 8TB HDD connected to HBA with Raid-z.

There are a couple of challenges here.

Your TrueNAS VM being "installed on a 120GB SSD" means that it will never be able to save a 128GB .vswp file to there. But if you have ESXi installed on an HBA-connected drive and are using other ports for local datastores, then it won't be able to be passed through (via PCIe passthrough) to your TrueNAS VM.

What needs to be done is for your ESXi install to exist on a completely separate storage controller - eg: the onboard SATA ports of your P700, or one of the NVMe devices on the Hyper3 x4, such as the 500GB stick - you then use the spare space on the installation device (eg: the 120GB SATA SSD) or another device on the same controller (another SATA port on the P700 motherboard) to create a VMFS datastore. In that datastore, local to ESXi, you make a virtual disk for TrueNAS and install the operating system there - you then use PCIe passthrough to assign the entire 9340-8i add-in-card (hopefully with IT firmware) to the TrueNAS VM. The only drives you connect to the HBA are those you want to use within TrueNAS. If you want to pass one of the NVMe devices from your Hyper3 x4 card (the 256GB cache card, for example) then you can add that as a second passthrough device.

If desired, you can then create a virtual storage network and join both the TrueNAS VM and a vmkernel interface to it - that lets you then consume the TrueNAS-presented NFS/iSCSI resources back within ESXi.

Note that a virtual TrueNAS setup is a fair bit more complex than a bare-metal one.

 

[diogen]

ESXi would not power the VM back on... ESXi event log says I ran out of storage space for the swap.

As a temporary fix: just change the swap file location

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-D888BE08-187B-4FF2-9918-7EC27D0EC376.html

By default it is located together with the OS what is a bad idea regardless of size...

But ideally you should work to set it up as outlined by @HoneyBadger.
If your ESXi version is 6.x, install it on a good USB3 stick...

 

[MotorN]

Right now it would seem that you've exhausted the space on a datastore somewhere (or powering the machine on would result in it) so you're dealing with a VMware level issue first and foremost.



There are a couple of challenges here.

Your TrueNAS VM being "installed on a 120GB SSD" means that it will never be able to save a 128GB .vswp file to there. But if you have ESXi installed on an HBA-connected drive and are using other ports for local datastores, then it won't be able to be passed through (via PCIe passthrough) to your TrueNAS VM.

What needs to be done is for your ESXi install to exist on a completely separate storage controller - eg: the onboard SATA ports of your P700, or one of the NVMe devices on the Hyper3 x4, such as the 500GB stick - you then use the spare space on the installation device (eg: the 120GB SATA SSD) or another device on the same controller (another SATA port on the P700 motherboard) to create a VMFS datastore. In that datastore, local to ESXi, you make a virtual disk for TrueNAS and install the operating system there - you then use PCIe passthrough to assign the entire 9340-8i add-in-card (hopefully with IT firmware) to the TrueNAS VM. The only drives you connect to the HBA are those you want to use within TrueNAS. If you want to pass one of the NVMe devices from your Hyper3 x4 card (the 256GB cache card, for example) then you can add that as a second passthrough device.

If desired, you can then create a virtual storage network and join both the TrueNAS VM and a vmkernel interface to it - that lets you then consume the TrueNAS-presented NFS/iSCSI resources back within ESXi.

Note that a virtual TrueNAS setup is a fair bit more complex than a bare-metal one.

Thanks for the detailed response. I will try to replicate the configuration you suggest.

* I am pretty sure my HBA is not in IT mode. Is the following firmware compatible with my HBA...

LSI 9300-xx Firmware Update

Hey Community, If you are using an LSI 9300 HBA with FreeNAS or the soon-to-be TrueNAS CORE, you may experience some performance issues causing the controller to reset when using SATA HDDs. After working with Broadcom, we’ve come up with a...

www.truenas.com

* I will install ESXi on the onboard SATA controller. I have a 750GB SSD. In addition to TrueNAS I want/need to install Vcenter server and PF Sense. Would the proper procedure be to install ESXi and then setup 3 virtual disks in the created VMFS datastore? With this configuration does that mean all my ESXi VMs will run on the 750GB SSD (current and future). If so seems the bigger the SSD the better, I do have a 1TB also.

 

[Redcoat]

* I am pretty sure my HBA is not in IT mode.

sas3flash -list on the command line will let you know...

 

[MotorN]

sas3flash -list on the command line will let you know...

I reinstalled TrueNAS Scale bare metal, seems to be the easiest way to flash the HBA. I installed on a 120GB SSD connected to the onboard SATA controller, nothing is connected to the HBA. I can't see the HBA.

*Update: I can't see the HBA from the USB boot drive either.

 

[Redcoat]

Oh, shoot - I should have been smart enough to read back and see that you don't have an LSI HBA. I was mislead by @HoneyBadger 's "the entire 9340-8i add-in-card" (where did that come from?).

I'm neither an ESXI nor a Scale (yet?) person, so I don't actually know if IT mode is required for the disk controller in ESXI. Someone else is going to have to chip in here and advise you on the path forward with respect to this storage adapter issue.

Good luck - I'll be following.

 

[MotorN]

I have a proprietary card for older IBM/Lenovo Thinkstations, RAID flex(slot) adapter. It appears I have a SAS-MFI BIOS that requires storcli not sas3flash.

 

[diogen]

I have a proprietary card for older IBM/Lenovo Thinkstations, RAID flex(slot) adapter.

Proprietary isn't good for TrueNAS. Get a LSI 9207-based HBA - Dell H310 or IBM M1015

Chinese PCIE to SATA Adapter

Anyone tried this: https://www.aliexpress.com/item/1005003504822556.html Will this work with TrueNas ?

www.truenas.com

They can be found on eBay for $30 or so...

Your P700 uses a C612 chipset, meaning only PCIe v.2 support.
Hence you don't need higher end HBA (e.g. Dell H330 that uses PCIe v.3.0)...

 

[MotorN]

I'm going to have to agree, proprietary is no good. Seems I can't flash the card to IT mode. Is IR mode that bad? The card and cables are incorporated into my system, mobo removal was needed. I actually have 2 of these P700s with the same configs.

Can you recommend a external HBA suitable for my system? I am thinking of keeping the current card instead of using the onboard sata and passing through the external HBA.

 

[diogen]

Can you recommend a external HBA suitable for my system?

By external you mean discrete? This one should be good enough...

The white slot in your third picture is a standard (old) PCI slot.
The black ones next to it are PCI-Express slots (what you need for the H310).

Based on this you have two PCI-E slots, x8 and x4, both open-ended (use the first).

Make sure you have the right bracket and cables to connect the HBA to the disks.

If the H310 is not flashed into IT mode (i.e. is not converted from PERC to HBA), use this writeup to do it.
Or get one that already is, this for example (it has cables and both brackets)...

 

[MotorN]

By external you mean discrete? This one should be good enough...

The white slot in your third picture is a standard (old) PCI slot.
The black ones next to it are PCI-Express slots (what you need for the H310).

Based on this you have two PCI-E slots, x8 and x4, both open-ended (use the first).

Make sure you have the right bracket and cables to connect the HBA to the disks.

If the H310 is not flashed into IT mode (i.e. is not converted from PERC to HBA), use this writeup to do it.
Or get one that already is, this for example (it has cables and both brackets)...

I guess the best thing to do is sell my current cards and buy one of the ones you recommend.

Thanks to all for your patience and assistance. I am sure I will be back once I receive the new cards but I will post in the virtualization sub-forum that is more appropriate.

 

[HoneyBadger]

Oh, shoot - I should have been smart enough to read back and see that you don't have an LSI HBA. I was mislead by @HoneyBadger 's "the entire 9340-8i add-in-card" (where did that come from?).

That's the chipset the card is based off of (it's a different form factor, similar to the "Mini Mono" cards in Dell servers) and it seems to share firmware with the ServeRAID M1215 (which does have an IT mode) but the cards could have some peculiarities that would make crossflashing it a risk of bricking.

I guess the best thing to do is sell my current cards and buy one of the ones you recommend.

If you're stuck with IR firmware on the built-in "Flex" card, that could be your ESXi storage (could even set up a RAID10 for redundancy at that level) and then use a separate H310 or other IT mode HBA for passthrough to TrueNAS.

 

[Redcoat]

That's the chipset the card is based off of

Thanks for the explanation - I did not find that connection when I searched (briefly).

 

[diogen]

The white slot in your third picture is a standard (old) PCI slot.

I stand corrected...
Based on this document only the x4 slot is PCIe v.2, the rest (five of them) are PCIe v.3.
And there are no regular PCI slots; some of them just happened to be white (see pic).

Hence, you might want to consider a H330 card like this and then you can reuse the cables that connect the drives to the flex card...

 

[MotorN]

I stand corrected...
Based on this document only the x4 slot is PCIe v.2, the rest (five of them) are PCIe v.3.
And there are no regular PCI slots; some of them just happened to be white (see pic).

Hence, you might want to consider a H330 card like this and then you can reuse the cables that connect the drives to the flex card...
View attachment 57653

The PCI slot are divided by CPUs. The white are only usable when a 2nd CPU is installed.

I can't use the flex adapter and an HBA together. Currently the drives are connected to the flex cable backplanes. I need to remove those and flip the drives to connect to the HBA however my onboard SATA controller does support RAID.

I bought a couple of M5110/LSI 9207-8I HBAs (little concerned about the cables, ideally I think I want right angled). I also grabbed a couple of 5.25 HDD adapters because I only have 4 HDD bays.

 

[diogen]

I can't use the flex adapter and an HBA together.

Why?
Do you mean there are not enough drives in the box to connect to two storage controllers?
Or this motherboard does not support running two controllers at the same time (that I really doubt)...?

A H330 HBA (and some others) has the same connectors as your flex adapter.
So, you could simply disconnect the cables from the flex adapter and connect to the H330...

I don't know what the other end of the cable is.
Since it is capable of accommodating SAS and SATA drives, I suspect it's some sort of a backplane...