Proxmox and TrueNAS

[adam23450]

Hi. Can the disk resources shared on TrueNas be linked to proxmox? If so, does it work well together? My next question is how large data centers cope with the expansion of space for virtual machines? On proxmox, you can connect several drives with TrueNas to get one virtual hard drive that I can scale and share on all nodes. Can it be solved differently?

 

[jgreco]

Can the disk resources shared on TrueNas be linked to proxmox? If so, does it work well together?

Yes, it's fine, of course.

My next question is how large data centers cope with the expansion of space for virtual machines?

Data centers do not typically do this; data centers provide space, power, cooling, and security. When you place your gear in a data center, you get to cope with the expansion of space for virtual machines in whatever way you find convenient. Usually dictated by choice of hypervisor.

Small to midscale vendors of "virtual machines", which sometimes like to call themselves "cloud vendors" etc., typically use some form of SAN storage. SAN storage allows one large storage system to provide raw space to a bunch of hypervisors. SAN typically has some sizing limits, but your FreeNAS system can definitely act in that role.

Larger vendors get to a point where it becomes more difficult to have centralized storage. The fastest storage tends to be local-to-the-hypervisor storage, but this is more difficult to manage and share between customers.

To expand the space of a virtual machine, you need available space on the backing store. Then you increase the space. And you're done.

On proxmox, you can connect several drives with TrueNas to get one virtual hard drive that I can scale and share on all nodes. Can it be solved differently?

Yes, of course it can.

TrueNAS probably should not be run as a virtual machine if you are using it in a serious role for providing VM storage. You should probably have a dedicated TrueNAS system. Plus all the other stuff in

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

But of course you can solve this however you wish. It's just that some solutions will work better, be more reliable, and be safer than others.

 

[adam23450]

Yes, of course it can.

TrueNAS probably should not be run as a virtual machine if you are using it in a serious role for providing VM storage. You should probably have a dedicated TrueNAS system. Plus all the other stuff in

I can't find anywhere how to create such a huge drive and scale it to proxmox late. Do you have any links to this?
TrueNas would be the physical machine from which the disks will be shared, I know TrueNas requires direct access to the disks.

 

[jgreco]

It isn't clear what you're asking for.

Proxmox can use NFS for VM backing store. This is probably the simplest method if you are not too familiar with setting things up.

Storage: NFS - Proxmox VE

pve.proxmox.com

It can also use iSCSI, which is more difficult for beginners, and tends to require more resources.

Storage: iSCSI - Proxmox VE

pve.proxmox.com

 

[adam23450]

It isn't clear what you're asking for.

I drew you what I mean, 1 squares are TrueNas and green is proxmox.

 

[jgreco]

I'm kinda lost here at what you're trying to ask. I think you're probably missing some basic foundations of shared storage design.

You have drawn ... what, exactly? You have four things labeled "TrueNas 3x 8TB". What does this represent? Physical TrueNAS hosts? Pools? vdevs in a shared pool? You then show three nodes below it, in a green box saying "one large disk".

You make "one large disk" by putting all your disks into a pool. To get a "disk" with 96TB of usable space, you probably want 48 8TB hard drives in mirror pairs. The total pool space of such would be 192TB of space, and given the 50% allocation rule, you'd get about 96TB of truly usable space out of that. You'd want a FreeNAS host with 256GB of more of memory in that role if you wanted it to be quick and efficient.

 

[Patrick M. Hausen]

This depends on the question which filesystems in Proxmox can be shared between multiple nodes and mounted simultaneously. VMware ESXi has got VMFS for that and you can have a SAN or iSCSI block storage unit mounted my several hypervisor hosts just fine.

If Proxmox has got something like this, good. If not, there is always NFS. You can create a single pool and dataset to share via NFS on your TrueNAS system and mount it on all your Proxmox nodes.

If what you are asking is how to use multiple TrueNAS hosts to create a "cluster" filesystem - that is not possible with release software at the moment. TrueNAS CORE is single node, only. So all your disks go into one server.

TrueNAS SCALE is trying to achieve a distributed backing store by the means of GlusterFS as an overlay over local ZFS storage. It is in early beta/RC and you might want to try it. A distributed filesystem in SCALE can to my knowledge not yet shared via NFS, but it can be shared via the Gluster proprietary mechanism. So if there is a GlusterFS client in Proxmox and a GlusterFS can be used as a backing store for VMs, that might work.

Many "ifs" here ...

 

[adam23450]

I'm kinda lost here at what you're trying to ask. I think you're probably missing some basic foundations of shared storage design.

You have drawn ... what, exactly? You have four things labeled "TrueNas 3x 8TB". What does this represent? Physical TrueNAS hosts? Pools? vdevs in a shared pool? You then show three nodes below it, in a green box saying "one large disk".

You make "one large disk" by putting all your disks into a pool. To get a "disk" with 96TB of usable space, you probably want 48 8TB hard drives in mirror pairs. The total pool space of such would be 192TB of space, and given the 50% allocation rule, you'd get about 96TB of truly usable space out of that. You'd want a FreeNAS host with 256GB of more of memory in that role if you wanted it to be quick and efficient.

Yes, I miss some basic knowledge about designing storage. That is why I am asking how I can solve it and how I would solve it. What I drew are physical hosts (I haven't written the fact)

 

[jgreco]

So if there is a GlusterFS client in Proxmox and a GlusterFS can be used as a backing store for VMs, that might work.

I'm interested in seeing what the performance tax is like. Part of the problem with GlusterFS to date has been analogous to the "ZFS problem"; conventional storage, just like UFS, is fairly straightforward and easy to understand. Going from NFS to Gluster is very much like going from UFS to ZFS, with lots of new hazards and stuff to know. Upgrades are tricky, and there are moving pieces in multiple places on the network, making it somewhat trickier to understand and administer correctly than ZFS, which at least has the benefit of being just one thing in one place.

I'm hopeful that Scale can bring to Gluster what FreeNAS did for ZFS. It's not that I mind getting down and dirty inside the thing, it's just that I resent *having* to become a SME on the whole thing in order to be able to use it "safely".

 

[adam23450]

@jgreco Can you tell me how I can get 1 - 96TB disk space? I need it to store the X-ray pictures

 

[ChrisRJ]

@adam23450 , I think you should consider hiring a local storage expert. This looks like a commercial setting and a forum cannot replace professional support in such a scenario.

 

[adam23450]

@adam23450 , I think you should consider hiring a local storage expert. This looks like a commercial setting and a forum cannot replace professional support in such a scenario.

I have to acquire this knowledge somewhere, in my opinion, it is best to do it on the forum and in practice. Each specialist started with something...

 

[caedmonstone]

@Adam 23450 After reading through a lot of post over several forums you post here sound the closest to what I am trying myself. Read laptop as machine if you want, the below is my setup that sound like what you were trying

Laptop1 (4core 16gig Memory 2TB Harddrive)
Proxmox Server

Laptop2 (2 core 16gig Memory 1TB Onboard HD and Growing External Storage)
TrueNas Core Server

Raspberry Pi
Bunch of things like
Pi-Hole
WireGuard etc

TPlink Router/Modem
Both laptops and Pi connect to router via ethernet cables.

Basically as I read it you tried to get Laptop 1 to access and use the pool storage on Laptop 2???

The question I have is did you succeed and if so how???

 

[danb35]

Using laptops--especially with only two cores--as TrueNAS servers is criminally-negligent design. But that aside, this:

Basically as I read it you tried to get a Proxmox host to access and use the pool storage on a TrueNAS server

is trivial, and has already been explained up-thread: Create a NFS share on the NAS, and mount it on your Proxmox host. Consult the respective docs for details on how to perform these respective operations. I've been doing this for the last several years.

 

[Dave41]

Here is a Lawrence Systems video on this exact subject from this week. . . https://www.youtube.com/watch?v=mdHmcwWTNWA

 

[Stux]

I'm kinda lost here at what you're trying to ask. I think you're probably missing some basic foundations of shared storage design.

You have drawn ... what, exactly? You have four things labeled "TrueNas 3x 8TB". What does this represent? Physical TrueNAS hosts? Pools? vdevs in a shared pool? You then show three nodes below it, in a green box saying "one large disk".

You make "one large disk" by putting all your disks into a pool. To get a "disk" with 96TB of usable space, you probably want 48 8TB hard drives in mirror pairs. The total pool space of such would be 192TB of space, and given the 50% allocation rule, you'd get about 96TB of truly usable space out of that. You'd want a FreeNAS host with 256GB of more of memory in that role if you wanted it to be quick and efficient.

Plus SLOGs if this is for VM storage