Server cluster

[adam23450]

I asked about it once, but I will ask again. I would like to make a server cluster. I need it to improve the performance of my applications and increase disk capacity. Is it even possible to do it at home? I would like to have a central management unit and 2 servers with disks connected to it, and so that I could add more. I would also like to have one virtual drive that they can expand by adding more servers.

Next question.

As data centers do, different hosting companies.

 

[sretalla]

You need to look into TrueNAS SCALE (only in BETA for now), but it's the open source way to get what you're talking about. (https://www.truenas.com/truenas-scale/)

If you want to cluster TrueNAS CORE, you are out of luck... you need Enterprise instead (only on iX Hardware) to have that for now (fully supported).

 

[adam23450]

You need to look into TrueNAS SCALE (only in BETA for now), but it's the open source way to get what you're talking about. (https://www.truenas.com/truenas-scale/)

If you want to cluster TrueNAS CORE, you are out of luck... you need Enterprise instead (only on iX Hardware) to have that for now (fully supported).

I saw truenas scale but how do I do what I described on a regular linux Ubuntu or debian system

 

[sretalla]

Scale does the file clustering using glusterfs, so start with that.

 

[danb35]

I doubt I'll be able to give you complete answers, but it will certainly help if you can more clearly define your objectives. It's one thing to have shared storage--Gluster is one way to do that (and it's what TrueNAS SCALE uses); Ceph is another (and it's what Proxmox VE uses). Either of them can act as a shared filesystem that multiple clients can use simultaneously. But if you're wanting to run a single application across more than one server, that's sounding very much like a custom thing.

 

[adam23450]

I doubt I'll be able to give you complete answers, but it will certainly help if you can more clearly define your objectives. It's one thing to have shared storage--Gluster is one way to do that (and it's what TrueNAS SCALE uses); Ceph is another (and it's what Proxmox VE uses). Either of them can act as a shared filesystem that multiple clients can use simultaneously. But if you're wanting to run a single application across more than one server, that's sounding very much like a custom thing.

I wonder how companies that sell, for example, web hosting or private VPS servers have this solution

 

[Heracles]

I wonder how companies that sell, for example, web hosting or private VPS servers have this solution

They have it from enterprise-class level components and not DIY / Gaming / Recycled parts...

 

[danb35]

I wonder how companies that sell, for example, web hosting or private VPS servers have this solution

Well, you've mentioned two quite different applications, and I expect they have two quite different sets of solutions. In the latter case, I have a few VPSs with Contabo, and they use Proxmox (which I determine by being VNC'd into the VPS when it boots; it shows the Proxmox splash screen), so I'd imagine their system is similar to mine, though on a much larger scale--a large cluster of Proxmox hosts, likely using Ceph for shared storage. Any given VPS "lives" on one host, though it can be seamlessly migrated to any other host in the cluster as needed. Proxmox supports clustering out of the box, and for free; nothing of what I've mentioned so far would need anything special.

For web hosting, I'd have to speculate more, but I'd expect a large number of web servers, likely using some form of shared storage, behind a load balancer of some sort.

As best I understand (and with the caution that there's likely a great deal I don't know), "clustering" is going to need a good deal of specificity in exactly what you're going to want to do. For some applications (like VM hosting), it can be pretty trivial--Proxmox has it all baked into the GUI, and xcp-ng also makes it very easy (I haven't worked with other hypervisors to say). SCALE is also moving that direction. If you have another specific use case in mind, someone else might be able to suggest how to go about it. But if you're just looking for a way to use the resources of n computers to run arbitrary code, you're likely to be coding it from scratch.

 

[morganL]

I asked about it once, but I will ask again. I would like to make a server cluster. I need it to improve the performance of my applications and increase disk capacity. Is it even possible to do it at home? I would like to have a central management unit and 2 servers with disks connected to it, and so that I could add more. I would also like to have one virtual drive that they can expand by adding more servers.

Next question.

As data centers do, different hosting companies.

You are asking about the reason that HA (dual controller) storage was developed. It's to have reliable systems with a shared drive where multiple compote planforms can use. Shared storage using HA is the most economic up to a point .. probably 10PB and 10GB/s. This is because the drives are shared between two controllers and so the data replication is minimal.

Clustered storage can do the same, but the software is more complex and the storage must be replicated across nodes so that any single node can fail. Because nodes can fail, you need to handle that condition which can be quite complex. Typically it easier with 3 nodes rather than 2 nodes. The data replication can be less (2 data + 1 Parity), and there is no chance of a "split brain". Recovery from node failure is more reliable. This is what TrueNAS SCALE and gluster do. So you can throw away "central management" unit and just run 3 nodes. You can use TrueCommand on docker on your laptop or other server to manage.