Two hot-pluggable systems (nodes) in a 2U form factor

[majona1se]

I am looking at SuperMicro SuperStorage 2029P-DN2R24L for running TrueNAS Scale.
This server is a 2U unit with two separate servers in it, but they share the drives via PCI-E
And it's the PLX PEX8717 chip on the motherboard.

The idea I were sold into is the ability to have failover on the node level, in one chassis sharing the same drives.
From what I understand this is technically possible due to the PLX NTB and the fact that the NVMe drives are dual channel.

Could someone lead me into a go or no-go for TrueNAS Scale in this server?

Two hot-pluggable systems (nodes) in a 2U form factor. Each node supports the following:

Dual Socket P (LGA 3647) support 2nd Gen Intel® Xeon® Scalable processors (Cascade Lake/Skylake)‡

24 U.2 Hot-swap dual-port NVMe drive bays

Dedicated node to node connectivity featuring high performance PLX NTB PCI-E 3.0 x8 and IPMI for robust node fail-over support

 

[Ericloewe]

PEX8717

Pricy little ICs...

Could someone lead me into a go or no-go for TrueNAS Scale in this server?

Well, here's the thing: This is very far from a plug-and-play thing. It would almost certainly involve quite a bit of OS customization if you wanted it to work properly. And a bunch more work for applications to nearly-seamlessly move from one node to the other. I don't think this is viable due to the amount of time, effort, testing and pain you would have to go through.

Bottom line is that if you need HA, iX sells that and it's fundamentally very similar to what you're looking at, but with all the hard work figured out ahead of time: https://www.truenas.com/m-series/
I don't know if they do more than a handful NVMe disks in HA, though. That gets really expensive, really quickly.

 

[HoneyBadger]

Hello,

You're correct in that dual-pathed drives and PCIe non-transparent bridge (NTB) makes this possible, but HA node failover with shared-storage is only supported in the Enterprise releases of TrueNAS.

If you're looking for this kind of redundancy and horsepower, I'd recommend getting in touch with iXsystems and looking at the M-series servers.

 

[majona1se]

I am so disappointed, back to the drawing table then. Were looking forward into learning Scale.
How do you other people build high available solutions with it? From a storage perspective.
What I compare to is a HPE MSA 2600 thats pretty much plug and play, but thats only storage and not HCI.

 

[Etorix]

How do you other people build high available solutions with it?

I guess the answer is that high availability is for business, and businesses just BUY solutions rather than building their own. After all, iXsystems has to make money with its Enterprise software for us amateurs to enjoy the free version.

 

[majona1se]

I guess the answer is that high availability is for business, and businesses just BUY solutions rather than building their own. After all, iXsystems has to make money with its Enterprise software for us amateurs to enjoy the free version.

Yes, you are spot on! I booked a meeting with them yesterday, so I'm waiting for a call.
Anyhow, I have to fire up a computer at home that I currently use as foot rest, with Scale! :D

 

[Patrick M. Hausen]

How do you other people build high available solutions with it?

"Do it right": buy from iX
"Poor man's solution": have two identical systems and set up replication - requires manual intervention in case of failure

We run the second variant. We can live with being offline for an hour or two. Risk assessment first, choice and implementation second

 

[HoneyBadger]

Yes, you are spot on! I booked a meeting with them yesterday, so I'm waiting for a call.
Anyhow, I have to fire up a computer at home that I currently use as foot rest, with Scale! :D

The good news is that you can pick up a lot of knowledge by using SCALE in a single-node setup that will transfer over into an HA setup. The bad news is, you're going to lose your footrest. ;)

HA storage is one thing certainly, as you indicated with your HP gear, but true HA from a compute perspective is more difficult. This can be done at an application level, with multiple software nodes forming a cluster and brokering or balancing connections. If one running node fails, another can take over the IP from separate hardware - as long as they have access to a shared piece of storage for logging in-progress transactions and keeping quorum, the client application sees a slight hiccup and then continues on.

Software that can't cluster in this manner needs to be supported by much more complex things, like running CPU/memory operations in lock-step across separate physical hosts. This usually means a very high-bandwidth interconnect required, and a reduction in performance from this "mirroring" of CPU/RAM.

 

[mytime34]

I picked up one of these servers, on the cheap. It came with 24x 1.92tb NVME drives, Silver 4110, 128gb ram

Due to this having dual 10gbe internal network links, between the nodes, I was going to combine them to get 20gbe.
So far I have 1 node running Truenas 13 and it sees all the drives and the internal network links.
Now I have to figure out what pool config for the drives. (it is very cool that both nodes can see the drives)

 

[anodos]

I picked up one of these servers, on the cheap. It came with 24x 1.92tb NVME drives, Silver 4110, 128gb ram

Due to this having dual 10gbe internal network links, between the nodes, I was going to combine them to get 20gbe.
So far I have 1 node running Truenas 13 and it sees all the drives and the internal network links.
Now I have to figure out what pool config for the drives. (it is very cool that both nodes can see the drives)

This is actually very dangerous. Importing the pool on both nodes at the same time will corrupt your pool.

 

[mytime34]

This is actually very dangerous. Importing the pool on both nodes at the same time will corrupt your pool.

Here are a few options
12x 1.92tb drives in a Raidz2 (Node1)Truenas
12x 1.92tb drives in a Raidz2 (Node2) Truenas
Proxmox Node 1 and Node 2 (cluster) and 24x 1.92 drives Draidz3

Since I got this for a steal and it uses less power than my current setup, I want to find the best configuration (since DDN will not provide a DOM or ISO image, to reinstall the OS.