Potential virtualized TrueNAS build

[Junx]

I have a Dell T7920 with dual Xeon Gold 6148's & 256gb of RAM. I've installed three quad NVME m.2 PCIE cards (Dell PN:6N9RH), and intend to run 12 x 4tb m.2 (undecided on specific model) and pass the controller cards through proxmox to a VM running TrueNAS Core. It has a few onboard interfaces and an older x520 based 10 gig adapter, which I'll likely dedicate to the TrueNAS box as well.

Use is primarily media storage - im leaning towards 12 x 4tb NVME because I believe they will put off less heat than a similar # of SAS or SATA drives, plus the much smaller footprint.

Summary:
40 cores, 80 threads
12 x 4tb m.2 NVME
256gb memory (can easily dedicate half or more to TrueNAS VM)

Feedback appreciated - my current TrueNAS Scale box is a Dell power edge T420 dual Xeon E5-2470V2, 192gb of ram and 8 x 4tb 7200 SAS drives w/x520 10g

 

[joeschmuck]

Use is primarily media storage

I suspect you are doing something else with it considering you are planning to use Proxmox and have a huge amount of RAM. If you are using this NAS primarily for media storage, why do you need such a great amount of RAM and CPU's?

My last comment is about the 12 x 4tb NVME cards. I'm not trying to be mean but this is very expensive for a media server. If you have the money then that a great for you but I'm going to drool a bit. For the use case I think it's all overkill, but doable. I don't care for Proxmox when used with TrueNAS. There have been some success stories but there have also been some big failures. ESXi seems to be more reliable of a hypervisor.

If you build it, post a few photos and tell us how it was building and setting up.

 

[Junx]

I suspect you are doing something else with it considering you are planning to use Proxmox and have a huge amount of RAM. If you are using this NAS primarily for media storage, why do you need such a great amount of RAM and CPU's?

My last comment is about the 12 x 4tb NVME cards. I'm not trying to be mean but this is very expensive for a media server. If you have the money then that a great for you but I'm going to drool a bit. For the use case I think it's all overkill, but doable. I don't care for Proxmox when used with TrueNAS. There have been some success stories but there have also been some big failures. ESXi seems to be more reliable of a hypervisor.

If you build it, post a few photos and tell us how it was building and setting up.

Media storage and plex, plus a few other utilities I run on a vm. quite honestly, the T7920 is an amazing piece of hardware that can be had pretty cheaply:
$595 - T7920 w/out cpu, mem or drives
$295 - 256gb DDR4 2666 RAM
$229 - Xeon 6148 PAIR (40 cores 80 threads)
$150 - 3x Dell PN:6N9RH

The T420 was even cheaper to build - it started as a T320 and I put a $100 t420 motherboard in it w/a pair of processors for $20 each. I picked up 10 used SAS 4tb 7.2k drives for $169 and its ram was ~$125 or so. The problem with it is heat and noise. The T7920 is a lot cooler and quieter.

Having the ability to run a few extra VM's is fun, intersting, but I've not been happy with TrueNaS scale as a VM server. co-locating my media collecting and streaming tools in the same hardware footprint as my storage will be cool.

 

[HoneyBadger]

I've not been happy with TrueNaS scale as a VM server

Sorry to hear that it didn't live up to your requirements.

Anything specifically that you found lacking you'd like me to bring back as feedback for our engineering team?

 

[Junx]

Sorry to hear that it didn't live up to your requirements.

Anything specifically that you found lacking you'd like me to bring back as feedback for our engineering team?

This wasn't intended as a hit piece, nor was it my reason for coming to the forum.

 

[HoneyBadger]

This wasn't intended as a hit piece, nor was it my reason for coming to the forum.

It's not being taken as one (and I hope I didn't come off as defensive!) - I'm just wondering if there's anything specifically about the experience that you might have wanted to share as feedback for us to improve on over time.

As mentioned by @joeschmuck there's probably several users who will need a moment to wipe the drool off after reviewing the specs of a system with a pair of Xeon Golds and 48TB of NVMe flash - as you mentioned, it's quiet, powerful, relatively inexpensive compared to something like an R740 - what's not to like? I believe it might also have the option for headless boot if you really wanted to make it into a server-like chassis; but since you plan to run some extra VM workload through the host hypervisor (Proxmox) you could install a powerful AMD or NVIDIA GPU and put it to use in containers or VMs.

 

[Junx]

It's not being taken as one (and I hope I didn't come off as defensive!) - I'm just wondering if there's anything specifically about the experience that you might have wanted to share as feedback for us to improve on over time.

As mentioned by @joeschmuck there's probably several users who will need a moment to wipe the drool off after reviewing the specs of a system with a pair of Xeon Golds and 48TB of NVMe flash - as you mentioned, it's quiet, powerful, relatively inexpensive compared to something like an R740 - what's not to like? I believe it might also have the option for headless boot if you really wanted to make it into a server-like chassis; but since you plan to run some extra VM workload through the host hypervisor (Proxmox) you could install a power GPU and put it to use in containers or VMs.

Perhaps it was I being defensive - apologies.

I do believe there is some oob mgmt, but I find a Displayport/USB3 KVM switch sufficient.

I should clarify that the primary use of TrueNAS is media. I'll probably run VM's here and there, but I'm as comfortable with a single machine running all (most?) of the things I need in linux userland as I would be using VM's or containers for separation of concerns.

I know the NVMe is overkill - I could probably get by w/4x 4tb u.2 NVME's for ~$200 each - and might be better off. We'll see how it goes.

 

[joeschmuck]

but I've not been happy with TrueNaS scale as a VM server.

This is why I prefer to use ESXi as my hypervisor and run TrueNAS as a VM. It was especially true about 10 years ago when I started, we only had jails back then. So I completely understand your choice.

 

[Junx]

I have a Dell T7920 with dual Xeon Gold 6148's & 256gb of RAM. I've installed three quad NVME m.2 PCIE cards (Dell PN:6N9RH), and intend to run 12 x 4tb m.2 (undecided on specific model) and pass the controller cards through proxmox to a VM running TrueNAS Core. It has a few onboard interfaces and an older x520 based 10 gig adapter, which I'll likely dedicate to the TrueNAS box as well.

Use is primarily media storage - im leaning towards 12 x 4tb NVME because I believe they will put off less heat than a similar # of SAS or SATA drives, plus the much smaller footprint.

Summary:
40 cores, 80 threads
12 x 4tb m.2 NVME
256gb memory (can easily dedicate half or more to TrueNAS VM)

Feedback appreciated - my current TrueNAS Scale box is a Dell power edge T420 dual Xeon E5-2470V2, 192gb of ram and 8 x 4tb 7200 SAS drives w/x520 10g

As an update: I've ordered 4x 8tb Intel P4510 u.3 NVME's for my primary TrueNAS storage. I'll be populating the m.2 NVME's out over time, and will likely go with 2TB HP FX900 Pro, which are rated at 1200TBW. I haven't seen much talk about them on this forum - does anyone have any feedback?

 

[Davvo]

As an update: I've ordered 4x 8tb Intel P4510 u.3 NVME's for my primary TrueNAS storage. I'll be populating the m.2 NVME's out over time, and will likely go with 2TB HP FX900 Pro, which are rated at 1200TBW. I haven't seen much talk about them on this forum - does anyone have any feedback?

As far as I am concerned, them having 2GB of dram each means you should consider an UPS.

 

[Junx]

As far as I am concerned, them having 2GB of dram each means you should consider an UPS.

Good call - and I do use an APC for this box.

 

[joeschmuck]

2TB HP FX900 Pro, which are rated at 1200TBW

Toms Hardware rated it pretty good. The cost off of Amazon is $120 USD, or @250 USD for 4TB so the pricing sounds good to me. Hum, makes me think that I could ditch the spinning rust drives with four 4TB M.2 sticks, less than my first set of drives. Eh, it's a dream for now, my drives still have some life left in them.