Please Comment my first TrueNAS Build

[HamStrangler]

Hi everyone, I've been reading here for a while. I've read through the hardware guide, and a few threads.

Now I have started to put together my first TrueNAS system. I put togehter some gaming PCs, but never before a server.
In the beginning I plan to use the TrueNAS server mainly as a Jellyfin media server. Over time, one or the other application will certainly be added. That's why I opted for a system with more power, and a lot room for improvement if necessary.

The server should be capable of transcoding because I want to use my Jellyfin outside of my network. Most films have a 1080 HD resolution, only a few films have a 4K resolution.

I would be happy if one or the other could write me his opinion on this and what can be done better if necessary. The server I put together has the following hardware:

• chassis: CSE-836E26-R1200B
• Backplane: BPN-SAS2-836EL1 / SAS bis 6Gb/s, SATA bis 6Gb/s
• Mainboard: Supermicro x10dri-LN4+
• RAM: 64GB ECC
• CPU: 2x CPU Xeon E5-2640 v3
• HBA: LSI 9271-8i
• Nvme Expasnion Card: AOC-SLG3-2M
• two 970 EVO Plus NVMe™ M.2 SSD - 250 GB
• Quadro P400

I plan to install TrueNAS on the 2 mirrored Nvme. I will start with 5 to 6 12TB HDDs in an RAIDZ 2. I am not sure if the backplane is the right one for 12GB HDDs.

I am looking forward to your comments and suggestions.

 

[DigitalMinimalist]

Hi,

looks good, but...

HBA: LSI 9271-8i - It's based on 2208 chip. It can't be flashed to IT mode. You need to get either 2008 or 2308 based cards. On the other hand: why not use the Mainboard SATA connectors? - HBA is nice if you need more SATA
OS: use the 970 Evo for VMs (mirrored) and use a simple SATA SSD >64GB for OS
NIC: I would consider a 10GBE PCIe NIC, but can be done later

 

[jgreco]

HBA: LSI 9271-8i

See

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

I am not sure if the backplane is the right one for 12GB HDDs.

Should be fine.

Backplane: BPN-SAS2-836EL1 / SAS bis 6Gb/s, SATA bis 6Gb/s

On the other hand: why not use the Mainboard SATA connectors? - HBA is nice if you need more SATA

Probably because the mainboard SATA connectors are SATA. You need SAS to run a backplane containing an SAS expander chip like the 836EL1.

RAM: 64GB ECC

chassis: CSE-836E26-R1200B

This feels imbalanced. We typically recommend shooting for 1GB of RAM per 1TB HDD, and a 36 bay chassis filled with 12TB drives is 432TB of space, so I'd think you'd be looking to be able to support north of 128GB of RAM. If you're not filling all the bays immediately, of course you can do it later, but do be aware that 64GB is not going to be particularly performant as you expand the system (because presumably that's why you've got a 36 bay unit).

Quadro P400

Make sure you get one that includes the half height bracket, which you will need in that chassis. Since this is an older card, purchases on eBay are often pulls out of workstations which may not come with the bracket.

• Nvme Expasnion Card: AOC-SLG3-2M
• two 970 EVO Plus NVMe™ M.2 SSD - 250 GB

I plan to install TrueNAS on the 2 mirrored Nvme.

This may not fail over in the way you might hope for, so it may not offer redundant booting capability. If that's a goal, write back and I can discuss this a bit.

 

[HamStrangler]

Thank you very much for your answers and suggestions.

HBA: LSI 9271-8i - It's based on 2208 chip. It can't be flashed to IT mode. You need to get either 2008 or 2308 based cards.

In this case I will look for an other HBA with the right chip. Thanks for the notice.

This feels imbalanced. We typically recommend shooting for 1GB of RAM per 1TB HDD, and a 36 bay chassis filled with 12TB drives is 432TB of space, so I'd think you'd be looking to be able to support north of 128GB of RAM. If you're not filling all the bays immediately, of course you can do it later, but do be aware that 64GB is not going to be particularly performant as you expand the system (because presumably that's why you've got a 36 bay unit).

You are right, in this case I will go for 128GB of RAM.

Make sure you get one that includes the half height bracket, which you will need in that chassis. Since this is an older card, purchases on eBay are often pulls out of workstations which may not come with the bracket.

Thanks for the hint. I definitely wouldn't have noticed until after I bought it.

This may not fail over in the way you might hope for, so it may not offer redundant booting capability. If that's a goal, write back and I can discuss this a bit.

You are right, I was aiming for redundant booting capability. Is the redundancy not given because both of the Nvme are placed on the same Card, so when the card fails, both Nvme are failing?

 

[jgreco]

You are right, I was aiming for redundant booting capability. Is the redundancy not given because both of the Nvme are placed on the same Card, so when the card fails, both Nvme are failing?

Well, maybe. That could happen, but typically a dual NVMe card is just a PCB with a bunch of traces and two sockets on it, no silicon involved. Hard for them to fail. Not impossible.

The concern is what is actually likely to fail. And that will be the SSD's themselves. So picture this. What happens when your BIOS is happily decided that NVMe #1 is the device to boot from, and it's reading sectors, and it's running that code, and suddenly it is unable to read some sectors? It halts, putting up some ugly message about "Unable to read". It has no idea that the same data is available on NVMe #2.

Even if NVMe #1 fails completely, there's no guarantee that the BIOS will pick up and start booting from NVMe #2 (because it cannot see #1) because of how these things get wired into the BIOS configuration. You may be able to tell it to "prefer NVMe #1" for boot, but when that's not there, is it going to know to try NVMe #2? Or maybe will it just try HDD #1?

So the trick here is that you really need a way to guarantee coverage of these cases, and that really means a RAID controller. As I frequently soapbox, RAID controllers are bad for ZFS, but there is actually an exception, and this is it. RAID controllers can guarantee bootability as long as either side of a RAID1 mirror is readable, because they are actually designed for that sort of thing.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

 

[Davvo]

I suggest you going SCALE instead of CORE if you want to use plugins, even if it's newer and less stable.

 

[HamStrangler]

Well, maybe. That could happen, but typically a dual NVMe card is just a PCB with a bunch of traces and two sockets on it, no silicon involved. Hard for them to fail. Not impossible.

The concern is what is actually likely to fail. And that will be the SSD's themselves. So picture this. What happens when your BIOS is happily decided that NVMe #1 is the device to boot from, and it's reading sectors, and it's running that code, and suddenly it is unable to read some sectors? It halts, putting up some ugly message about "Unable to read". It has no idea that the same data is available on NVMe #2.

Even if NVMe #1 fails completely, there's no guarantee that the BIOS will pick up and start booting from NVMe #2 (because it cannot see #1) because of how these things get wired into the BIOS configuration. You may be able to tell it to "prefer NVMe #1" for boot, but when that's not there, is it going to know to try NVMe #2? Or maybe will it just try HDD #1?

So the trick here is that you really need a way to guarantee coverage of these cases, and that really means a RAID controller. As I frequently soapbox, RAID controllers are bad for ZFS, but there is actually an exception, and this is it. RAID controllers can guarantee bootability as long as either side of a RAID1 mirror is readable, because they are actually designed for that sort of thing.

What you are saying makes sense, I didn't think about that when planning the server. Thanks a lot for explaining.
I guess I will just went with a single SSD as boot device then. The redundant booting capability is nice to have, but not necessary.

 

[HamStrangler]

I suggest you going SCALE instead of CORE if you want to use plugins, even if it's newer and less stable.

That is the plan, going for scale, because of the plugins.

 

[jgreco]

What you are saying makes sense, I didn't think about that when planning the server. Thanks a lot for explaining.
I guess I will just went with a single SSD as boot device then. The redundant booting capability is nice to have, but not necessary.

Not a problem. I enjoy the educational aspect and passing this stuff on.