Which Hardware for a 50TB Storage Server?

[LogiX]

Hi,
Do you have any experience of which hardware configuration is suitable for a storage server with 50TB?

It is 1 each HDD, SSD, Nvme storage pools

I would also like to use Ecc Ram. Plus a 10 or 40gbe network card.

Thanks for your help.

 

[Arwen]

Other than looking at the hardware recommended guide, we don't generally recommend complete servers. That is because lots of factors come in to play, like power consumption, size, noise and usage purpose.

Hardware Recommendations Guide

This is the latest edition of the FreeNAS Community hardware recommendations guide. The current major version is R2, dated January 2021, with the last minor update on 2021-01-24. The format has moved away from the forum post form factor, to...

www.truenas.com

The 10Gbit/ps networking primer below may help. But to actually get high network performance usually requires beefier CPU, memory and storage.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Based on your forum post count and when you joined the forum, we have to guess you may not have much experience with ZFS. So I suggest some reading about what ZFS is, can do and can not do;

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Picking a ZFS Pool Layout to Optimize Performance

Part 1 of the blog post is here, covering basics, striped pools, and mirrored vdevs: https://www.ixsystems.com/blog/zfs-pool-performance-1/ Part 2 covers RAID-Z and some example workload scenarios...

www.truenas.com

If, after reading the hardware recommendations guide and others, and come up with possible server hardware list, you can post it here for us to review.

 

[LogiX]

I thought about this Configuration:

CPU: Ryzen 9 3900x
RAM: 128GB ECC
Mainboard: GIGABYTE B550 AORUS Elite AX V2 (Rev. 1.1)
PSU: Seasonic Prime TX
HBA: LSI 9201-16i

Pool1:

vDev1:
5x WD HDDs 3TB Raidz2
vDev2:
5x WD HDDs 3TB Raidz2

Pool2:
vDev1:
3x Crucial BX500 2TB
vDev2:
3x Crucial BX500 2TB

Pool3:
vDev1:
2x Samsung Pro 990 2TB Raidz0

 

[Arwen]

Good start.

In general, you don't want a gaming board. Some things are less than suitable for a NAS, like audio or more than a few USB ports. (TrueNAS & ZFS don't work well with USB attached disks for data pools, though boot pool can work on USB.) Plus, on occasion gaming boards default to over-clocking CPU, Memory or PCIe ports, which can be problematic for a server on 24x365. But, if you have it already, or like Gigabyte boards, it may work fine for you.

You use the term "Raidz0", which does not exist in ZFS. Perhaps you meant Striped vDev. But, you could have just as easily meant Mirror vDev. Thus, I suggested reading the "Terminology and Abbreviations Primer".

Which TrueNAS line were you thinking of using, Core, (based on FreeBSD), or SCALE, (based on Linux), which also supports Apps.

Now if you can describe your use cases for each pool.
Then what are you going to use for your boot disk / pool?

Boot devices in general don't need to be very large. For example 16GByte to 32GByte are perfectly fine. But, if you use a SATA SSD in a ZFS Mirror, and use TrueNAS SCALE, then having 64GByte or 128GByte can be useful to have all the Linux swap on the boot devices.

 

[Etorix]

Specific issue with the gamer board: It has several x2/x1 PCIe slots, which are useless in a server and only 4 SATA ports, which is not enough for your 6 SATA SSDs. How are you going to fit a HBA and a SFP+ (or QSFP+) NIC in there?

LSI 9200 are PCIe cards; they are fine for HDDs, but become a bottleneck for SSDs so you'd want to put the BX550 on chipset ports and keep the 9200 for HDDs only. Or get a 9300.

Without knowing your use case, it's not possible to comment on the layout. For bulk storage (HDDs), a single 10-wide raidz2 may be possible. (Or less drives, but bigger.) I suppose that the 2*2 TB NVMe pool is for apps and/or VMs. But what's the purpose of the SATA SSD pool? For VMs, you'd want a stripe of three 2-way mirrors rather than a stripe of two 3-wide raidz1.

Incidentally, I suspect that @Patrick M. Hausen (and others) may want to know why (oh why???) you're posting in the subforum for "FreeNAS", which is for legacy releases—that's even written in the subforum name. What's the thought process for going there rather than in either "TrueNAS CORE" or "TrueNAS SCALE", which are the current branches?

 

[LogiX]

I thought about TrueNas Scale

Pool1: all data where i dont need very high speed.
Pool2: Network boot server/VMs
Pool3: Steam cache

Wich Mainboard would you suggest?
Or should i use another CPU? If yes which one do u suggest?

For Pool2 I am between Raidz1 and 2
But I think to use Enterprise SSDs (Kingstone DC600M). Then I would use an Raidz1.

So sorry that i posted my Thread in the wrong category. Can I or someone else move this Thread in the right Category?

 

[Constantin]

Do not build separate pools unless one requires SSDs, the other one HDDs, mirrors vs. Z. You just lose speed and/or capacity. If the underlying hardware stays the same, build one pool then set up separate datasets if you like.

For example, a 2 VDEV Z2 pool will perform better than two separate Z2 pools with a single VDEV each. If you need to build a VM pool, go SSD but then you’d also want something smaller and more write-tolerant.

 

[LogiX]

Do not build separate pools unless one requires SSDs, the other one HDDs, mirrors vs. Z. You just lose speed and/or capacity. If the underlying hardware stays the same, build one pool then set up separate datasets if you like.

For example, a 2 VDEV Z2 pool will perform better than two separate Z2 pools with a single VDEV each. If you need to build a VM pool, go SSD but then you’d also want something smaller and more write-tolerant.

Do you mean i should do this in this configuration?

Pool1:

vDev1:
5x WD HDDs 3TB Raidz2
vDev2:
5x WD HDDs 3TB Raidz2

Pool2:
vDev1:
3x Crucial BX500 2TB
vDev2:
3x Crucial BX500 2TB

Pool3:
vDev1:
2x Samsung Pro 990 2TB Striped vDev

 

[Etorix]

Pool1: all data where i dont need very high speed.

So you may consider a wider raidz2, at the expense of flexibility to expand.

Pool2: Network boot server/VMs

For VMs, go for mirrors. No raidz.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Pool3: Steam cache

No idea what that is, but "cache" in ZFS likely does not work the way you think…

Wich Mainboard would you suggest?

Anything with at least 6-8 SATA ports to begin with. And preferably ECC.
Keeping your Ryzen CPU, an AsRockRack X470D4U, X570D4U or B550D4U.
But it depends how many cores and how much RAM you need for the VMs—which, at some point, could tilt the choice towards a platform with RDIMM.

So sorry that i posted my Thread in the wrong category. Can I or someone else move this Thread in the right Category?

No need to apologise. But an explanation of the process would be nice. Any reason to chose this category? Or was it just "the first place to come up", because the "TrueNAS SCALE" section comes further down in the list? We're curious, and looking for ways to improve the forum.

 

[Saoshen]

Do not build separate pools

I disagree.

Having multiple pools helps keep pools smaller, faster to rebuild, easier to upgrade, and most of all, all your ISO's are not in the same basket (pool).

If a pool goes belly up, due to drives or other hardware issue, other pools should be unaffected (assuming the hardware issue doesn't apply to all drives).

it also makes it much easier to export when needed (less drives to interact with/prevent damage to).

Most people in home media/nas/homelab usage, I would bet do not need huge single pools. Particularly media servers and similar, do not need single large pools, because media can be played from any number of source pools/datasets/shares.

I have 4 primary pools, and 2 temporary pools in a 24 bay das enclosure, general/movies/tv/audio + overflows.

That certainly doesn't mean that there are not use cases for giant hundreds of tb pools, but most people don't need them.

TLDR;
single giant pools = putting all your eggs in one basket. one pool disaster and all your eggs are gone.

 

[Constantin]

TLDR;
single giant pools = putting all your eggs in one basket. one pool disaster and all your eggs are gone.

The resources, etc. here advocate for backups for a reason.

All things being equal, a bigger pool with multiple VDEVs is inherently more prone to a breakdown than a smaller pool by virtue of cumulative probability. At the same time, a pool with multiple VDEVs is far more performant re: IOPS and the user has the option to closely monitor the health of the pool (SMART tests, scrubs, etc. in addition to temp-monitoring, snapshots, backups, and so on) to prevent a pool loss. Large pools do not as a matter of normal operation go belly up all the time, otherwise no one would use them. HDDs and SSDs, if of the right quality and used normally, are fairly reliable.

However, and this is the beauty of TrueNAS, you can configure your pools however you want. If smaller, less-performant pools make you sleep better at night, then that is the way to go for you. In general, I don't see the benefit of using multiple pools in parallel vs. using one large pool w/multiple VDEVs as long as the underlying hardware and disk configuration per VDEV remains the same. But then again, my pool is relatively small and I'm OK with only having one VDEV.

 

[Constantin]

Do you mean i should do this in this configuration?

Pool1:

vDev1:
5x WD HDDs 3TB Raidz2
vDev2:
5x WD HDDs 3TB Raidz2

Pool2:
vDev1:
3x Crucial BX500 2TB
vDev2:
3x Crucial BX500 2TB

Pool3:
vDev1:
2x Samsung Pro 990 2TB Striped vDev

Given how reliable SSDs are in general (if OK quality), I'd consider a 2x2 SSD mirror pool instead, as long as you periodically back up the SSD pool. The speed of that pool will be plenty fast that way and the likelihood of total pool loss quite low (~2x lower than my laptop, for example, since it only has one SSD in it). But I'd get some more opinions. Depends too how important the SSD data is. For very-important stuff, a 3-way mirror is appropriate.