looking for opinions on setup

[trex]

Hello all.

I am looking to get a new box built with the best read write speeds i can get with some fault tolerance.

15x 1TB Crucial MX500's

2x 1tb Samsung 970 pro nvme's

Dell R730xd 2x E5-2637 v3 3.5ghz with 64GB DDR4 H730 ( hoping it has a pass-through mode )

Use cases will be NFS / ISCSI on 10g network ports for Hypervisors for vm storage.

The goal is to have the best outcome for vm performance while also being able to have some fault tolerance on the drives.

I was thinking of going rz3 raid with ZLOG and Cache... but do i just slam them all in 1 group? i need a sanity check please!

 

[Ericloewe]

The MX500s have a nasty habit of reporting errors that disappear and reappear constantly. Makes for huge warning spam.

If you want the best read and write speeds, you will want as many vdevs as possible, so something like 7 mirrored pairs plus a hot spare.

I’m dubious you’d benefit from L2ARC and you don’t have any SSDs listed that would be adequate for an SLOG, so let’s go with neither of them for now.

Finally, the RAID controller… Get rid of it. Yes, it nominally has an HBA mode. No, you do not want it. Sell it, buy an HBA330 mini and spare yourself the trouble.

 

[trex]

well crap wish i would have known that before buying 16 of them... why wouldnt the nvme with gen 3 ports not work for anything? perhaps im not understanding how they will work one each doing totally different things. can you or someone explain that a bit more? lastly 7 mirrored pairs, is that two groups of 7 rz3? and just to be sure that would still give 3 drive failures?

 

[jgreco]

H730 ( hoping it has a pass-through mode )

It does. You won't want to use it. Read the following linked article.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Please do not argue with me that it has an "HBA mode".

rz3 raid

Nope. Read the following linked articles.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to translate terms on the reader's side when reading posts...

www.truenas.com

We don't know what "rz3 raid" is, but if you meant RAIDZ3, then move on to this article.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

ZLOG

What's that?

64GB DDR4

That's really pretty low for a production VM filer.

10g network ports

Try to make sure you pick a decent 10G card. We discuss the good ones over in this article:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

2x 1tb Samsung 970 pro nvme's

Cache

You probably want more like 512GB of RAM for that much L2ARC. No less than 256GB. But worth noting you probably wouldn't see much benefit since you can already retrieve content from SATA SSD at a really good clip.

15x 1TB Crucial MX500's

The MX500's are dodgy, and I've had weird issues with WD Blues, the other cheap name brand SSD option, as well. Samsung 870 Evo is worth the price IMO.

 

[trex]

noted on the hba, no one was arguing. it was what came with the "config'd" server. thats an easy swap.. :) ive just read the links you posted, this was very helpful. I do plan to use the intel x710.. basically im open for change so the HBA totally makes sense. but now this leaves me with 16 drives i may or not get a ton of spam from and two nvme's.. i can change out the drives eventually good news is that i have up to 24 slots plus 4 pcie 3.0 slots.. what would you do in this situation if you had to use this case? i currently have a need for 5TB of data not all of it is used for high iop related tasks... oh and i can always add more memory slots for days :)