SOLVED High Performance ZPool

[adityaharsh]

Trying to make a High Performance ZPool for my business server.
This Server will be accessed by 8-10 people always and will contain mostly documents and pdfs, images and spreadsheets. (In 100,000 + quantity)

So, my plan is:
2x 8TB Seagate Exos Drives (MIRROR)
2x WD RED SN700 250GB SSD (CACHE)
2x WD RED SN700 250GB SSD (SLOG)
2x WD RED SN700 250GB SSD (METADATA)

I want this ZPool to write at best speeds available, read and search files at the topmost speed.
Any suggestions, or changes that can help me achieve my goal?

My Configuration:
10Gbe Network
AMD Epyc 7542 (32C/64T)
128GB ECC RDIMM 3200MT/s

 

[NugentS]

So what wrong with this.
1. 2x 8TB Seagate Exos Drives (MIRROR) - Fine with that. Not much else you can do with 2 disks (thats sensible)
2. 2x WD RED SN700 250GB SSD (CACHE) - Only need 1 - its not pool critical. May not actually achieve anything though
3. 2x WD RED SN700 250GB SSD (SLOG) - Complete waste of time - won't do anything. Which you would know if you had done your research
4. 2x WD RED SN700 250GB SSD (METADATA) - Probably a good idea. Pool critical

An alternative might be ditch the special and slog and use the L2ARC as metadata only. Its not as if you are short of memory and the advantage would be that the disk(s) are not pool critical.

 

[kspare]

if you want it to write fast, max out your ram. then you can skip the Cache/L2Arc/metadata. Those drives for slog will be your next bottle neck. they won't be fast enough.

 

[adityaharsh]

if you want it to write fast, max out your ram. then you can skip the Cache/L2Arc/metadata. Those drives for slog will be your next bottle neck. they won't be fast enough.

Something else like Sabrent Rocket 4 Plus?

 

[adityaharsh]

An alternative might be ditch the special and slog and use the L2ARC as metadata only. Its not as if you are short of memory and the advantage would be that the disk(s) are not pool critical

By L2ARC if you're meaning the RAM then, I'll have to soon upgrade to 256GB as my TrueNAS Scale will have 3 windows VM too, each with 16GB RAM?

2x WD RED SN700 250GB SSD (METADATA) - Probably a good idea. Pool critical

I don't mind pool critical because I'll be having cold spares of the same drive readily available.
But to be exact how much performance can be expected out of this?

 

[NugentS]

And now you have changed your requirements by adding VM's
Can you please go back over your requirements and try again. Listing them all in one go

 

[Dice]

Hello and welcome!
The budget put into the first quote:

My Configuration:
10Gbe Network
AMD Epyc 7542 (32C/64T)
128GB ECC RDIMM 3200MT/s

Does not nearly line up with the budget put into this:

2x 8TB Seagate Exos Drives (MIRROR)
2x WD RED SN700 250GB SSD (CACHE)
2x WD RED SN700 250GB SSD (SLOG)
2x WD RED SN700 250GB SSD (METADATA)

And leaves a lot of performance on the table for this:

High Performance ZPool for my business server.

Beyond that, these two requirements are typically <vastly> different in how a a setup is planned in TN.

This Server will be accessed by 8-10 people always and will contain mostly documents and pdfs, images and spreadsheets. (In 100,000 + quantity)

TrueNAS Scale will have 3 windows VM too, each with 16GB RAM?

How much actual data do you have in terms of these pdfs, images, spreadsheets etc?

For your hardware;
Already as pointed out,
- forget about SLOG. Its function does not apply here.
- CACHE, a single drive would do, no need for redundancy. With as much RAM as you have, there is probably no use for the cache drive. It is only in very particular situations I'd see it sensible to build a system with a CACHE device without benchmarks/data indicating the system would benefit from a CACHE device, L2ARC.
Build the system without L2ARC from the beginning, and the have a look at ZFS/arc stats to see if there is any sense in using one. I'm convinced your system would not benefit even marginally. Until you mentioned VM's then things became slightly different again. The more RAM you have, the better off you are.

I do think a special vdev (metadata) might make more sense, to achieve a blazing fast browsing experience and offloading some of those small files from the Exos.

If you really want to push speeds, you could opt for nvme a few nvme drives instead of the exxos. Then you could remove the L2ARC, and metadata drives all together. I like looking at Samsung PM9A3.

 

[adityaharsh]

Until you mentioned VM's then things became slightly different again. The more RAM you have, the better off you are.

I mentioned VM because those VMs will take away RAM and so too much stuff on it worries me for power loss, even though I'd have a UPS.
VMs will be on a seperate 1tb NVMe mirror, and these data disks on seperate pool

 

[adityaharsh]

How much actual data do you have in terms of these pdfs, images, spreadsheets etc?

Just considering these, over 1.5tb in my current usage but this might increase soon.

 

[adityaharsh]

To make it all CLEAR this is what I'm trying to achieve with this Hardware on TrueNAS Scale:
1. 3 Windows VMs with Dedicated GPU running off of a MIRROR NVMe SSD (2x M.2)
2. 1 High Performance Pool to Store all documents only, like PDFs, word files, spreadsheets (Drive Layout can change as per suggestion)
3. 1 Massive RAIDZ2 Pool with 12 Disks (Dual Group of 6, with RAIDZ2) (SEAGATE EXOS 20TB DRIVES) (This will contain all sort of data like large video files, windows backup files, Mac time machine backup, etc.)

 

[Dice]

This is slightly "amusing" as this changes things again.

Here's my two cents for this scenario.

1. Mirror nvme: ok
2. I'd look for something along the lines of a Raidz1 with a Couple of SSDs. No L2ARC or SLOG or Metadata vdev required. Easily backed up to the large HDD pool, there is no need for higher resiliency. Some Z1 at least gives you protection and 'fixable' bit rot errors.
3. Here you could look at potentially making use of metadata vdev (mirror or tripple mirror SSDs). 6wide Z2, is geared towards "I want performance" out of this pool rather than mere space utilization. Maybe some 10 clients will really make the system bog during backups, but if it doesnt - 12 drives makes a good sense for a Z3, for storage.
I've not seen any data suggesting that time machine/other backups really benefit from L2ARC, but if they do - (research for real evidence based on arc_summary) that's something to look into further along the way.
Adding metadata vdevs needs to be there from the start as they will only "come in action" when there are files written in the system. L2ARC however, can be added and "populated" after each reboot, and thus does not make sense to hastely add for possibly actually worse performance due to improper setup.

 

[adityaharsh]

1. Mirror nvme: ok

Great. So, any suggestions for the 2 M.2 NVMe SSDs?

2. I'd look for something along the lines of a Raidz1 with a Couple of SSDs. No L2ARC or SLOG or Metadata vdev required.

Again, M.2 SSDs or Regular SATA SSD will suffice? Also Suggestions if any. (Going by your suggestion I'll either go for 3 disks or 4 disks in RAIDZ1)

6wide Z2, is geared towards "I want performance" out of this pool rather than mere space utilization. Maybe some 10 clients will really make the system bog during backups, but if it doesnt - 12 drives makes a good sense for a Z3, for storage.

So I'm implying that you're saying dual 6 disks will give more read write performance? And 12 disks at once is not feasible for the time being because even that would be perfect, I might be way out of my budget for the Year

 

[Dice]

I might be way out of my budget for the Year

Here we go again, what do you intend on getting <now> and <later>?
The upgrade paths make a difference.

1. no suggestions on particular models.
2. I'd go for m.2, and make use of that nice pcie bi-furcation, to put 2 or 4st drives on a single pcie slot.
If you are on budget constraints or happen to have access to a bunch of SATA SSDs, performance wise you'll be still fine, but of course the nvme's are a completely different beast.
In case you feel tight on budget, you could use mirrors for superior performance here too, rather than Z1. Mirrors are far less of a hassle to expand (see why upgrade paths make a difference...). Which sometimes makes more sense to loose RAW space, traded off for an easier upgrade path.

3. Again, how much are you going to get durgin the initial purchase? It makes all the damn difference.

This is slightly "amusing" as this changes things again.

The amusement was irony, in case that was not "CLEAR".

 

[adityaharsh]

Here we go again, what do you intend on getting <now> and <later>?
The upgrade paths make a difference.

1. no suggestions on particular models.
2. I'd go for m.2, and make use of that nice pcie bi-furcation, to put 2 or 4st drives on a single pcie slot.
If you are on budget constraints or happen to have access to a bunch of SATA SSDs, performance wise you'll be still fine, but of course the nvme's are a completely different beast.
In case you feel tight on budget, you could use mirrors for superior performance here too, rather than Z1. Mirrors are far less of a hassle to expand (see why upgrade paths make a difference...). Which sometimes makes more sense to loose RAW space, traded off for an easier upgrade path.

3. Again, how much are you going to get durgin the initial purchase? It makes all the damn difference.


The amusement was irony, in case that was not "CLEAR".

Thanks man for CLEARing that out

I have a newly purchased Samsung 980 Pro 1TB, so I guess I’ll buy another and make it the VM Pool in Mirror.

For the High Performance Pool I’ve several SATA SSDs lying around but I’ve a Asus Hyper x16 Gen 4 M.2 Card so I’ll buy 2 more m.2 SSDs for that with a future plan to add 2 more. Between Samsung 980 Pro and Sabrent Rocket 4 which one would you suggest?

As fas the High capacity pool is considered, i’ll buy 6 20tb drives now and add another slot when I reach around 50% capacity.

Thanks man, you really helped a lot

 

[Dice]

Between Samsung 980 Pro and Sabrent Rocket 4 which one would you suggest?

I'd go for the Samsung any day of the week.

i’ll buy 6 20tb drives now and add another slot when I reach around 50% capacity.

Ok, that's fine.

Thanks man, you really helped a lot

cheers!