Sudden drop in file transfer speed

[AndroidBot]

Ok, thank you @Arwen and @Davvo

For the 36x18TB disks that I got, do you think 3x12 is best? I want to get max performance out of my NAS, if I lose some available space to me, but I get better performance and better data security, I would go that route. So what do you suggest for my setup?

 

[Arwen]

In theory you can also use 4 vDevs of 9 disks in a RAID-Z2. That could potentially give better performance than 3 x 12 disk RAID-Z2. But, it does cost 2 more disks for redundancy;

3 x 12 disk RAID-Z2 = 6 disks of redundancy
4 x 9 disk RAID-Z2 = 8 disks of redundancy

Some people want the ultimate in performance, and some redundancy, so they might go for Mirrors;

18 x 2 disk Mirror vDevs = 18 disks of redundancy
12 x 3 disk Mirror vDevs = 24 disks of redundancy

We can give you choices and explain certain configuration's flaws, like 36 disks in a RAID-Z2 vDev is way too wide. In general, more vDevs is better performance.

But, in the end, unless you buy a prepackaged set of hardware and software, the choices are yours. You have to live with the results. Most of us can't guarantee what will work best for you. (Nor do we want to be responsible for choices.)

 

[Davvo]

Depends on the performance you need: if you are after streaming reads/writes (ie storing of large video files) this layout should give you the best performance for a single/few users; if you are after small file or random reads/writes (ie block storage) you want to go mirrors.

Since it looks you like you are the first type, you only needs to decide about how many VDEVs you want... more VDEVs means more IOPS (useful for multiple users), but also reduced space efficency and streaming performance.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

 

[AndroidBot]

Thanks for the info.
Basically, I have 18TB x 36 = 648TB theoretically, if I get around 300-350TB with maximum performance and relatively good enough redundancy and security for my data, I'm happy.

Based on this, what would you suggest? Priority number one is performance, my LAN is all 10GBe, so I would love to have 1-2GBps file transfer speeds over SMB if possible. Also I had cases of bad disks and replaced and went through resilvering process in TrueNAS, so would be nice to have proper security for my data as second priority. Size of available space is least of my concerns honestly.

 

[Davvo]

3x 12 drives in RAIDZ2 (or RAIDZ3) and 12x 3-way mirrors are both nice layouts to start testing.

I suggest you to experiment in order to see which is better for your use.

 

[AndroidBot]

Awesome, will do, thanks again

 

[AndroidBot]

@Davvo @Arwen
I'm just here to let you know that you were right.

I went with 6 vdevs and performance has been amazing. Now I think I'm pretty much limited to 10GBe ethernet speed. Basically SMB transfers does not drop below 1.1GB/s
Thank you very much for your help