Best performance for SATA disks

[curtishall]

I currently have 12 x 4TB SATA drives and 6 x SATA 6TB drives in 3 RAIDZ2 groups. I don't care so much about write speed but I do about read (these are 11,000+ video files between 800MB and 3GB) that need to be exported randomly.

Is this the "best" method of decent read speeds? I'm fixing to add another 6-8 drives so if there is a better suggestion I'm open.

The data is being synced (via zfs send) to another server offsite with the same storage capacity.

root@nas1[~]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 15.5G 6.01G 9.49G - - 2% 38% 1.00x ONLINE -
tank 87.2T 59.2T 28.0T - - 18% 67% 1.00x ONLINE /mnt
root@nas1[~]# zpool status tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 1 days 21:09:53 with 0 errors on Mon Apr 17 21:10:01 2023
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/3d66fa44-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
gptid/3e2acdaa-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
gptid/402d2899-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
gptid/40e71ffc-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
gptid/40d44ded-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
gptid/41b77291-4d53-11ea-b641-001ec93b000b ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/985735b2-43de-11ec-af19-d1b50866d222 ONLINE 0 0 0
gptid/42a5e0cd-4503-11ec-af19-d1b50866d222 ONLINE 0 0 0
gptid/cb9f607e-4644-11ec-af19-d1b50866d222 ONLINE 0 0 0
gptid/ec5b75b6-2236-11ec-8ee7-452968d79e2c ONLINE 0 0 0
gptid/65308723-4752-11ec-af19-d1b50866d222 ONLINE 0 0 0
gptid/50f4c4fa-310c-11ec-af19-d1b50866d222 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/ad3a77b2-254c-11ec-9821-210c850d60a2 ONLINE 0 0 0
da9p2 ONLINE 0 0 0
gptid/adbe68ef-254c-11ec-9821-210c850d60a2 ONLINE 0 0 0
gptid/ae8e158e-254c-11ec-9821-210c850d60a2 ONLINE 0 0 0
gptid/ae4c400d-254c-11ec-9821-210c850d60a2 ONLINE 0 0 0
gptid/ae9b66f9-254c-11ec-9821-210c850d60a2 ONLINE 0 0 0

errors: No known data errors
root@nas1[~]# zpool list tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 87.2T 59.2T 28.0T - - 18% 67% 1.00x ONLINE /mnt

 

[Johnny Fartpants]

How much RAM do you have? The more RAM the better when it comes to improving read performance. After that mirrors are always your best bet if it’s just about speed so long as you can live with the lack of resilience compared to that of Z2. If you need Z2 then the more vdevs you have the faster it will be.

 

[NugentS]

Whats the rest of the hardware? Please post as per forum rules

 

[curtishall]

Whats the rest of the hardware? Please post as per forum rules

Sorry!

The server is a E5-2650v2 w/ 128GB of memory and a SAS2308 controller connected to a NetApp DS4243. VMs are stored on a Proxmox server with a 10GB SFP+ connection. The VM's boot drive and initial storage is nvme / ssd. The TrueNAS pool is shared over NFS....in short videos are imported to SSD / NVME on the local VM server and migrated over to "slow" (SATA) storage via NFS.

I'm working on replacing the server with a dual Silver 4112 w/ 64GB of memory (I may try to upgrade to 128GB). This new server also has an unused NVME m.2 slot if you think using nvme would help in this case.

 

[NugentS]

You have a muliple vdev pool - and are planning on adding another vdev - so unless you want to move to mirrors (and I don't see the need). Also sequential reads from RAIDZ can be better than Mirrors. For video streaming / serving RAIDZ is probably optimal

In general terms to improve read add more memory and consider an L2ARC (more slow memory [but potentially faster than HDD] at the cost of some fast memory) if you have enough memory. Also in general tems L2ARC shouldn't exceed 5-10X the ARC.

I would put 128GB (or more but again how busy is it) in the new server and consider a 1TB L2ARC on the M.2. How much difference it makes depends really on how busy the server is and wether some videos are more popular than others. YMMV

In general you seem to be doing things right

The L2ARC would have two advantages.
1. Its additional cache for files
2. It caches metadata, meaning that (after a while) your directory listing type functions would be snappier - this may or may not have an effect on what you are doing

Its also not pool critical - so if it fails - it doesn't really matter

 

[curtishall]

Thank you! I appeciate your time.