Slow setup - what improvements can be made?

[overshoot]

Hello,

I've just migrated my TrueNAS setup from the core version running on a Dell Poweredge R320 to the scale version now running on a Dell Poweredge R430.
I am experiencing very poor performance, to the point that I am considering switching back.
The R320 setup as always been slow but not to that extent (waiting seconds just to display the content of a folder or previewing small JPGs)
That is when accessing for the first time a folder or a file. Once already browsed, it gets quite "ok".

I understand that the TrueNAS core is more optimized than the "scale" and that it's not surprising to have lower performance using the later.
But my goal is to find out if there is anything that can be done to make the TrueNAS scale in my case usable as it's not currently.

Dell R320 config:
- Intel Xeon E5-2420 v2 (6 cores 2.2Ghz)
- 96GB of RAM
- 8 x 4TB Seagate ST4000LM024 (2.5*)
- Dell Perc H310 controller in HBA mode
- Proxmox host with TrueNAS core guest

Dell R430 config:
- Intel Xeon E5-2630 v3 (8 cores 2.4Ghz)
- 128GB of RAM
- 8 x 4TB Seagate ST4000LM024 (2.5*) *** moved over drives
- LSI MegaRAID SAS-3 3108 in HBA mode
- TrueNAS scale baremetal

I have 2 VMs that I need to run in addition to the TrueNAS server:
- 1 Ubuntu for my Unifi devices + recording CCTV footages
- 1 Windows that I start/stop on demand, so not running constantly

I have often had emails from the system mentioning some drives were causing slow I/O on pool Main.
So I know my drives are slow but budget wise, I couldn't do any better at the time.
It's a small office setup for 3 Mac (macOS Big Sur) users + 1 Windows 8 user.

Also, I understand that SMBD is maxing out 1 CPU core and because of the low frequency of that CPU, there is a bottleneck.

My RaidZ2 is used at 50% (10TB/20TB).

My thoughts so far:
- braking my RaidZ2 pool in 2 pools
- Adding a SLOG device

Let me know if I am lacking any informations in my description please.

 

[LarsR]

If those are Seagate Baracuda drives they're SMR and not CMR drives and tend to give that slow i/o error message. ZFS does not like smr drives.
Send them back and buy seagate ironwolf, ironwolf pro or exos drives.

 

[overshoot]

These are SMR drives based on that information: https://www.seagate.com/as/en/internal-hard-drives/cmr-smr-list/
2.5" Seagate drives are SMR only.

And these drives are out of warranty so that's not an option.

 

[morganL]

These are SMR drives based on that information: https://www.seagate.com/as/en/internal-hard-drives/cmr-smr-list/
2.5" Seagate drives are SMR only.

And these drives are out of warranty so that's not an option.

Running slow drives in mirror mode can improve their performance.

 

[overshoot]

so

Running slow drives in mirror mode can improve their performance.

So basically, to get faster performance I would have to lose half of the storage capacity if I hear you well?!

By the way, checking the system logs, I am getting these messages:

[ 1071.027401] sd 0:0:1:0: [sdb] tag#789 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 1071.029178] sd 0:0:1:0: [sdb] tag#789 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00

That is happening on every drive.
I recall having these messages at some point on Proxmox but I can't recall if I just had to discard them or they meant something (like my HBA card was misconfigured).

 

[morganL]

so

So basically, to get faster performance I would have to lose half of the storage capacity if I hear you well?!

By the way, checking the system logs, I am getting these messages:

[ 1071.027401] sd 0:0:1:0: [sdb] tag#789 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 1071.029178] sd 0:0:1:0: [sdb] tag#789 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00

That is happening on every drive.
I recall having these messages at some point on Proxmox but I can't recall if I just had to discard them or they meant something (like my HBA card was misconfigured).

ATA command pass-thru could be related to SMART tests.. worthwhile checking.

The suggestion of mirroring is because SMR drives are very poor at small writes..... they are designed for PC use and very low I/O rates.
By moving to mirroring, the number of I/Os is dropped significantly.
There is no way of making the drives spin faster or changing them to CMR.. so you do have few options that i can see.

 

[overshoot]

I don't know what I smoke that day but when I answered LarsR, but I was sure SMR was the right drive type to have for TrueNAS.
Oh boy...
Anyhow, I guess I'll need to deal with the fact that I have slow drive.
I already did change some settings like Atime and Sync to off and that did speed up things significantly.
Thanks for your help guys!

 

[mervincm]

In my experience it is helpful to separate your loads into random and sequential categories. Mixing them leaves both tasks starving for disk. One group of disks with raid choices that favour IOPS, put your OS disks, your container /app folders there. another group of disks with RAID choices that favour sequential read/write rates, put your media, installation media, archives, backup versions there. Its even better if you can purchase the right disks for the right category as well (SSD for IOPS) but it any case it is still a good idea.