djjudas21
Cadet
- Joined
- Oct 21, 2022
- Messages
- 5
Long story short, I replaced an old TrueNAS SCALE 22.x system with 4xHDDs with a TrueNAS SCALE 23.10 system with 6xSSDs. It's a budget system for home use and while it doesn't need blazing performance (I've only got 1Gbit networking), the new system seems to give worse performance than the old one. Both my old and new TrueNAS systems exist primarily to provide persistent volumes to a 5-node Kubernetes cluster, using the Democratic-CSI operator. My shares are all NFS.
When copying large files to/from the old NAS, it would easily saturate the 1Gbit network and sustain 70-80MB/s reliably. The new NAS starts off at ~70MB/s but after a while it drops down 2-3MB/s and continues at this speed. Browsing NFS shares, listing directories, etc, is slow.
My suspicion is that it's not related to the network or the NFS protocol, but something to do with the storage configuration. I don't know how to prove this. Can anyone suggest a way of benchmarking, testing for bottlenecks, etc?
Is it better to spread the disks across as many controllers as possible, or as few as possible? The PCIe bandwidth is much faster than the SATA bandwidth so it shouldn't be a bottleneck in any case.
I know Fanxiang is not the best brand of SATA SSDs, but they get decent reviews and I've also used a set of them in a Ceph cluster, where they performed fine.
System: Custom build in Silverstone CS280 chassis
Motherboard: Asrock Rack C2750D4I
CPU: Intel® Avoton C2750 Octa-Core Processor
Memory: 32GB DDR3
Storage controller: Intel® C2750 (2xSATA3, 4xSATA2), Marvell SE9172 (2xSATA3), Marvell SE9230 (4xSATA3)
Boot device: Intel 120GB SSD on SATA2 port in C2750 controller
Storage devices: 6 x Fanxiang S101 2TB SSD in RAIDZ2, on SATA3 ports across all 3 controllers
Network controller: Intel® i210 Gigabit LAN port (also tried a a PCIe NIC to rule this out)
System: HP MicroServer N54L
Motherboard:
CPU: AMD Turion II Neo N54L
Memory: 16GB DDR3
Storage controller: AMD RS785E
Boot device: 16GB USB stick in internal USB2 port
Storage devices: 4 x WD Red 4TB in RAIDZ2
Network controller: NC107i
When copying large files to/from the old NAS, it would easily saturate the 1Gbit network and sustain 70-80MB/s reliably. The new NAS starts off at ~70MB/s but after a while it drops down 2-3MB/s and continues at this speed. Browsing NFS shares, listing directories, etc, is slow.
What I've tested so far
- Can't recreate slow behaviour on old NAS
- Slow behaviour observed when transferring between the new NAS and other LAN devices over NFS
- Have tried configuring network with i210 as a single NIC, as a pair of bonded NICs, and a PCIe NIC. No change.
- CPU not maxed out, but does show more iowait than I would expect during light use. The graph below shows CPU usage while I copied a few hundred MB of JPGs in small batches
My suspicion is that it's not related to the network or the NFS protocol, but something to do with the storage configuration. I don't know how to prove this. Can anyone suggest a way of benchmarking, testing for bottlenecks, etc?
Is it better to spread the disks across as many controllers as possible, or as few as possible? The PCIe bandwidth is much faster than the SATA bandwidth so it shouldn't be a bottleneck in any case.
I know Fanxiang is not the best brand of SATA SSDs, but they get decent reviews and I've also used a set of them in a Ceph cluster, where they performed fine.
New NAS
OS: TrueNAS-SCALE-23.10.0.1System: Custom build in Silverstone CS280 chassis
Motherboard: Asrock Rack C2750D4I
CPU: Intel® Avoton C2750 Octa-Core Processor
Memory: 32GB DDR3
Storage controller: Intel® C2750 (2xSATA3, 4xSATA2), Marvell SE9172 (2xSATA3), Marvell SE9230 (4xSATA3)
Boot device: Intel 120GB SSD on SATA2 port in C2750 controller
Storage devices: 6 x Fanxiang S101 2TB SSD in RAIDZ2, on SATA3 ports across all 3 controllers
Network controller: Intel® i210 Gigabit LAN port (also tried a a PCIe NIC to rule this out)
Old NAS
TrueNAS: TrueNAS-SCALE-22.xSystem: HP MicroServer N54L
Motherboard:
CPU: AMD Turion II Neo N54L
Memory: 16GB DDR3
Storage controller: AMD RS785E
Boot device: 16GB USB stick in internal USB2 port
Storage devices: 4 x WD Red 4TB in RAIDZ2
Network controller: NC107i
