Slow performance on rebuild

[Siege801]

Hi Everyone,

Sorry, I know the title is generic, but there are enough performance quirks going on that I don't know how to be more specific.

Thanks in advance for reading.

I'm looking for any advice available. The purpose of this rig is to serve CalDAV via Nextcloud to a handful of users; Occasional Nextcloud browsing; and Media sharing via NFS to a Plex server (hosted on another device).

Summary list of issues:

1. Poor transfer / write speeds
2. Unable to mount with nfs (possibly performance related?)
3. GUI unable to save changes

Here is my rig:

• CPU: 6 CPUs x Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz
• RAM: 32Gb RAM
• Motherboard: ASRock Z390 Pro4 ATX LGA1151
• Software: TrueNAS-12.0-U4

Hard Drives:

• 6 x 2Tb Seagate ATA Disks running in RaidZ2 (~8Gb usable)
  • POOL_BigData
    • MEDIA Dataset
    • NCDATA Dataset
    • iocage Dataset (currently with only Nextcloud installed into it)
• 2 x 16Gb USB3 boot-pool. N.B. The original plan was to run TrueNAS from a 500Gb NVM drive, but I found out all too late that the motherboard disables one of the SATA ports if I use one of the NVM connectors. I figured a striped USB install wouldn't be too much of an issue. Please advise if I'm wrong as there seem to be differing opinions out there.

Initially, I had this configured within a VMWare / ESXi configuration, but for data reliability reasons I've done a rebuild to a baremetal TrueNAS installation. While configured under ESXi, the TrueNAS VM had only 16Gb RAM assigned, with 4x2Tb SATAs at RaidZ1. Under this configuration, the read/write/mount performance was acceptable. Somehow, now, having installed TrueNAS directly to the box (no more ESXi here), the performance has bombed out:


Issue #1: Transfer speeds

I'm in the process of trying to copy data back to the TrueNAS machine. I currently have one computer uploading data via rsync/ssh. Each file is transferring at an average of 60.0MB/s, which I think is reasonable? However, if I spin up a second computer to upload data, computer A continues as is, while computer B can't even upload a single file - rsync tells me the transfer starts at about the same (60.0Mb/s), but before it's even uploaded more than a few Kbs, it drops down to a rate of around 100Kb/s and basically freezes. I would have thought, at worst, both transfers would share out the resources and level out with an even split of transfer speed?

Issue #2: NFS Mount woes

Secondly - I can't seem to mount to the TrueNAS box using NFS. From one Debian-based laptop, I have the following fstab entry:

Code:

192.168.1.x:/mnt/POOL_BigData/MEDIA /media/user/nas_media nfs rw,noatime,hard,intr,rsize=131072,wsize=131072,timeo=150,retrans=3 0

Then, running:

Code:

$ sudo mount /media/user/nas_media

results in:

Code:

mount.nfs: requested NFS version or transport protocol is not supported

Previously the error after a long pause was:

Code:

NFS mount error: mount.nfs mount system call failed

The change between the errors was either a TrueNAS reboot, or attempting to switch between NFSv3 and NFSv4. I say attempting, because when I go into Services -> NFS and toggle the NFS version, hitting Save then just gives me a blue spinning wheel of "Please wait" death (Issue #3 if you will).

NFS Configuration

Services -> NFS: Enabled
Number of servers: This originally was set to 4 - I've also increased to 6 as per output from sysctl -n kern.smp.cpus
NFSv4 - Enabled: I have also tried disabled - see above on issue with toggling


Relevant Usage Statistics:

Memory:
31.9GiB total available

Free: 1.5 GiB
ZFS Cache: 20.3 GiB
Services: 9.8 GiB

Physical over an hour:
Min; Mean; Max

• Wired: 0; 28.05GiB; 28.95GiB
• Inactive: 20.31GiB; 28.35GiB; 29.11GiB
• Laundry: 1.36GiB; 1.39GiB; 1.61GiB
• Active:1.24GiB; 1.26GiB; 1.49GiB
• Free: 704.3MiB; 1.43GiB; 9.34GiB

Swap Utilization over an hour:
Min; Mean; Max

• Used: 0; 260.51MiB; 289.37MiB
• Free: 4GiB; 4GiB; 4GiB

CPU:
Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz
24% Avg Usage

• Threads: 7 threads
• Highest Usage: $ (Thread #) - doesn't seem to be populating - under Reporting - it shows the following values:
  • Interrupt: 0.09 Mean; 0.38 Max
  • System: 4.98 Mean; 16.36 Max
  • User: 8.67 Mean; 21.85 Max
  • Nice: 8.67 Mean; 21.85 Max
  • Idle: 99.45 Mean; 100 Max
• Hottest:doesn't seem to be populating - under Reporting - it's sitting at:
  • Mean of 93F

Network (Min; Mean; Max)

• Octets Rx: 4.61Kb; 1.9Mb; 136.68Mb
• Octets Tx: 55.32Kb; 188.95Kb; 3.08Mb
• Overlap:4.61Kb; 102.44Kb; 3.08Mb

Disk I/O
da0 and da1 - the boot-pool over an hour period:

Min; Mean; Max (both drives had very similar values)

• Read: 0; 7.89KiB; 395.88KiB
• Write: 0; 367.96; 30.81KiB

ada0, 1, 2, 3, 4 and 5 - the data pool over an hour period:

Min; Mean; Max (all six drives had very similar values)

• Read: 8.81KiB; 13.71MiB; 38.01 MiB
• Write: 167.66KiB; 19.6MiB; 94.97MiB

I look forward to any advice / observations or requests for further information.

Kind Regards.

 

[HoneyBadger]

6 x 2Tb Seagate ATA Disks running in RaidZ2

Before we dig too deeply, I want to rule out those drives as a potential pain point - please post exact model numbers eg: ST2000DM008

Compare against the list of known SMR drives maintained by @Yorick here:

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

and/or use the script created by @Basil Hendroff here:

Scripted Detection of SMR Drives

SMR technology is a timebomb waiting to go off in a ZFS file system. This quote from Slashdot... At absolute best, SMR disks underperform significantly in comparison to CMR disks; at their worst, they can fall flat on their face so badly that...

www.truenas.com

 

[Siege801]

Hi There,

Thanks for the reply - and good shot.

Seagate - 3.5"

ST2000DM008

2TB Barracuda

DM-SMR

(2)

These are my drives... So... what does this mean? TrueNAS and my drives will never co-exist well?

 

[ChrisRJ]

Yes, you will need to get other disks, unfortunately

 

[QonoS]

You should be aware that Nextcloud runs a database. HDDs are too slow by some orders of magnitude to serve as storage for a database. (And those SMR drives esspecially.) Databases by default do writes with synchronous I/O and that is definitely a job for SSDs, or better enterprise SSDs.

Bargain: Buy 2 Intel Optane 16GB NVME drives. In used condition these are cheap to get and they last for an eternity. Create a mirrored pool of them and put the database there. It's amazingly performant.

 

[ChrisRJ]

@QonoS , I would add that your statement applies more to heavily utilized database servers in an enterprise context. Yes an HDD is not ideal for Nextcloud, but it should also not be a serious problem in a private context.

 

[ChrisRJ]

@Siege801 , does your board have a Realtek NIC by any chance? Those are known to cause loads of problems on FreeBSD and the recommendation is to buy an Intel server NIC instead.