SCALE higher CPU usage than CORE?

[flashdrive]

yes, SCALE is still beta...

I am running the nightly build of Sep 9th.

Out of the box, no tuning

A single SMB share which is receiving Data and 10 GBit LAN (around 400 MByte/s write speed).

The data is residing on a raidz1, encrypted. The CPU has AES hardware support.

However the CPU with 4 cores will get hit frequently up to 75% x 4 Cores, with 1 core being at 100%.

This was not true for TN CORE, where the CPU load was very low in the same scenario /hardware.

 

[morganL]

However the CPU with 4 cores will get hit frequently up to 75% x 4 Cores, with 1 core being at 100%.

This was not true for TN CORE, where the CPU load was very low in the same scenario /hardware.

Linux and FreeBSD are very different. For example the scheduler for using multiple cores is very different.

https://www.usenix.org/system/files/conference/atc18/atc18-bouron.pdf

Its only a bug, if it performs badly... which doesn't seem to be the case here.

 

[flashdrive]

@morganL

Thank you for your info.

How are you sure regarding this statement?

Its only a bug, if it performs badly... which doesn't seem to be the case here.

 

[morganL]

@morganL
How are you sure regardings this statement?

My response was a little flippant.. the point is that CORE and SCALE will show different measurements. However, if SCALE underperforms CORE significantly, we should treat that as a bug and check why.

 

[yottabit]

I have noticed a much higher idle load average than CORE. For me, it seems to be caused by k3s running around 15-25% thread time constantly, even when no containers are installed and/or active.

 

[morganL]

I have noticed a much higher idle load average than CORE. For me, it seems to be caused by k3s running around 15-25% thread time constantly, even when no containers are installed and/or active.

Not unexpected... most "virtualization" technologies have some CPU overhead.

 

[yottabit]

Yes, perhaps, but I think there are a couple specific points in my reference to k3s that are unexpected:

1. I agree orchestration overhead of k3s -> docker -> lxc -> userland would be higher than iocage (but should be minimal... this is lxc/container after all, not hypervisor)
2. When no apps/containers are active (or even instantiated), I expect the k3s orchestrator should be either disabled, or virtually zero load
3. When apps/containers are deployed, but inactive, I expect the k3s orchestrator should be virtually zero load
4. When apps/containers are active, I have no particular expectation for the k3s orchestrator load, except that I would hope it's reasonable

 

[anodos]

However the CPU with 4 cores will get hit frequently up to 75% x 4 Cores, with 1 core being at 100%.[/code]
What process is/are consuming CPU? smbd?

 

[flashdrive]

@anodos
How may I found this out? Via "top"?

 

[flashdrive]

@morganL

What should I have a special look into to ascertain if this performance is bug-worthy?

 

[yottabit]

Yes, top is a good place to start. But as to whether something is bug worthy, it's unknown at this time.

 

[flashdrive]

Ok, I took a look into top - however cannot post the values since this bug is not letting me boot the system:

https://www.truenas.com/community/t...rt-zfs-pools-on-scale-21-06.94234/post-659426

I did see two other entries besides smbd with some CPU usage. Let me try later on to report the top results.

 

[flashdrive]

here are some top shell results (write to NAS using SMB, encrypted and unencrypted datasets)

1) encrypted

top - 14:12:32 up 7 min, 1 user, load average: 1.87, 1.30, 0.70
Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 41.8 sy, 0.0 ni, 38.8 id, 15.0 wa, 0.0 hi, 2.8 si, 0.0 st
MiB Mem : 31770.5 total, 12106.4 free, 19502.7 used, 161.3 buff/cache
MiB Swap: 4095.9 total, 4095.9 free, 0.0 used. 11920.9 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1870 root 1 -19 0 0 0 S 33.7 0.0 0:18.45 z_wr_iss
7779 user 20 0 60512 25412 12824 S 24.7 0.1 0:24.89 smbd
7952 root 20 0 0 0 0 R 19.3 0.0 0:18.13 io_wqe_worker-0
1872 root 0 -20 0 0 0 S 13.3 0.0 0:09.77 z_wr_int
284 root 0 -20 0 0 0 S 7.7 0.0 0:02.60 spl_kmem_cache
290 root 39 19 0 0 0 S 6.0 0.0 0:05.37 dbuf_evict
283 root 0 -20 0 0 0 S 1.0 0.0 0:00.82 spl_dynamic_tas
2 root 20 0 0 0 0 S 0.7 0.0 0:01.00 kthreadd
1903 root 39 19 0 0 0 S 0.7 0.0 0:00.16 dp_sync_taskq
2331 root 20 0 419736 188964 24516 S 0.7 0.6 0:03.52 middlewared (wo
12 root 20 0 0 0 0 S 0.3 0.0 0:00.19 ksoftirqd/0


2) unencrypted

top - 14:12:32 up 7 min, 1 user, load average: 1.87, 1.30, 0.70
Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 41.8 sy, 0.0 ni, 38.8 id, 15.0 wa, 0.0 hi, 2.8 si, 0.0 st
top - 14:14:09 up 8 min, 1 user, load average: 2.25, 1.62, 0.88
Tasks: 235 total, 6 running, 229 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.0 us, 50.1 sy, 0.0 ni, 31.6 id, 8.8 wa, 0.0 hi, 5.6 si, 0.0 st
MiB Mem : 31770.5 total, 23429.4 free, 8179.2 used, 161.8 buff/cache
MiB Swap: 4095.9 total, 4095.9 free, 0.0 used. 23244.2 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7779 user 20 0 59488 24372 12840 R 53.9 0.1 0:39.41 smbd
7952 root 20 0 0 0 0 R 44.1 0.0 0:28.63 io_wqe_worker-0
1870 root 1 -19 0 0 0 S 28.3 0.0 0:29.41 z_wr_iss
1872 root 0 -20 0 0 0 S 15.8 0.0 0:15.61 z_wr_int
284 root 0 -20 0 0 0 S 12.2 0.0 0:04.26 spl_kmem_cache
290 root 39 19 0 0 0 R 5.6 0.0 0:08.73 dbuf_evict
626 root 20 0 2712588 261908 26004 S 3.9 0.8 0:25.84 asyncio_loop
4461 root 20 0 419644 191824 27532 S 1.3 0.6 0:03.77 middlewared (wo
2 root 20 0 0 0 0 S 1.0 0.0 0:01.60 kthreadd

 

[yottabit]

Especially since your encrypted and unencrypted loads are virtually identical, CPU speed is not a factor. It sure looks like you're simply limited by the speed of your disks here.

 

[flashdrive]

hm, than I do not understand the CPU load being displayed in the dashboard where at least 1 thread is maxed out at 100 %

 

[yottabit]

Your ps output doesn't show a single thread occupying 100%. The dashboard is showing aggregate of all threads working in parallel. Remember that ZFS threads will also be occupying cputime and generally look poor of they're blocked waiting on I/O to or from the disks. That's where the sys and iowait metrics from top come into play, and you'll see these are fairly high.

Also, which model of CPU are you using? Your load average is 1.87, so pretty much using 2 full threads. Depending on your CPU model and numa configuration, that might also be a contributing bottleneck.

 

[flashdrive]

I am using as per Signature:

• Intel Core i5-4460 @ 3.20GHz with AES-NI support

 

[yottabit]

Ah cool, thanks. I don't see your signature on mobile.

Assuming you only have one of these in your system, that numa config is 1x4x1. Therefore I would expect your system load average could achieve up to 4.0 without much user impact, and could sustain even 6 to 8 with minimal impact, depending on exact workloads.

Considering you are achieving only 1.87, this means one of two things, or perhaps a combination of them, are at fault:

1. You are being constrained by low disk I/O throughput, which means your CPU cores are unable to be fully utilized because they're starved for input by the slow disks, or
2. SMB is your limiting factor since, unlike NFS, it is single-threaded and cannot take advantage of a multi-core CPU.

Based on your iowait and smb cpu%, I think it's the former causing your bottleneck. Your CPU is capable of achieving higher throughput but your disks are too slow. You need to either use faster disks, or a faster striping topology (e.g., mirror(s) instead of Z1, etc.), etc.

Your original post indicates you were able to achieve faster throughput on BSD. Is everything else in the environment exactly the same? E.g., the other host you're using to transfer data from?

And you indicate a thread running at 100%, but that is not what your top & ps output shows. It shows you're running at 187% aggregate load out of a nominal capability of 400% based on your CPU numa config, and not a single process is running at 100%.

As an ix'er said up-thread, the BSD and Linux kernel schedulers are dramatically different, too. So this also can account for some difference in the observable statistics.