CPU bottlenecking write speeds

[arcadeperfect]

Hello

System is Xeon-D with 10g networking.

My write speeds are capping at about 600 MB/s, reads at about 900 MB/s with 4 sata ssd's in RaidZ1.

As a test I created a "striped" pool on a single NVME drive with no compression. The speeds are the same, and during writes the CPU is maxing out on a single thread. Could that be the bottleneck? Since there's no parity to calculate, what's the CPU doing?

Thanks

 

[ChrisRJ]

Please specify your hardware (see forum rules for level of detail that will get you good answers) and how you conducted the test (machines, protocol, file size and number, number of iterations, etc.)

 

[arcadeperfect]

Hardware details:

Motherboard: Supermicro X10SDV-4C-TLN2F
CPU: Xeon D-1521, 4 core / 8 thread @ 2.4 Ghz
Ram: 32 GB

Pools:
4 x WD Red 2tb SATA ssd connected directly to motherboard SATA ports, in RaidZ1
1 x Samsung NVME "striped" (just for this test)

Networking:
Truenas: 10Gbase-T built into SOC, with rj45 -> sfp+ transceiver on the switch end
Worksation: Mellanox ConnectX-3 Ethernet Adapter 10G SFP+
Switch: Mikrotik CRS305-1G-4S+IN

All shares are SMB

Methodology:

Crystal Diskmark set to 64GB file sizes (double the system ram) running from Windows workstation
Manually copying ~100GB movie files to and from the shares
"diskinfo -t /dev/nvd0" to get a rough idea of how fast the ssd is interfacing internally

The results are pretty much identical on the single NVME and the RaidZ pool, and consistent. ~900 MB/s read, 620 MB/s write (with maxed out cpu thread).

I am testing sequential reads / writes of large video files to test throughput, I understand that for other workloads the data is less relevant. One use-case for this NAS is to serve uncompressed video for VFX work to 2 - 3 users so the scenario is somewhat real worldy.

Thanks

 

[ChrisRJ]

Is it SMB that is maxing out the CPU?

 

[LarsR]

As far as i can recall, smb and crystal diskmark are single threaded, and if you test it with a mapped network drive that is your limiting factor.
SMB also benefits from higher core clocks, so your 2.4Ghz could also be a limiting factor.

 

[arcadeperfect]

Just to be clear it was the freenas CPU that was maxing out, although there is an active thread on the the workstation when writing to the nas. Workstation is a 5950X doing around 4.3 Ghz.

Tried some different protocols -

iSCSI was actually worse with a lot of CPU activity on the client (I forgot to write down the speeds)

ZFS with client booted into Mint had slightly better writes at ~720 MB/s using Kdiskmark and maual file copying, but slightly worse reads at ~620MB/s

Both reads and writes slower on ZFS when accessing the RaidZ as opposed to the single NVME at ~510MB/s read and 600MB/s write

In the case of ZFS the server CPU showed one thread fluctuating around 60 - 80%, SMB 90 - 100%. The process was "kernal".

I guess I'm just at the limits of the platform. If I get around to it I might boot the nas box into linux and serve the NVME with ext4 and see what happens.

 

[mav@]

It would help to see exact output of `top -SHIz`, showing separate threads that are CPU-bottlenecked.

 

[arcadeperfect]

Hmm

I tried that and actually it doesn't appear that the CPU is bottlenecking after all with the SMB thread sitting around 60%

Iperf verified the network can push about 9Gbe in either direction and that's what I get with reads.


I do see a burst of ~9Gb/s writes at the start of the transfer, which I am assuming is Truenas writing to ram. But could it actually be the SLC cache on the NVME drive in the nas?

 

[SweetAndLow]

Hmm

I tried that and actually it doesn't appear that the CPU is bottlenecking after all with the SMB thread sitting around 60%

Iperf verified the network can push about 9Gbe in either direction and that's what I get with reads.


I do see a burst of ~9Gb/s writes at the start of the transfer, which I am assuming is Truenas writing to ram. But could it actually be the SLC cache on the NVME drive in the nas?

do a dd test on your truenas system. You need to use a dataset that has compression disabled but then do something like

Write test: dd if=/dev/zero of=/mnt/zpool/dataset/file bs=1M count=10000
Read test: dd if=/mnt/zpool/dataset/file of=/dev/null bs=1M count=10000

this will test the throughput on your pool and give you a baseline of max you can get. The smb single thread thing is nonsense, ignore that.