SOLVED Incompatible SMB and NFS speed

[hasore]

I am getting speeds incompatible to what should be expected when reading from ARC, that would be ~1.2 GB/s.

- Isn't a issue related to disk performance, data is being read from the ARC.
- Client and server are connected using DAC cables.
- MTU in client and server is set to 9014

Client details:
Client: Fedora 35 Workstation ( Ryzen 9 3950x, 64 GB RAM )
Client NIC: Intel X520-DA1
Client Driver:

Show : ixgbe 5.13.4

[root@workstation ~]# modinfo ixgbe
filename: /lib/modules/5.16.9-200.fc35.x86_64/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version: 5.13.4
license: GPL
description: Intel(R) 10GbE PCI Express Linux Network Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: 78F4EFD306374B79FBDA641
alias: pci:v00008086d000015E5sv*sd*bc*sc*i*
alias: pci:v00008086d000015E4sv*sd*bc*sc*i*
alias: pci:v00008086d000015CEsv*sd*bc*sc*i*
alias: pci:v00008086d000015CCsv*sd*bc*sc*i*
alias: pci:v00008086d000015CAsv*sd*bc*sc*i*
alias: pci:v00008086d000015C8sv*sd*bc*sc*i*
alias: pci:v00008086d000015C7sv*sd*bc*sc*i*
alias: pci:v00008086d000015C6sv*sd*bc*sc*i*
alias: pci:v00008086d000015C4sv*sd*bc*sc*i*
alias: pci:v00008086d000015C3sv*sd*bc*sc*i*
alias: pci:v00008086d000015C2sv*sd*bc*sc*i*
alias: pci:v00008086d000015AEsv*sd*bc*sc*i*
alias: pci:v00008086d000015ADsv*sd*bc*sc*i*
alias: pci:v00008086d000015ACsv*sd*bc*sc*i*
alias: pci:v00008086d000015ABsv*sd*bc*sc*i*
alias: pci:v00008086d000015B0sv*sd*bc*sc*i*
alias: pci:v00008086d000015AAsv*sd*bc*sc*i*
alias: pci:v00008086d000015D1sv*sd*bc*sc*i*
alias: pci:v00008086d00001563sv*sd*bc*sc*i*
alias: pci:v00008086d00001560sv*sd*bc*sc*i*
alias: pci:v00008086d00001558sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Asv*sd*bc*sc*i*
alias: pci:v00008086d00001557sv*sd*bc*sc*i*
alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias: pci:v00008086d00001528sv*sd*bc*sc*i*
alias: pci:v00008086d000010F8sv*sd*bc*sc*i*
alias: pci:v00008086d0000151Csv*sd*bc*sc*i*
alias: pci:v00008086d00001529sv*sd*bc*sc*i*
alias: pci:v00008086d0000152Asv*sd*bc*sc*i*
alias: pci:v00008086d000010F9sv*sd*bc*sc*i*
alias: pci:v00008086d00001514sv*sd*bc*sc*i*
alias: pci:v00008086d00001507sv*sd*bc*sc*i*
alias: pci:v00008086d000010FBsv*sd*bc*sc*i*
alias: pci:v00008086d00001517sv*sd*bc*sc*i*
alias: pci:v00008086d000010FCsv*sd*bc*sc*i*
alias: pci:v00008086d000010F7sv*sd*bc*sc*i*
alias: pci:v00008086d00001508sv*sd*bc*sc*i*
alias: pci:v00008086d000010DBsv*sd*bc*sc*i*
alias: pci:v00008086d000010F4sv*sd*bc*sc*i*
alias: pci:v00008086d000010E1sv*sd*bc*sc*i*
alias: pci:v00008086d000010F1sv*sd*bc*sc*i*
alias: pci:v00008086d000010ECsv*sd*bc*sc*i*
alias: pci:v00008086d000010DDsv*sd*bc*sc*i*
alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias: pci:v00008086d000010C8sv*sd*bc*sc*i*
alias: pci:v00008086d000010C7sv*sd*bc*sc*i*
alias: pci:v00008086d000010C6sv*sd*bc*sc*i*
alias: pci:v00008086d000010B6sv*sd*bc*sc*i*
depends: dca
retpoline: Y
name: ixgbe
vermagic: 5.16.9-200.fc35.x86_64 SMP preempt mod_unload
parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)
parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)
parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm: FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of int)
parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)
parm: LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
parm: dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)
parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

Tunables:

Show : sysctl

net.core.somaxconn = 2048
net.core.netdev_max_backlog = 250000

net.core.wmem_default = 524288
net.core.wmem_max = 16777216
net.core.rmem_default = 524288
net.core.rmem_max = 16777216
net.core.optmem_max = 16777216

net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 1
#net.ipv4.tcp_adv_win_scale = 1
net.ipv4.ip_no_pmtu_disc = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_rmem = 524288 4194304 16777216
net.ipv4.tcp_wmem = 524288 4194304 16777216
net.ipv4.tcp_mem = 524288 4194304 16777216
net.ipv4.udp_mem = 524288 4194304 16777216

Server details:
Server: TrueNAS 12.0 U8 ( Intel G5400, 32 GB ECC RAM )
Server NIC: Intel X520-DA1
Tunables:

Show : testparm -s:

[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
netbios name = TRUENAS
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server role = standalone server
server string = TrueNAS Server
unix extensions = No
username map = /usr/local/etc/smbusername.map
username map cache time = 60
idmap config *: range = 90000001-100000000
fruit:nfs_aces = No
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes

Show : Loader

cc_htcp_load: YES
hw.ix.flow_control: 1
hw.ix.max_interrupt_rate: 16000
kern.random.fortuna.minpoolsize: 256
net.inet.tcp.hostcache.cachelimit: 0
net.inet.tcp.soreceive_stream: 0
net.isr.bindthreads: 1
net.isr.maxthreads: -1
net.pf.source_nodes_hashsize: 131072

Show : sysctl

kern.ipc.maxsockbuf: 16777216
kern.ipc.soacceptqueue: 2048
kern.random.sys.harvest.ethernet: 0
kern.random.sys.harvest.random: 0
net.inet.tcp.abc_l_var: 44
net.inet.tcp.cc.abe: 1
net.inet.tcp.cc.algorithm: htcp
net.inet.tcp.cc.htcp.rtt_scaling: 1
net.inet.tcp.delayed_ack: 0
net.inet.tcp.htcp.adaptive_backoff: 1
net.inet.tcp.initcwnd_segments: 44
net.inet.tcp.minmss: 536
net.inet.tcp.mssdflt: 1460
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.recvbuf_inc: 524288
net.inet.tcp.recvbuf_max: 16777216
net.inet.tcp.recvspace: 4194304
net.inet.tcp.rfc6675_pipe: 1
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.sendbuf_inc: 524288
net.inet.tcp.sendbuf_max: 16777216
net.inet.tcp.sendspace: 4194304
net.inet.tcp.syncache.rexmtlimit: 0
net.inet.tcp.syncookies: 0
net.inet.udp.recvspace: 4194304
vfs.aio.max_aio_procs: 128
vfs.aio.max_aio_queue: 8192
vfs.aio.max_aio_queue_per_proc: 32
vfs.aio.target_aio_procs: 64

Tests:

Show : iperf3 - client -> server

[hasore@workstation ~]$ iperf3 -c server
Connecting to host server, port 5201
[ 5] local client port 44682 connected to server port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 9.97 Gbits/sec 0 2.60 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.91 Gbits/sec 0 2.60 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.92 Gbits/sec 0 2.60 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.91 Gbits/sec 0 2.60 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 9.92 Gbits/sec 0 2.60 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 9.91 Gbits/sec 0 2.60 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.91 Gbits/sec 0 3.00 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 3.00 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 9.92 Gbits/sec 0 3.00 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.91 Gbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.92 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.91 Gbits/sec receiver

Show : iperf3 - server -> client

[hasore@workstation ~]$ iperf3 -c server -R
Connecting to host server, port 5201
Reverse mode, remote host server is sending
[ 5] local client port 44686 connected to server port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.91 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.91 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.92 Gbits/sec 1 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.91 Gbits/sec receiver

Show : writing to disk (special vdev) using server

root@storage[/]# dd if=/dev/zero of=/mnt/frostfire/disabled/tmp.dat bs=2M count=10k status=progress
21109932032 bytes (21 GB, 20 GiB) transferred 21.016s, 1004 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 21.754994 secs (987 MB/s)

Show : client - smb - reading from ARC

[root@workstation ~]# dd if=/dev/zero of=/mnt/disabled_smb/tmp.dat bs=2M count=10k status=progress
21422407680 bytes (21 GB, 20 GiB) copied, 25 s, 857 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 44.1511 s, 486 MB/s
[root@workstation ~]# dd if=/dev/zero of=/mnt/disabled_smb/tmp.dat bs=2M count=10k status=progress
21023948800 bytes (21 GB, 20 GiB) copied, 24 s, 876 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 44.3122 s, 485 MB/s
[root@workstation ~]# dd if=/dev/zero of=/mnt/disabled_smb/tmp.dat bs=2M count=10k status=progress
21464350720 bytes (21 GB, 20 GiB) copied, 25 s, 858 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 44.4327 s, 483 MB/s
[root@workstation ~]# dd if=/dev/zero of=/mnt/disabled_smb/tmp.dat bs=2M count=10k status=progress
21070086144 bytes (21 GB, 20 GiB) copied, 24 s, 878 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 44.4912 s, 483 MB/s

Show : client - nfs - reading from ARC

[root@workstation ~]# sysctl -w vm.drop_caches=3 && dd of=/dev/null if=/mnt/disabled_nfs/tmp.dat bs=2M count=10k status=progress
vm.drop_caches = 3
20891828224 bytes (21 GB, 19 GiB) copied, 27 s, 774 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 27.7523 s, 774 MB/s
[root@workstation ~]# sysctl -w vm.drop_caches=3 && dd of=/dev/null if=/mnt/disabled_nfs/tmp.dat bs=2M count=10k status=progress
vm.drop_caches = 3
20753416192 bytes (21 GB, 19 GiB) copied, 27 s, 769 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 27.9419 s, 769 MB/s
[root@workstation ~]# sysctl -w vm.drop_caches=3 && dd of=/dev/null if=/mnt/disabled_nfs/tmp.dat bs=2M count=10k status=progress
vm.drop_caches = 3
21135097856 bytes (21 GB, 20 GiB) copied, 27 s, 783 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 27.4277 s, 783 MB/s
[root@workstation ~]# sysctl -w vm.drop_caches=3 && dd of=/dev/null if=/mnt/disabled_nfs/tmp.dat bs=2M count=10k status=progress
vm.drop_caches = 3
21000880128 bytes (21 GB, 20 GiB) copied, 27 s, 778 MB/s
10240+0 records in
10240+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 27.615 s, 778 MB/s

Show : ping

[hasore@workstation ~]$ ping -c100 ***
PING *** (***) 56(84) bytes of data.
64 bytes from ***: icmp_seq=1 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=2 ttl=64 time=0.223 ms
64 bytes from ***: icmp_seq=3 ttl=64 time=0.219 ms
64 bytes from ***: icmp_seq=4 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=5 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=6 ttl=64 time=0.219 ms
64 bytes from ***: icmp_seq=7 ttl=64 time=0.223 ms
64 bytes from ***: icmp_seq=8 ttl=64 time=0.222 ms
64 bytes from ***: icmp_seq=9 ttl=64 time=0.208 ms
64 bytes from ***: icmp_seq=10 ttl=64 time=0.224 ms
64 bytes from ***: icmp_seq=11 ttl=64 time=0.217 ms
64 bytes from ***: icmp_seq=12 ttl=64 time=0.224 ms
64 bytes from ***: icmp_seq=13 ttl=64 time=0.217 ms
64 bytes from ***: icmp_seq=14 ttl=64 time=0.237 ms
64 bytes from ***: icmp_seq=15 ttl=64 time=0.227 ms
64 bytes from ***: icmp_seq=16 ttl=64 time=0.222 ms
64 bytes from ***: icmp_seq=17 ttl=64 time=0.236 ms
64 bytes from ***: icmp_seq=18 ttl=64 time=0.240 ms
64 bytes from ***: icmp_seq=19 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=20 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=21 ttl=64 time=0.074 ms
64 bytes from ***: icmp_seq=22 ttl=64 time=0.205 ms
64 bytes from ***: icmp_seq=23 ttl=64 time=0.216 ms
64 bytes from ***: icmp_seq=24 ttl=64 time=0.224 ms
64 bytes from ***: icmp_seq=25 ttl=64 time=0.219 ms
64 bytes from ***: icmp_seq=26 ttl=64 time=0.203 ms
64 bytes from ***: icmp_seq=27 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=28 ttl=64 time=0.225 ms
64 bytes from ***: icmp_seq=29 ttl=64 time=0.214 ms
64 bytes from ***: icmp_seq=30 ttl=64 time=0.219 ms
64 bytes from ***: icmp_seq=31 ttl=64 time=0.223 ms
64 bytes from ***: icmp_seq=32 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=33 ttl=64 time=0.222 ms
64 bytes from ***: icmp_seq=34 ttl=64 time=0.221 ms
64 bytes from ***: icmp_seq=35 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=36 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=37 ttl=64 time=0.222 ms
64 bytes from ***: icmp_seq=38 ttl=64 time=0.227 ms
64 bytes from ***: icmp_seq=39 ttl=64 time=0.217 ms
64 bytes from ***: icmp_seq=40 ttl=64 time=0.237 ms
64 bytes from ***: icmp_seq=41 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=42 ttl=64 time=0.229 ms
64 bytes from ***: icmp_seq=43 ttl=64 time=0.224 ms
64 bytes from ***: icmp_seq=44 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=45 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=46 ttl=64 time=0.239 ms
64 bytes from ***: icmp_seq=47 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=48 ttl=64 time=0.228 ms
64 bytes from ***: icmp_seq=49 ttl=64 time=0.226 ms
64 bytes from ***: icmp_seq=50 ttl=64 time=0.234 ms
64 bytes from ***: icmp_seq=51 ttl=64 time=0.226 ms
64 bytes from ***: icmp_seq=52 ttl=64 time=0.221 ms
64 bytes from ***: icmp_seq=53 ttl=64 time=0.222 ms
64 bytes from ***: icmp_seq=54 ttl=64 time=0.242 ms
64 bytes from ***: icmp_seq=55 ttl=64 time=0.212 ms
64 bytes from ***: icmp_seq=56 ttl=64 time=0.227 ms
64 bytes from ***: icmp_seq=57 ttl=64 time=0.236 ms
64 bytes from ***: icmp_seq=58 ttl=64 time=0.233 ms
64 bytes from ***: icmp_seq=59 ttl=64 time=0.237 ms
64 bytes from ***: icmp_seq=60 ttl=64 time=0.239 ms
64 bytes from ***: icmp_seq=61 ttl=64 time=0.239 ms
64 bytes from ***: icmp_seq=62 ttl=64 time=0.250 ms
64 bytes from ***: icmp_seq=63 ttl=64 time=0.238 ms
64 bytes from ***: icmp_seq=64 ttl=64 time=0.224 ms
64 bytes from ***: icmp_seq=65 ttl=64 time=0.215 ms
64 bytes from ***: icmp_seq=66 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=67 ttl=64 time=0.235 ms
64 bytes from ***: icmp_seq=68 ttl=64 time=0.242 ms
64 bytes from ***: icmp_seq=69 ttl=64 time=0.238 ms
64 bytes from ***: icmp_seq=70 ttl=64 time=0.225 ms
64 bytes from ***: icmp_seq=71 ttl=64 time=0.232 ms
64 bytes from ***: icmp_seq=72 ttl=64 time=0.459 ms
64 bytes from ***: icmp_seq=73 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=74 ttl=64 time=0.230 ms
64 bytes from ***: icmp_seq=75 ttl=64 time=0.239 ms
64 bytes from ***: icmp_seq=76 ttl=64 time=0.227 ms
64 bytes from ***: icmp_seq=77 ttl=64 time=0.228 ms
64 bytes from ***: icmp_seq=78 ttl=64 time=0.214 ms
64 bytes from ***: icmp_seq=79 ttl=64 time=0.210 ms
64 bytes from ***: icmp_seq=80 ttl=64 time=0.219 ms
64 bytes from ***: icmp_seq=81 ttl=64 time=0.218 ms
64 bytes from ***: icmp_seq=82 ttl=64 time=0.231 ms
64 bytes from ***: icmp_seq=83 ttl=64 time=0.547 ms
64 bytes from ***: icmp_seq=84 ttl=64 time=0.220 ms
64 bytes from ***: icmp_seq=85 ttl=64 time=0.232 ms
64 bytes from ***: icmp_seq=86 ttl=64 time=0.239 ms
64 bytes from ***: icmp_seq=87 ttl=64 time=0.241 ms
64 bytes from ***: icmp_seq=88 ttl=64 time=0.240 ms
64 bytes from ***: icmp_seq=89 ttl=64 time=0.238 ms
64 bytes from ***: icmp_seq=90 ttl=64 time=0.213 ms
64 bytes from ***: icmp_seq=91 ttl=64 time=0.204 ms
64 bytes from ***: icmp_seq=92 ttl=64 time=0.237 ms
64 bytes from ***: icmp_seq=93 ttl=64 time=0.227 ms
64 bytes from ***: icmp_seq=94 ttl=64 time=0.178 ms
64 bytes from ***: icmp_seq=95 ttl=64 time=0.237 ms
64 bytes from ***: icmp_seq=96 ttl=64 time=0.248 ms
64 bytes from ***: icmp_seq=97 ttl=64 time=0.236 ms
64 bytes from ***: icmp_seq=98 ttl=64 time=0.235 ms
64 bytes from ***: icmp_seq=99 ttl=64 time=0.226 ms
64 bytes from ***: icmp_seq=100 ttl=64 time=0.222 ms

--- *** ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 101398ms
rtt min/avg/max/mdev = 0.074/0.230/0.547/0.043 ms

 

[sretalla]

dd if=/dev/zero

Here's one problem with your tests... that's highly compressible data and won't be measuring anything in terms of your pool speed.

Also, what makes you think that dd command is reading from ARC? (only on the second and subsequent tasks)

You should re-start your testing after doing a little reading up on how to use fio.

 

[hasore]

Here's one problem with your tests... that's highly compressible data and won't be measuring anything in terms of your pool speed.

Also, what makes you think that dd command is reading from ARC? (only on the second and subsequent tasks)

You should re-start your testing after doing a little reading up on how to use fio.

- I forgot to share that detail, the test dataset /mnt/frostfire/disabled has compression and sync disabled.
- About the disks, i did reboot before either SMB or NFS tests, read the files into ARC and the tests were done while monitoring using gstat -I 1s -p

---

These are the mount options i am using on /etc/fstab:

Code:

# Tests
//server/disabled /mnt/disabled_smb cifs credentials=/root/.samba/gaming.txt,vers=3.1.1,uid=1000,gid=1000,rw 0 0

 

[hasore]

Managed to solve this issue by changing smb from 3.1.1 to 3.0/3.0.2

Code:

# Tests
//server/disabled /mnt/disabled_smb cifs credentials=/root/.samba/gaming.txt,vers=3.0.2,uid=1000,gid=1000,rw 0 0

Probably some tuning need to be done in either Fedora (cifs) or TrueNAS (smbd) in order to properly use smb 3.1.1.

compression = lz4, sync = standard:

Show : smb 3.1.1

Show : smb 3.0.2

Show : smb 3.0

Show : smb 2.1

Show : smb 2.0

compression = lz4, sync = disabled:

Show : smb 3.0.2

compression = off, sync = standard:

Show : smb 3.0.2

compression = off, sync = disabled:

Show : smb 3.0.2

mtu 1500, compression = lz4, sync = disabled:

Show : smb 3.0.2 - mtu 1500

 

[anodos]

Regarding SMB 3.11 vs 3.02, perform mount from your client then check smbstatus in both cases to ensure that signing / encryption for sessions is the same. There are differences between supported features for the protocols.

 

[hasore]

Regarding SMB 3.11 vs 3.02, perform mount from your client then check smbstatus in both cases to ensure that signing / encryption for sessions is the same. There are differences between supported features for the protocols.

Did some tests using smbstatus, the perfomance "issue" was caused due packet signing, the mount was using either ntlmv2i or ntlmsspi before the tests done with smb 3.0 or 3.0.2.

Show : smb3.1.1, ntlmssp (default)