sotiris.bos
Explorer
- Joined
- Jun 12, 2018
- Messages
- 56
Hi everyone,
I have a FreeNAS server that is not behaving as expected and I need some help with troubleshooting.
The FreeNAS hardware is:
Dell R730xd server, 2x E5-2630L v3 CPUs, 4x 16GB of 1866MHz DDR4 ECC RAM, Intel X550 rNDC 10Gb NIC and 2x Dell PM1725a NVMe SSDs in a mirror, running FreeNAS-11.3-U4.1.
Here is the pool configuration:
I ran tests on a dataset, switching between 4K and 128K recordsize to compare. My intention is to use the NVMe pool as VM storage, sharing it with NFS to two XCP-NG or Proxmox servers. Here are the dataset properties:
For testing, I used fio both on the NAS itself and then through NFS using a temporary Proxmox server and a temporary XCP-NG server on different hardware.
Proxmox server: Dell R620, 2x E5-2637 v2, 16x 2GB 1333MHz DDR3 RAM, Intel X550 rNDC 10Gb NIC
XCP-NG server: HP DL360e Gen8, 2x E5-2430L v2, 10x 8GB 1333MHz DDR3 RAM, Broadcom BCM57810 10Gb NIC
Switch: Buffalo BS-MP2008 10Gb switch
All machines are set with MTU 9000.
Here are the fio tests ran directly from FreeNAS:
4K read, 128K recordsize, 30GB file:
4K read, 4K recordsize, 30GB file
4K write, 4K recordsize, 30GB file
4K write, 128K recordsize, 30GB file
4K read, 128K recordsize, 80GB file to avoid caching
4K read, 4K recordsize, 80GB file
Here is the zpool iostat output for the entire testing above: https://pastebin.com/JKy101D0
I have a FreeNAS server that is not behaving as expected and I need some help with troubleshooting.
The FreeNAS hardware is:
Dell R730xd server, 2x E5-2630L v3 CPUs, 4x 16GB of 1866MHz DDR4 ECC RAM, Intel X550 rNDC 10Gb NIC and 2x Dell PM1725a NVMe SSDs in a mirror, running FreeNAS-11.3-U4.1.
Here is the pool configuration:
Code:
zpool get all ssd NAME PROPERTY VALUE SOURCE ssd size 1.45T - ssd capacity 62% - ssd altroot /mnt local ssd health ONLINE - ssd guid 16147601102770798998 default ssd version - default ssd bootfs - default ssd delegation on default ssd autoreplace off default ssd cachefile /data/zfs/zpool.cache local ssd failmode continue local ssd listsnapshots off default ssd autoexpand on local ssd dedupditto 0 default ssd dedupratio 1.00x - ssd free 551G - ssd allocated 937G - ssd readonly off - ssd comment - default ssd expandsize - - ssd freeing 0 default ssd fragmentation 11% - ssd leaked 0 default ssd bootsize - default ssd checkpoint - - ssd feature@async_destroy enabled local ssd feature@empty_bpobj active local ssd feature@lz4_compress active local ssd feature@multi_vdev_crash_dump enabled local ssd feature@spacemap_histogram active local ssd feature@enabled_txg active local ssd feature@hole_birth active local ssd feature@extensible_dataset enabled local ssd feature@embedded_data active local ssd feature@bookmarks enabled local ssd feature@filesystem_limits enabled local ssd feature@large_blocks enabled local ssd feature@sha512 enabled local ssd feature@skein enabled local ssd feature@device_removal enabled local ssd feature@obsolete_counts enabled local ssd feature@zpool_checkpoint enabled local ssd feature@spacemap_v2 active local
I ran tests on a dataset, switching between 4K and 128K recordsize to compare. My intention is to use the NVMe pool as VM storage, sharing it with NFS to two XCP-NG or Proxmox servers. Here are the dataset properties:
Code:
zfs get all ssd/emporiko_database NAME PROPERTY VALUE SOURCE ssd/emporiko_database type filesystem - ssd/emporiko_database creation Tue Mar 9 17:26 2021 - ssd/emporiko_database used 109G - ssd/emporiko_database available 504G - ssd/emporiko_database referenced 17.2G - ssd/emporiko_database compressratio 1.05x - ssd/emporiko_database mounted yes - ssd/emporiko_database quota none default ssd/emporiko_database reservation none default ssd/emporiko_database recordsize 128K local ssd/emporiko_database mountpoint /mnt/ssd/emporiko_database default ssd/emporiko_database sharenfs off default ssd/emporiko_database checksum on default ssd/emporiko_database compression lz4 local ssd/emporiko_database atime off local ssd/emporiko_database devices on default ssd/emporiko_database exec on default ssd/emporiko_database setuid on default ssd/emporiko_database readonly off default ssd/emporiko_database jailed off default ssd/emporiko_database snapdir hidden default ssd/emporiko_database aclmode passthrough received ssd/emporiko_database aclinherit passthrough inherited from ssd ssd/emporiko_database createtxg 1013 - ssd/emporiko_database canmount on default ssd/emporiko_database xattr off temporary ssd/emporiko_database copies 1 received ssd/emporiko_database version 5 - ssd/emporiko_database utf8only off - ssd/emporiko_database normalization none - ssd/emporiko_database casesensitivity sensitive - ssd/emporiko_database vscan off default ssd/emporiko_database nbmand off default ssd/emporiko_database sharesmb off default ssd/emporiko_database refquota none default ssd/emporiko_database refreservation none default ssd/emporiko_database guid 13401708004503438074 - ssd/emporiko_database primarycache all default ssd/emporiko_database secondarycache all default ssd/emporiko_database usedbysnapshots 91.9G - ssd/emporiko_database usedbydataset 17.2G - ssd/emporiko_database usedbychildren 0 - ssd/emporiko_database usedbyrefreservation 0 - ssd/emporiko_database logbias latency default ssd/emporiko_database dedup off default ssd/emporiko_database mlslabel - ssd/emporiko_database sync disabled local ssd/emporiko_database refcompressratio 1.35x - ssd/emporiko_database written 72K - ssd/emporiko_database logicalused 115G - ssd/emporiko_database logicalreferenced 23.2G - ssd/emporiko_database volmode default default ssd/emporiko_database filesystem_limit none default ssd/emporiko_database snapshot_limit none default ssd/emporiko_database filesystem_count none default ssd/emporiko_database snapshot_count none default ssd/emporiko_database redundant_metadata all default
For testing, I used fio both on the NAS itself and then through NFS using a temporary Proxmox server and a temporary XCP-NG server on different hardware.
Proxmox server: Dell R620, 2x E5-2637 v2, 16x 2GB 1333MHz DDR3 RAM, Intel X550 rNDC 10Gb NIC
XCP-NG server: HP DL360e Gen8, 2x E5-2430L v2, 10x 8GB 1333MHz DDR3 RAM, Broadcom BCM57810 10Gb NIC
Switch: Buffalo BS-MP2008 10Gb switch
All machines are set with MTU 9000.
Here are the fio tests ran directly from FreeNAS:
4K read, 128K recordsize, 30GB file:
Code:
fio read - 128K blocks dataset, 30GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=30G --readwrite=read
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 30720MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=717MiB/s][r=184k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2533: Thu Mar 11 19:15:39 2021
read: IOPS=181k, BW=709MiB/s (743MB/s)(30.0GiB/43338msec)
bw ( KiB/s): min=684095, max=735920, per=99.78%, avg=724294.43, stdev=7686.88, samples=86
iops : min=171023, max=183980, avg=181073.26, stdev=1921.77, samples=86
cpu : usr=14.41%, sys=85.07%, ctx=26678, majf=0, minf=2
IO depths : 1=0.1%, 2=1.8%, 4=5.0%, 8=12.0%, 16=26.4%, 32=53.2%, >=64=1.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.1%, 32=0.1%, 64=1.6%, >=64=0.0%
issued rwts: total=7864320,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=709MiB/s (743MB/s), 709MiB/s-709MiB/s (743MB/s-743MB/s), io=30.0GiB (32.2GB), run=43338-43338msec4K read, 4K recordsize, 30GB file
Code:
fio read - 4K blocks dataset, 30GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=30G --readwrite=read
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 30720MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=748MiB/s][r=192k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2632: Thu Mar 11 19:18:49 2021
read: IOPS=191k, BW=747MiB/s (783MB/s)(30.0GiB/41141msec)
bw ( KiB/s): min=427481, max=799400, per=99.56%, avg=761222.11, stdev=43627.57, samples=82
iops : min=106870, max=199850, avg=190305.17, stdev=10906.88, samples=82
cpu : usr=13.80%, sys=85.88%, ctx=13877, majf=0, minf=2
IO depths : 1=0.1%, 2=1.6%, 4=6.0%, 8=12.5%, 16=25.2%, 32=52.9%, >=64=1.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.3%, 8=0.1%, 16=0.1%, 32=0.1%, 64=1.6%, >=64=0.0%
issued rwts: total=7864320,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=747MiB/s (783MB/s), 747MiB/s-747MiB/s (783MB/s-783MB/s), io=30.0GiB (32.2GB), run=41141-41141msec4K write, 4K recordsize, 30GB file
Code:
fio write - 4K blocks dataset, 30GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=30G --readwrite=write
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 30720MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=183MiB/s][w=46.8k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2694: Thu Mar 11 19:21:50 2021
write: IOPS=84.7k, BW=331MiB/s (347MB/s)(30.0GiB/92890msec)
bw ( KiB/s): min=121117, max=589676, per=100.00%, avg=338710.50, stdev=151492.97, samples=185
iops : min=30279, max=147419, avg=84677.28, stdev=37873.20, samples=185
cpu : usr=13.71%, sys=51.66%, ctx=1147270, majf=0, minf=2
IO depths : 1=0.1%, 2=0.2%, 4=1.1%, 8=4.4%, 16=16.5%, 32=73.5%, >=64=4.2%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=96.4%, 8=0.4%, 16=1.3%, 32=0.7%, 64=1.2%, >=64=0.0%
issued rwts: total=0,7864320,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=30.0GiB (32.2GB), run=92890-92890msec4K write, 128K recordsize, 30GB file
Code:
fio write - 128K blocks dataset, 30GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=30G --readwrite=write
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 30720MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=488MiB/s][w=125k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2815: Thu Mar 11 19:23:52 2021
write: IOPS=117k, BW=455MiB/s (477MB/s)(30.0GiB/67496msec)
bw ( KiB/s): min=302559, max=571976, per=99.63%, avg=464343.55, stdev=52047.25, samples=134
iops : min=75639, max=142994, avg=116085.50, stdev=13011.84, samples=134
cpu : usr=16.21%, sys=63.58%, ctx=429919, majf=0, minf=2
IO depths : 1=0.1%, 2=0.1%, 4=0.7%, 8=3.7%, 16=15.7%, 32=76.2%, >=64=3.6%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=96.8%, 8=0.2%, 16=0.6%, 32=1.1%, 64=1.3%, >=64=0.0%
issued rwts: total=0,7864320,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=30.0GiB (32.2GB), run=67496-67496msec4K read, 128K recordsize, 80GB file to avoid caching
Code:
fio read - 128K blocks dataset, 80GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=80G --readwrite=read
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 81920MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=682MiB/s][r=175k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2942: Thu Mar 11 19:29:33 2021
read: IOPS=176k, BW=689MiB/s (723MB/s)(80.0GiB/118872msec)
bw ( KiB/s): min=643568, max=715592, per=99.81%, avg=704337.76, stdev=6473.07, samples=237
iops : min=160892, max=178898, avg=176084.16, stdev=1618.27, samples=237
cpu : usr=13.29%, sys=86.04%, ctx=75534, majf=0, minf=2
IO depths : 1=0.1%, 2=1.9%, 4=5.1%, 8=12.2%, 16=26.3%, 32=52.8%, >=64=1.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=98.4%, 8=0.1%, 16=0.1%, 32=0.1%, 64=1.6%, >=64=0.0%
issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=689MiB/s (723MB/s), 689MiB/s-689MiB/s (723MB/s-723MB/s), io=80.0GiB (85.9GB), run=118872-118872msec4K read, 4K recordsize, 80GB file
Code:
fio read - 4K blocks dataset, 80GB file
fio --randrepeat=1 --ioengine=posixaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=80G --readwrite=read
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=64
fio-3.16
Starting 1 process
test: Laying out IO file (1 file / 81920MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=779MiB/s][r=199k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3123: Thu Mar 11 19:37:26 2021
read: IOPS=185k, BW=722MiB/s (757MB/s)(80.0GiB/113521msec)
bw ( KiB/s): min=346235, max=804087, per=99.80%, avg=737439.05, stdev=46985.94, samples=227
iops : min=86558, max=201021, avg=184359.55, stdev=11746.43, samples=227
cpu : usr=21.04%, sys=76.28%, ctx=312697, majf=0, minf=2
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=2.1%, 32=92.6%, >=64=5.3%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.2%, 8=0.2%, 16=1.4%, 32=2.9%, 64=0.3%, >=64=0.0%
issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=722MiB/s (757MB/s), 722MiB/s-722MiB/s (757MB/s-757MB/s), io=80.0GiB (85.9GB), run=113521-113521msecHere is the zpool iostat output for the entire testing above: https://pastebin.com/JKy101D0
