FreeNAS SSD iSCSI not performing as expected

Status
Not open for further replies.

resuni

Cadet
Joined
Aug 4, 2017
Messages
4
I'm looking at using FreeNAS to act as the iSCSI storage in a large QEMU/KVM environment. The idea is to find a build that works really well, and add identical SANs to the environment as it grows. I just assembled the first prototype and am concerned with the initial performance tests.

Let's start with the build...
Right now, the environment contains 6 KVM hypervisors running CentOS 6. Each hypervisor connects to the FreeNAS box via iSCSI. The FreeNAS iSCSI LUN is then configured with LVM. Each VM's virtual disk is a logical volume saved on this LVM volume group.

All 6 hypervisors have 10 Gb/s connections to the NAS. Jumbo frames are enabled.

For testing, I used fio from one of the VMs to test read & write performance.

Code:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.2.8
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [114.5MB/39348KB/0KB /s] [29.3K/9837/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=37650: Sat Jul 29 14:46:08 2017
read : io=3071.7MB, bw=115737KB/s, iops=28934, runt= 27177msec
write: io=1024.4MB, bw=38596KB/s, iops=9648, runt= 27177msec
cpu : usr=9.56%, sys=42.55%, ctx=53933, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: io=3071.7MB, aggrb=115737KB/s, minb=115737KB/s, maxb=115737KB/s, mint=27177msec, maxt=27177msec
WRITE: io=1024.4MB, aggrb=38595KB/s, minb=38595KB/s, maxb=38595KB/s, mint=27177msec, maxt=27177msec

Disk stats (read/write):
vda: ios=785091/261828, merge=0/5, ticks=1046681/445653, in_queue=1492404, util=99.67%


I'm comparing this test to a test I did on a single SSD connected to my PC (same make & model).

Code:
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.10
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=0): [f(1)] [100.0% done] [249.3MB/84912KB/0KB /s] [63.9K/21.3K/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=15536: Tue Aug  8 11:32:47 2017
  read : io=3070.4MB, bw=262436KB/s, iops=65609, runt= 11980msec
	bw (KB  /s): min=252608, max=269802, per=100.00%, avg=265775.61, stdev=4300.26
  write: io=1025.8MB, bw=87673KB/s, iops=21918, runt= 11980msec
	bw (KB  /s): min=82892, max=90553, per=100.00%, avg=88759.48, stdev=1763.77
  cpu		  : usr=18.17%, sys=54.43%, ctx=567202, majf=0, minf=9
  IO depths	: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
	 submit	: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
	 issued	: total=r=785996/w=262580/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
	 latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3070.4MB, aggrb=262436KB/s, minb=262436KB/s, maxb=262436KB/s, mint=11980msec, maxt=11980msec
  WRITE: io=1025.8MB, aggrb=87672KB/s, minb=87672KB/s, maxb=87672KB/s, mint=11980msec, maxt=11980msec

Disk stats (read/write):
	dm-0: ios=771113/257612, merge=0/0, ticks=597272/124032, in_queue=722384, util=99.13%, aggrios=782967/262195, aggrmerge=3029/393, aggrticks=606372/126344, aggrin_queue=733456, aggrutil=98.65%
  sdb: ios=782967/262195, merge=3029/393, ticks=606372/126344, in_queue=733456, util=98.65%


I may be performing/reading these tests wrong, but this is what I'm seeing:
  • FreeNAS RAID10 test read/write IOPS: 28934/9648
  • Local single-SSD test read/write IOPS: 65609/21918
These tests are indicating that a single SSD is performing better than 4 in a RAID, which doesn't make sense. Where do I start looking for bottlenecks? It's not the network, because I'm not even close to saturating the 10Gb/s links.

I would like to perform the same test on the FreeNAS box itself, bypassing iSCSI, but I'm not sure how to access the ZFS from there.

I also have a question about RAM, which may be related. I found this post, Performance tuning iSCSI - system review, which indicates that a large amount of RAM is required to run VMs over iSCSI. Why is this? Is ZFS using it for caching? If so, is a large cache really necessary with SSD-only storage? Is this possibly what's ruining our IOPS? Increasing the RAM is definitely not out of the question for us, I'm just wondering what all that RAM is used for.

To sum up my post in a list of questions:
  • What's the next step in finding out what's causing this performance loss?
  • How can I benchmark the disks directly on the FreeNAS machine, instead of benchmarking over iSCSI?
  • What does FreeNAS use RAM for in my ZFS/iSCSI configuration?
Thanks in advance.

EDIT: I forgot to clarify that currently only 3 test VMs are running on this right now. While 6 hypervisors are attached, only 3 VMs between those are using this storage.
 
Last edited:

Thomas102

Explorer
Joined
Jun 21, 2017
Messages
83
Hi,

Juste a thought about the test. You are using 4gb and have 32gb memory. You might be testing your arc only.
 
Status
Not open for further replies.
Top