resuni
Cadet
- Joined
- Aug 4, 2017
- Messages
- 4
I'm looking at using FreeNAS to act as the iSCSI storage in a large QEMU/KVM environment. The idea is to find a build that works really well, and add identical SANs to the environment as it grows. I just assembled the first prototype and am concerned with the initial performance tests.
Let's start with the build...
All 6 hypervisors have 10 Gb/s connections to the NAS. Jumbo frames are enabled.
For testing, I used fio from one of the VMs to test read & write performance.
I'm comparing this test to a test I did on a single SSD connected to my PC (same make & model).
I may be performing/reading these tests wrong, but this is what I'm seeing:
I would like to perform the same test on the FreeNAS box itself, bypassing iSCSI, but I'm not sure how to access the ZFS from there.
I also have a question about RAM, which may be related. I found this post, Performance tuning iSCSI - system review, which indicates that a large amount of RAM is required to run VMs over iSCSI. Why is this? Is ZFS using it for caching? If so, is a large cache really necessary with SSD-only storage? Is this possibly what's ruining our IOPS? Increasing the RAM is definitely not out of the question for us, I'm just wondering what all that RAM is used for.
To sum up my post in a list of questions:
EDIT: I forgot to clarify that currently only 3 test VMs are running on this right now. While 6 hypervisors are attached, only 3 VMs between those are using this storage.
Let's start with the build...
- Chassis: Supermicro SYS-1028R-WTNRT - http://a.co/hknytiN
- CPUs: 2x Intel Xeon E5-2603 v4 @ 1.79 GHz - http://a.co/fbbdeS0
- RAM: 32 GB (8x 4 GB) - http://a.co/aX0D6ds
- OS storage: 2x SanDisk Ultra Fit 16GB - http://a.co/hbBkVrx
- iSCSI storage: 4x Crucial MX300 1TB (RAID 10) - http://a.co/jfW1kxG (this will be expanded to 8 later)
All 6 hypervisors have 10 Gb/s connections to the NAS. Jumbo frames are enabled.
For testing, I used fio from one of the VMs to test read & write performance.
Code:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.2.8 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m(1)] [100.0% done] [114.5MB/39348KB/0KB /s] [29.3K/9837/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=37650: Sat Jul 29 14:46:08 2017 read : io=3071.7MB, bw=115737KB/s, iops=28934, runt= 27177msec write: io=1024.4MB, bw=38596KB/s, iops=9648, runt= 27177msec cpu : usr=9.56%, sys=42.55%, ctx=53933, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3071.7MB, aggrb=115737KB/s, minb=115737KB/s, maxb=115737KB/s, mint=27177msec, maxt=27177msec WRITE: io=1024.4MB, aggrb=38595KB/s, minb=38595KB/s, maxb=38595KB/s, mint=27177msec, maxt=27177msec Disk stats (read/write): vda: ios=785091/261828, merge=0/5, ticks=1046681/445653, in_queue=1492404, util=99.67%
I'm comparing this test to a test I did on a single SSD connected to my PC (same make & model).
Code:
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio-2.10 Starting 1 process test: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=0): [f(1)] [100.0% done] [249.3MB/84912KB/0KB /s] [63.9K/21.3K/0 iops] [eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=15536: Tue Aug 8 11:32:47 2017 read : io=3070.4MB, bw=262436KB/s, iops=65609, runt= 11980msec bw (KB /s): min=252608, max=269802, per=100.00%, avg=265775.61, stdev=4300.26 write: io=1025.8MB, bw=87673KB/s, iops=21918, runt= 11980msec bw (KB /s): min=82892, max=90553, per=100.00%, avg=88759.48, stdev=1763.77 cpu : usr=18.17%, sys=54.43%, ctx=567202, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=785996/w=262580/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3070.4MB, aggrb=262436KB/s, minb=262436KB/s, maxb=262436KB/s, mint=11980msec, maxt=11980msec WRITE: io=1025.8MB, aggrb=87672KB/s, minb=87672KB/s, maxb=87672KB/s, mint=11980msec, maxt=11980msec Disk stats (read/write): dm-0: ios=771113/257612, merge=0/0, ticks=597272/124032, in_queue=722384, util=99.13%, aggrios=782967/262195, aggrmerge=3029/393, aggrticks=606372/126344, aggrin_queue=733456, aggrutil=98.65% sdb: ios=782967/262195, merge=3029/393, ticks=606372/126344, in_queue=733456, util=98.65%
I may be performing/reading these tests wrong, but this is what I'm seeing:
- FreeNAS RAID10 test read/write IOPS: 28934/9648
- Local single-SSD test read/write IOPS: 65609/21918
I would like to perform the same test on the FreeNAS box itself, bypassing iSCSI, but I'm not sure how to access the ZFS from there.
I also have a question about RAM, which may be related. I found this post, Performance tuning iSCSI - system review, which indicates that a large amount of RAM is required to run VMs over iSCSI. Why is this? Is ZFS using it for caching? If so, is a large cache really necessary with SSD-only storage? Is this possibly what's ruining our IOPS? Increasing the RAM is definitely not out of the question for us, I'm just wondering what all that RAM is used for.
To sum up my post in a list of questions:
- What's the next step in finding out what's causing this performance loss?
- How can I benchmark the disks directly on the FreeNAS machine, instead of benchmarking over iSCSI?
- What does FreeNAS use RAM for in my ZFS/iSCSI configuration?
EDIT: I forgot to clarify that currently only 3 test VMs are running on this right now. While 6 hypervisors are attached, only 3 VMs between those are using this storage.
Last edited: