Microsoft, iSCSI and Hyper-V

[MartynW]

I've searched the forums on this, but all I seem to find is posts on ZFS, ESXi and iSCSI, so I'm a bit confused as to if I should be following the same tips?

My Config is
FreeNAS 9.2.1.6

• Supermicro Motherboard MBD-X10SL7-F-O
• Intel I3-4340
• 32 GB ECC Hynix Memory
• 6 x Seagate NAS HDD 4TB SATA in RAIDZ2 Encrypted
• Intel D99083 10GB XF Series 10GbE Single Port PCI-e Server Adapter Dell RN219

Windows 8.1

• Shuttle SH87R6 Barebone XPC with Intel H87 MBD and 1 x 1 gbe NIC
• 16 GB 1600MHz RAM (Might up this to 32GB)
• Intel Core i7 4770S
• Intel D99083 10GB XF Series 10GbE Single Port PCI-e Server Adapter Dell RN219

The 10 GBe NIC's are direct linked with Jumbo Packet of 9014. I have a iSCSI drive formatted to NTFS on the Windows PC.

To this drive I'm getting what I think to be OK'ish performance on Crystal Benchmark (1 gb file)
SEQ
Read 323.4
Write 451.5
512K
Read 320.2
Write 260.8
4K
Read 9.948
Write 19.77
4K QD32
Read 230.9
Write 209.5

Then on the Windows PC I am running Hyper-V with the VM's disks on the iSCSI drive.

The problem is I'm getting terrible disk I/O on the VM's? I'm seeing this in the Task Manager on the VM, sort of 2MB/s with the odd burst. Copying big files is better, 50MB/s odd, but still not great.

I'm assuming this is because its lots of small random I/O?

Is there anything glaringly obviously bad with my setup, if not, where should I start my investigations?

Thanks in advance

 

[Dalma]

I did an install with hyperv and a Dell San solution a while back and there were two issues I encountered. The first was jumbo frames where I had missed enabling jumbo frames on the switch port. Obviously jumbo frames are required on the vm host, vm client Nas nic and all of the switch ports used.

The second issue was a problem with nic drivers on the windows box where I had to disable Vm queuing on the nic. This change more than tripled my speed.

 

[diehard]

Might want to switch to mirrors instead of RAIDZ2, if iops is of any concern then a single RAIDZ2 vdev just isnt going to get it done.

 

[zambanini]

with only a 1gb file in crystal benchmark, he is obviously only testing ARC with iscsi on the freenas site. so the raidz2 is not the issue with the 4k blocksize test.

I have almoast the same 4k random iscsi limit on one of my boxes. but I have 96gb ram and 12x 2tb sas drives running as stripe mirror, 20÷ fill rate. about 18mbyte/s read write (4k, no encryption) with the above tool with three mulitpath iscsi connections (1gbit).

with a HP p2000 g3 and 12 disks, raid10 and the same testsetup I get about 200mb/s on random 4k.

on my testplan next month: create a new iscsi target with 4096 blocksize, instead of the default 512.
iscsi mutlithread is enabled?

for produktive use: please do not use raid z1 or z2, the speed punishment is really bad.

depending on your workload, 32gb ram might be not enough. btw.. do not say benchmarks three times in a row within this forum.

 

[MartynW]

@Dalma, its a direct link 10gbe, i.e no switch and the cards don't have a VM Queue setting, but thanks for the suggestions
@diehard I installed 2 x 3TB dives and mirrored them, moved the VM, but it was still slow, thanks
@zambanini tried a Zpool with 4K block size, no noticeable difference, the default is 8k, is that what you meant?

The vhd are 60GB, i have the max RAM the motherboard goes to, i.e. 32GB, with iscsi are what is the ZIL doing?

I set sync=disabled on the ZFS and it FLEW, does this mean i should look at a SLOG? zilstat had plenty of non-zero's?

With 32GB memory, I take it a Intel S3500 Series 80GB SSD would be more than enough? Is there any point mirroring?

 

[jgreco]

If you've twiddled sync=disabled off and it went much faster, ... big shock there. Please read the "insights into SLOG" sticky conveniently posted at the top of this forum.

 

[zambanini]

@MartynW please go back to the drawing board.

You really have now clue about ZFS. sorry.


Read the FAQs. the documentation, cyberdogs noob guide..and understand the system. if you do not..start again.
FreeNAS ist not a plug and play system (if you expect speed, relibaility and so on)
If you do not understand the basics, you are on a good way to destroy your data too : )


I talked about the iscsi target block size, not the zfs record size...and please do not change the block size on an existing target, you will destroy your data on the first write access.

regarding your test with crystal benchmark: did you at least understand, that this benchmark will not help you, because you just tested ARC and the iscsi service?
this test will not show you your bottleneck. (if you understand the basics.. yes...you got it!)


and btw:
* get sure your hyperV iscsi ist not the problem here, you may use vpshere on another box, to have something to compare, you may bark up to wrong tree, too. (and just use 1gbit link, it should be enough for your test)
* do a test without the pool encryption

 

[jgreco]

Yeah, after all the hours spent trying to help people, it is kind of disheartening to see someone making such basic errors as trying to do VM storage on RAIDZ2. With an i3. With encryption. Without a coherent SLOG gameplan. And only 32GB of memory on a system serving VM's. (((((sigh)))))

Fundamentally, hard disk based storage is never going to be rocket fast but you sure can drop more boat anchors to make it go slower.

 

[MartynW]

Some people had some bad weekends eh? My NAS was built with a budget and it does many things apart from VM, I can live with the performance, I would just like to know if anything cheap can do to improve it. No, I'm not an ZFS expert, if I was I wouldn't need to post here. I had read all those articles, and couldn't deduce the answer I sort.

@zambanini Crystal Benchmark was doing just what you suggested, it was proving the iSCSI and ARC wasn't to blame. I was comparing real world performance when I said it was poor. It ran at 2mb/s (task manager opening up programs, but 50 MB/s copying large files. To my untrained eye, seems that it can run at higher speeds, and probably is to do with small files?

Reading between the lines. jreco, you said "Without a coherent SLOG gameplan", so I'm hoping you are hinting spending $200 on a SLOG might help here? I can't afford to change the motherboard/RAM/CPU and I tried mirrors and dropping encryption but to no avail.

 

[zambanini]

you should really reread the answers. I am out, wasting time is not nice.

 

[jgreco]

Yeah, you have to remove ALL the boat anchors AT THE SAME TIME, not one or two at a time..