Issues with high D/AVG on hosts

[KentShades]

We have a issue with high d/AVG on only one of our presentations to ESXi hosts. Happens to all. Setup exactly the same way as a perfectly fine performing presentation. D/AVG spikes to like 500 in VMware when doing just a file copy. Uses the same Targets, and same initiators as well.

Below is the setup: 10GIG links, 15TB presentation, lz4 compression. ESXi 5.5 U2.

VMware VMkernel logs: 2016-01-04T19:13:03.599Z cpu9:33079)ScsiDeviceIO: 1203: Device naa.6589cfc000000e9865b16c425c74e0b9 performance has improved. I/O latency reduced from 198298 microseconds to 39214 microseconds. 2016-01-04T19:13:09.256Z cpu8:32821)WARNING: ScsiDeviceIO: 1223: Device naa.6589cfc000000e9865b16c425c74e0b9 performance has deteriorated. I/O latency increased from average value of 9932 microseconds to 246699 microseconds. 2016-01-04T19:13:12.093Z cpu11:35552)ScsiDeviceIO: 1203: Device naa.6589cfc000000e9865b16c425c74e0b9 performance has improved. I/O latency reduced from 246699 microseconds to 48951 microseconds.

Screenshots:

http://imgur.com/a/U224P[1]

So the ESXi presentation works fine with no D/AVG spiking ever does a lot more I/O as well. But the Ahsay one gets hit very hard.

Any help here guess?

 

[jgreco]

You'd need to present a lot more information in order to get any help. Please follow the guidelines posted in the Forum Rules, describing your system, configuration, pool, etc.

 

[KentShades]

Sorry my bad!

FreeNAS-9.3-STABLE-201511040813
Hardware:

FREENAS-CERTIFIED-2U-64-A2Z FreeNAS 2U Certified Server: 12 x 3.5” SATA (up to 96TB RAW Capacity) Dual Expanders 64GB RAM (Upgradable to 128GB RAM) - Write Cache Optional + Up to 400GB MLC High-Performance Read Cache (Optional) 4 x 1GbE Ethernet Standard - Mirorred Boot Devices Redundant Power Supply — NFS, CIFS, HTTP, FTP, iSCSI, AFP, Encryption, Snapshots, Compression, RAID, Dual/Triple-parity RAID, Triple-mirroring, Shadow Migration, Cloning, Remote Replication (9/9 - DESSV) 1
HD-HI-SAS-8TB/128-H4 HGST 8TB Helium Ultrastar Nearline SAS 12Gb/s 7200RPM 128MB Cache (HE8) - (H4K) 12
SSD-INTEL-S3710-200GB-2.5 Intel S3710 Series 200GB MLC SATA3 6Gb/s SSD NAND Flash Drive - 2.5” (ZIL) 1
SSD-INTEL-S3510-480GB-2.5 Intel DC S3510 Series 480GB MLC SATA3 6Gb/s SSD NAND Flash Drive - 2.5” (L2ARC) 1

Using iSCSI VMware 5.5 only two extends being done, one portal on 10GIG fibre connection. Both presentations are the exact same setup on the exact same network and config.

 

[cyberjock]

Can you post a debug file for your system?

 

[KentShades]

Here you go! Let me know and I can give you the exact other debug you might want.

 

[cyberjock]

I would redo the zpool as mirrors instead of 3 RAIDZ1 vdevs. That's going to give you a pretty big performance boost right there, and your problem with high d/avg is almost always the zpool's layout (in your case it is RAIDZ1). We recommend mirrors only for iSCSI, especially for VMs.

Note: I changed your account tag to say "FreeNAS Certified Owner". If you would rather not have the tag please PM me. I do this whenever I can so people know that you spent money on hardware that helps the project.

 

[KentShades]

No way to do this on the fly without losing the data correct?

 

[cyberjock]

No way to do this on the fly without losing the data correct?

Correct.

 

[KentShades]

Ok, sounds like I'll need to move the data off and re-do this! This is how it got shipped so figured it was the ideal setup. Strange that it would only effect the one presentation, but this should get done for the best performance anyways.

 

[jgreco]

There aren't any "ideal setups", not really. A NAS machine can be targeted at any of at least half a dozen major use models, which very definitely affect the design of the pool.

For VM storage, you want to build mirror vdevs. Here, for example, the rule is that loss of a drive shall not compromise redundancy. If one of two disks failed, you no longer have redundancy. So the rule effectively means that three way mirror vdevs are needed. Also, you want to avoid filling your pool. The fuller it gets, the harder the system has to work to allocate space, and anything past 50% full will eventually turn into not-fun-ness. So the 48TB raw space filer we have here for VM's is organized as three-way mirror vdevs of 2TB drives, giving us ~14TB of usable pool space plus hot spares. But the 50% rule means that only 7TB of that is usable.

By comparison, another 48TB raw space filer here is made of 12 4TB disks, with 11 in RAIDZ3, making for a ~30TB pool, and because it's archival in nature, it'd be fine to get up to around 28TB of storage on it. The same exact amount of raw space, optimized for a different purpose, yields FOUR TIMES the usable space.

 

[KentShades]

Yeah, here is my issue I'm not running VM's off of this really. I'm running Veeam backups, and running another backup repo to this. So I don't need amazing performance. But the performance I'm getting on only the one presentation (ESXi one works fine has more writes to it and is larger, just the Ahsay one doesn't perform right) is a issue. Otherwise this thing performs amazing with the current setup. We also bought this in the hope of mass storage on the cheaper end. Having 7TB's of usuable space for 20K I could have gone with a full on SAN for that price. So really space vs performance here but a simple file transfer going 200 KB/s and spiking the D/AVG to 500 vs 80 MB/s and not even moving the D/AVG is the difference between the presentations I've made. It's just hard to imagine if this was a RAIDZ1 issue I'd see it across the board and not just on one of my two presentations. I could be wrong, but generally goes against everything I know about storage and SAN's.

 

[jgreco]

The speed at which writes occur is generally tied fairly closely to how much free space and fragmentation exists on your pool. As your pool fragments, write speeds will tend to decrease. Are you comparing write speeds from the past with write speeds today, by any chance, or are you actually seeing slow write speeds on one, and then going to the other and testing and it's still fast? And are you using the same tool to test?

As for 7TB of usable space for $20K, I can tell you it should be a lot cheaper than that.

Supermicro X10SRL plus E5-1650v3 CPU - $1000
Supermicro SC846BE16-R1K28B - $1200
4 x 32GB M393A4K40BB0-CPB - $800
Intel 400GB DC P3700 SSD for SLOG - $900
Intel 750 400GB HHHL for L2ARC - $300
LSI 9211-8i HBA - $200
Intel X520-SR2 - $400

There's an uncompromising 24 drive storage server with 128GB of RAM, a hell-fast CPU, best-in-class SLOG, reasonable L2ARC, and dual 10GbE for $4800. Cram 12 units of WD WD4001FYYG in there as four vdevs of 3-way mirror, for $3000 more, and you end up with 16TB of pool space, about ~6-8 of which you can use. That's only $7800.

But...! I'm a clever bastard, and I included 12 more bays for free. So you get to double your space for only $3000 at some point down the road. That's still only around $11K... about half this $20K number you mention. I just built twice the storage for about half the price.

And if you take some of my advice, you ditch the SAS hard drives and use WD Red instead, and suddenly those numbers look even better. Especially if you go with 5TB or 6TB Red.

 

[tvsjr]

Or for not too much more, bump to an SC847 chassis and you get 36 bays. You can then get triple the storage, and you're still well under $20K.

 

[jgreco]

Yeah, but I was going for uncompromising. The 847's rear drives run a bit warmer than I'd care for.

 

[cyberjock]

@KentShades - I've talked to your Sales Manager at iXsystems and he'd like to discuss this situation with you further as the situation is more complex than this thread may be leading. I'm not convinced that mirrors are the best way forward, but we can discuss that on the phone. He should be leaving you a voicemail today. We'd like to get this resolved.

Thanks.