Out of the blue vmware esx + iscsi issues

[cyberjock]

What? Your thread title says "... + iscsi issues". But you're not having issues with iscsi? So why did you create the thread? I'm confused as all hell now.

And your Q9300 doesn't do 32GB of RAM. 8GB max with socket 775. ;)

 

[Nick Binder]

I'm having issues with iscsi + esx with my home machine, the issues in the very first post, and posts after it. You decided it was because of lacking in RAM, I just
said I hadn't had the issues in the past, and if that's the case, I'm glad it's not another issue because I was worried it might happen in production.

Also, we were both wrong, it's 16GB:
https://www.asus.com/Motherboards/P5Q3/#specifications

 

[cyberjock]

Hmm. First 775 board I've seen that took 4GB DDR2 sticks. I knew for 100% certainty that you didn't have 8GB sticks because those didn't work with Intel CPUs that weren't registered RAM, and your board won't work with registered RAM. Only server boards do registered RAM.

The RAM thing is something that can start up suddenly and without warning. I couldn't even stream a single movie from the machine until I upgraded from 12 to 20GB of RAM. I went from having no problems when I went to bed to having lots of problems when I woke up. More RAM was the only solution.

 

[jgreco]

iSCSI is kind of a finicky thing due to a variety of issues, including block size issues, fragmentation, etc. As a pool ages, an iSCSI extent tends to become highly fragmented due to the need to be rewriting inconveniently sized chunks. In the ZFS model, ARC and L2ARC are the answer to fragmentation (put the entire working set in ARC/L2ARC!), and/or "stop everything, make a fresh copy, then restart it" to forcibly reorganize the blocks on disk.

I have long said that the normal "80%" pool fill limit for ZFS needs to be SUBSTANTIALLY less for iSCSI, 60% or less, in a potentially hopeless bid to allow the system to more reasonably allocate adjacent chunks. Put simply, if you write two things, they might be related, and so you might luck out when reading them too. This could at least slow the inevitable increase of fragmentation down.

You can see the impressive damage done to throughput as a pool fills, see Fig. 4 of this presentation. One might reasonably conclude from this that my 60% number is insanely optimistic, but I will point out that the use pattern being tested is rather closer to pathological than it is to average use cases. But is very telling that the chart shoots down from 6000 at 10% full to ~3000 at ~20% full.

I guess that I wouldn't expect to see a sudden and catastrophic performance falloff on an established pool without some significant trigger event, but it is possible that one happened without your being aware of it. For example, writing large portions of the disk, such as "make world" on a FreeBSD system, would be hell on iSCSI + ZFS. But your initial description of the problem still sounds like something else may be amiss.

 

[cyberjock]

I disagree. I had to add more RAM for my iscsi to recover. But even then, because of the caching and the heavy fragmentation, more RAM is needed to help offset the I/O penalties with the data being scattered all over the platter as usage increases. For me, I had to increase the VM for FreeNAS and decrease for my VM to get increased performance. As it stand, because my system is limited to 32GB of RAM I had to rob from Peter to pay Paul. I'm not thrilled with the performance of my iscsi device now at 20MB/sec typically and 40MB/sec in best case scenarios. But I do acknowledge that since my pool is so full that I shouldn't be complaining about my performance(so I don't). I'm also not too upset that it takes 2 minutes to boot when it used to take 30 seconds.

 

[jgreco]

What are you disagreeing about?

I did say the best ZFS answer to frag was ARC/L2ARC for the working set...

 

[cyberjock]

The comment that you wouldn't have expected a sudden drop in performance. Mine went from being roughly as fast as the SSD I had it on(20 seconds or so to bootup and be logged in) to more than 2 minutes. I had to go change my expectations for auto-shutdown of the VMs in ESXi to account for the longer time for shutdowns.

 

[jgreco]

So one day it went fron real fast to real slow with no other obvious triggers?

 

[cyberjock]

Yep. I could have sworn you and I discussed this in PM.

It was just like the drop in speed I saw back in February, but that was with my entire pool(even dd tests showed crappy performance).

 

[Nick Binder]

Yep. Putting a couple drives back in a dual core 4GB ram box I was using 6 months with no issues with a fresh install of 9.2 ALPHA on :) dedicated just to iscsi to see if maybe it's actually a hardware(NIC) issue.

 

[jgreco]

Yep. I could have sworn you and I discussed this in PM.

Maybe we did. But you know those old guys who play chess in Central Park? I'm a little like that... I don't necessarily bother to remember stuff if I can scroll back in a conversation.

It was just like the drop in speed I saw back in February, but that was with my entire pool(even dd tests showed crappy performance).

Mmm. Metadata-pocalypse of some sort? Interesting. What percent full is meant when you say "so full"? And you've got a pretty big pool if I recall, 24 2TB's is it? 12GB to 20GB... mmm... Do you have atime disabled or is it possible that "reads" from the pool were resulting in some writes? Because if it was full enough that it was struggling to write some trite writes, that could shoot the whole thing to hell. But I think that'd have to be a very full pool.

 

[cyberjock]

It was about 80% full at that time. It is 18x2TB drives. Back then I had to go from 12 to 20GB. Now I'm at 24GB of RAM. Atime has always been disabled. The issue was that the pool seemed to do very small quantities of data per second. We're talking 10MB/sec if I was lucky. It was a mess as CPU usage was low, disk activity was low, and I could find no bad disks or anything. Someone said I should give it more RAM. So I upped it to 24GB of RAM and iSCSI got MUCH faster. Before that I tried everything. Reboots, tunables/sysctls, all sorts of crap.

 

[daimi]

iSCSI is kind of a finicky thing due to a variety of issues, including block size issues, fragmentation, etc. As a pool ages, an iSCSI extent tends to become highly fragmented due to the need to be rewriting inconveniently sized chunks. In the ZFS model, ARC and L2ARC are the answer to fragmentation (put the entire working set in ARC/L2ARC!), and/or "stop everything, make a fresh copy, then restart it" to forcibly reorganize the blocks on disk.

I have long said that the normal "80%" pool fill limit for ZFS needs to be SUBSTANTIALLY less for iSCSI, 60% or less, in a potentially hopeless bid to allow the system to more reasonably allocate adjacent chunks. Put simply, if you write two things, they might be related, and so you might luck out when reading them too. This could at least slow the inevitable increase of fragmentation down.

You can see the impressive damage done to throughput as a pool fills, see Fig. 4 of this presentation. One might reasonably conclude from this that my 60% number is insanely optimistic, but I will point out that the use pattern being tested is rather closer to pathological than it is to average use cases. But is very telling that the chart shoots down from 6000 at 10% full to ~3000 at ~20% full.

I guess that I wouldn't expect to see a sudden and catastrophic performance falloff on an established pool without some significant trigger event, but it is possible that one happened without your being aware of it. For example, writing large portions of the disk, such as "make world" on a FreeBSD system, would be hell on iSCSI + ZFS. But your initial description of the problem still sounds like something else may be amiss.

As you have mentioned "...put the entire working set in ARC/L2ARC..." and if SSDs are being used for ARC & L2ARC. Can I just used SSD directly as data storage/zpool to solve this performance issue?

 

[daimi]

## Q1 #########
And you have also mentioned ".....most iSCSI is 512 byte block stuff (like ESXi) and ZFS on modern drives is oriented towards 4K blocks ...... So drives write slow when they have to seek, and they write even a bit slower if they have to seek and then write a large block...."
(reference : http://forums.freenas.org/threads/freenas-9-veeam-iscsi-time-slow-performance.16396/#post-84178)

Based on above comment:
- SSD as data storage/zpool should help in write performance (SSD writes faster than conventional HDD)?
- Would I gain any performance if FreeNAS presents the data storage/zpool to ESXi using NFS (and avoid using iSCSI)?

## Q2 #########
My plan is to have:
i) ESXi hosting a FreeNAS VM (on a small 60GB **SSD/HDD?**),
ii) Then controller direct pass through to FreeNAS VM
iii) Finally FreeNAS VM presents the data storage/zpool (on another bigger 512GB **SSD/HDD?**) to the same ESXi using iSCSI or NFS for other VMs

Please advice will there be a big difference using **SSD/HDD?** as indicated in above 2 places?

## Q3 #########
In another perspective, which of the following give better write performance?
1) buy more RAM for FreeNAS (say 32GB) RAM and use conventional 512GB 7200rpm HDD as data storage/zpool,
-- OR --
2) buy 512GB SSD as data storage/zpool and gives FreeNAS a limited 8GB RAM

They work out to be around the same cost

 

[jgreco]

As you have mentioned "...put the entire working set in ARC/L2ARC..." and if SSDs are being used for ARC & L2ARC. Can I just used SSD directly as data storage/zpool to solve this performance issue?

I'm having trouble coming up with a reason that mirrored (not RAIDZ) SSD would not be equivalent or better than L2ARC in practice.

Based on above comment:
- SSD as data storage/zpool should help in write performance (SSD writes faster than conventional HDD)?
- Would I gain any performance if FreeNAS presents the data storage/zpool to ESXi using NFS (and avoid using iSCSI)?

SSD can potentially write faster than conventional HDD. However, the ZFS transaction group write mechanism is still likely to be a bottleneck. Choice of a good SLOG device is probably still required.

NFS and iSCSI are roughly equivalent when you look at the big picture as far as reading and writing blocks, which is fundamentally what ESXi is doing. They differ in implementation details.

The remainder of your questions are answered elsewhere, including the two virtualization stickys.

 

[daimi]

I'm having trouble coming up with a reason that mirrored (not RAIDZ) SSD would not be equivalent or better than L2ARC in practice.



SSD can potentially write faster than conventional HDD. However, the ZFS transaction group write mechanism is still likely to be a bottleneck. Choice of a good SLOG device is probably still required.

NFS and iSCSI are roughly equivalent when you look at the big picture as far as reading and writing blocks, which is fundamentally what ESXi is doing. They differ in implementation details.

The remainder of your questions are answered elsewhere, including the two virtualization stickys.

Sound like you are suggesting the following:
1) Assume 500GB SSD (good for read/write) for zpool (non RAID)
2) Assign just enough RAM for FreeNAS : 7GB (6 GB ZFS baseline + 1GB/TB of storage)
3) Buy a SLOG device for sync write enhancement

Please advice on what kind of SLOG device I should get?
- a simple SATA 6GB/s SSD drive (e.g. samsung 840 pro), or
- a dedicated PCIe SSD (e.g. OCZ RevoDrive 3 - PCI Express)
- or it MUST be **SUPER CAPACITOR** (e.g. samsung SM1625 as stated in link below)
http://www.samsung.com/global/business/semiconductor/Downloads/CeBIT_DRAM_SSD_Synopsis_Customers.pdf

It took me more than 45 minutes to move 15GB of VM to zpool (based on conventional HDD) via NFS.

 

[jpaetzel]

For what it's worth, the 9.2.0-BETA has a number of ZFS improvements that directly benefit iSCSI.

 

[jgreco]

Tease.

 

[cyberjock]

Sound like you are suggesting the following:
1) Assume 500GB SSD (good for read/write) for zpool (non RAID)
2) Assign just enough RAM for FreeNAS : 7GB (6 GB ZFS baseline + 1GB/TB of storage)
3) Buy a SLOG device for sync write enhancement

Please advice on what kind of SLOG device I should get?
- a simple SATA 6GB/s SSD drive (e.g. samsung 840 pro), or
- a dedicated PCIe SSD (e.g. OCZ RevoDrive 3 - PCI Express)
- or it MUST be **SUPER CAPACITOR** (e.g. samsung SM1625 as stated in link below)
http://www.samsung.com/global/business/semiconductor/Downloads/CeBIT_DRAM_SSD_Synopsis_Customers.pdf

It took me more than 45 minutes to move 15GB of VM to zpool (based on conventional HDD) via NFS.

I think what he actually said was...

The remainder of your questions are answered elsewhere, including the two virtualization stickys.

I will tell you that you will want LOTS of RAM. Why you'd think that the minimum is a good place to start is beyond me. RAM is basically the single best way to increase performance. Not to mention that you MUST have lots of RAM to use an l2arc, so 7GB of RAM is a joke. There's a bunch of rules for picking l2arc size and stuff, and one of them is LOTS of RAM.

 

[jgreco]

Poster might do OK since we were discussing SSD for pool, not L2ARC. It is not clear that lots of memory would be particularly advantageous especially if it is eating into an architecture-imposed limit like the E3's 32GB and reducing the availability to other VM's. It would at least be smart to try the SSD pool and see if the increased access speed and elimination of seek overhead overcame the small-memory performance hit; my best uninformed guess is "yes!"

As for SLOG, I still haven't found a better suggestion than abusing a BBU RAID card and hard drives. This works out neatly on an ESXi build for the local boot device and datastore and does double duty as SLOG. Cool.