iSCSI Crash/Hang VMware Storage vMotion

[bmh.01]

i3-2130
X9SCM-F
32GB Samsung ECC
3x LSI SAS1068e (old I know)
2x Intel 1000 PT Server Adapter
16x 1TB Seagate
4x 240GB OCZ ARC100

3 Pools:
2* 3x2 Mirror
1 2x2 SSD Mirror

All providing iSCSI storage for a 'production' lab VMware cluster.

I'm seeing a repeated (but difficult to reproduce) 'crash' which the only common thing is it is always during a storage vmotion operation. ESXi will show APD, SSH will freeze then eventually disconnect, the console over IPMI will be responsive in the shape that pressing enter will produce a line feed and key presses seem to be accepted but with no response, no shell, nothing. Only way out is to hard reset freenas over IPMI.

This has been happening ever since I moved from nexentastor (due to VAAI instability!) to FreeNAS lured by the apparent feature completeness of ctl vaai, both originally when I was running no ecc and a Z68 motherboard and now when i'm a bit closer to a recommended configuration. I can run memtest until the cows come home (days) without any errors and nothing memory error wise is reported in the IPMI event log either.

No crash dump, nothing in the logs, it literally just stops doing anything when it happens. It's also pretty random I could do 10 SvMotions and it will fall over on the 11th which makes it even more annoying as you can never be sure when it will happen. The only quasi indicator I get is was the idle process will drop to under 250% usage before it dies.

Anybody else experienced anything like this? I've run out of ideas and options until NexentaStor 5 goes GA but i'm not sure I want to go back there either.

 

[jgreco]

Does it happen on both the SSD and HDD pools? I haven't had any issues with storage vmotion and I've been hitting it quite a bit as I shuffled a bunch of VM's off of and then back onto our VM filer after some recent hardware upgrades.

 

[bmh.01]

Yep, seems to be pool agnostic. This time I was moving between two zvols on the second HDD pool so I could recreated the zvol with a larger volblocksize as l2arc headers were killing me (11gb+). I've had it happen between or inside any of the pools.

 

[jgreco]

You didn't describe any L2ARC in your post. If you had, I would immediately have pointed to that as being potentially problematic. Your system really doesn't have enough RAM to be able to cope with L2ARC for block sharing purposes.

 

[bmh.01]

Sorry missed that out of the first post. Problematic to the point of a crash? I was aware that i'd need more RAM in a perfect world but didn't expect it to cause problems outside of potential better use of the ARC for data rather than headers, I will remove the l2arc and try running without.

 

[jgreco]

What you're describing is a lockup, not a crash. And, yes, there seem to be cases where L2ARC on systems with too-little RAM can lead to that. I've got an N36L that we use for backups here, 16GB of RAM with an L2ARC, and was using it for dedup. It would occasionally just wedge even though most of the time it was fine. Turned off dedup and it basically stopped doing anything significant with the L2ARC and it's been fine ever since.

Storage vmotion is likely to be placing a lot of stress on the ARC, which in turn will cause a lot of L2ARC activity. Observationally, we know that there seem to be problems with L2ARC on systems with less than 32GB of RAM, and that it's a really good idea to size RAM for block storage systems more along the lines of what's suggested for dedup ( > ~5GB per TB). The VM filer here has 128GB of RAM and 512GB of L2ARC to serve a pool with about 7TB of usable space, of which only two or three is currently used. I would classify it as pleasant but not lightning fast. HDD is still a limiting factor.

 

[bmh.01]

Just to follow this up, after removing the l2arc (and reutilizing the drives as vFRC as to still get some benefit), the issue seems to be resolved I haven't observed any crashes (touch wood) despite some heavy svmotion use.

Thanks for your help @jgreco, not sure I would have made the connection until much later and probably not for the right reasons.

 

[jgreco]

Happy to hear a positive resolution. vFRC is a good option though it means needing to add some to each host, or losing vmotion on the vFRC-enabled VM's, I believe.

 

[bmh.01]

Indeed it does, you can still vmotion but you'll have to drop the cache. There is an option to do this during a vmotion operation between vFRC enabled hosts as well to reduce the time taken to move the guest and esxi will rewarm the cache afterwards on the new host. VMware has a pretty good whitepaper on it which is worth a read for anyone looking to use it.

Fortunately I had 4 drives and 4 hosts so it worked out well in that respect.

 

[sfcredfox]

Anybody else experienced anything like this? I've run out of ideas and options until NexentaStor 5 goes GA but i'm not sure I want to go back there either.

OP,
I have had the same issue on occasion, however my experience has thus far been only when doing multiple simultaneous storage motions at the same time.

After reading jgreco's response regarding L2ARC, and having been reviewing other arcstat posts regarding ARC performance, I'm wondering if I shouldn't remove mine also. It's hit percentage is terrible (like 6%), but I haven't invested enough time studying it to know if there is anything I need to adjust with it servicing a block storage pool rather than a file based pool.

Have you still not experienced the problem after removing the L2ARC?



FreeNAS 9.3-Stable 64-bit

Show : Main System

FreeNAS Platform:
SuperMicro 826 (8XDTN+)
(x2) Intel(R) Xeon(R) CPU E5200 @ 2.27GHz
72GB RAM (ECC Always!)
APC3000 UPS (Always!)

Show : Peripherals

Intel Pro 1000 (integrated) for CIFS
(2x) Intel Pro 1000 PT/MT Duals Port (Quad ports for iSCSI)
(2x) Intel 3500 SSD SLOGs
IBM M1015 IT Mode HBA (Port 0) -> BPN-SAS2-826EL1 (12 port backplane with expander)
IBM M1015 IT Mode HBA (Port 1) -> SFF-8088 connected -> HP MSA70 3G 25 bay drive enclosure
HP SAS HBA (Port 0) -> SFF-8088 connected -> HP DS2700 6G 25 bay drive enclosure

Show : Pools

Pool1 (VM Datastore) -> 24x 3G 146GB 10K SAS into 12 vDev Mirrors
Pool2 (VM Datastore) -> 12x 6G 300GB 10K SAS into 6 vDev Mirrors
Pool3 (Media Storage) -> 8x 3G 2TB 7200 SATA into 1vDev[Z2]

Show : Network Scheme

Network Infrastructure:
Cisco SG200-26 (26 Port Gigabit Switch)

Four separate vLANs/subnets for iSCSI

• em2 - x.x.101.7/24
• em3 - x.x.102.7/24
• em0 - x.x.103.7/24
• em1 - x.x.104.7/24

Separate vLAN/subnet for CIFS (Always!)

• itb - x.x.0.7/24

I apologize if this post is too old to be replying in, but I don't remember what the limit is.

 

[bmh.01]

No I still haven't had the issue again since removing the l2arc from my pool(s) I can throw multiple svmotions at it without issue. The drives work just as well (if not better) for vFRC in front of the spinning rust and for out and out performance I decided to add a flash only pool.

Would've thought that your amount of RAM would have negated the issue perhaps but maybe there is a hard to reproduce VAAI/L2ARC issue, mine was very random how/when it would hang.

 

[sfcredfox]

bmh,

Thanks for your reply. I'm entertaining the same thing (vFRC) or a future SSD pool. Do you mind sharing the details of your SSD pool if you have time? Other than cost, which can be handled over time, I'm concerned about bandwidth of my hardware being the limiting factor for an SSD pool.

Thanks.

 

[bmh.01]

Yes no problem, the SSD pool is now 8x OCZ ARC100 240GB SSDs in striped mirrors so the pool can be expanded and (hopefully!) give max IOPs. I started with 4x and added another 4x and balanced up with svmotioning vms around.

I'm limited by 1GBe throughput wise, other than that it gives sub 1ms average latency 99% of the the time (bearing in mind this is a lab) I put most of my horizon vdi vms on there as its where you'd notice it most.

I'd like to move to 4GB FC seeing as the hardware is now so cheap but the lack of an interface in FreeNAS to configure it puts me off for the time being as i'd like to be able to configure masking etc rather than having it setup ALL/ALL.

 

[jgreco]

bmh,

Thanks for your reply. I'm entertaining the same thing (vFRC) or a future SSD pool. Do you mind sharing the details of your SSD pool if you have time? Other than cost, which can be handled over time, I'm concerned about bandwidth of my hardware being the limiting factor for an SSD pool.

Thanks.

Yeah, the most likely issue over time is that part of the idea of ZFS is that it isn't hard to make CPU faster than HDD. But it may be hard to make CPU competitive with the faster generation SSD's. The upside is that most FreeNAS installations will be limited to 20Gbps or less.

 

[sfcredfox]

Yeah, the most likely issue over time is that part of the idea of ZFS is that it isn't hard to make CPU faster than HDD. But it may be hard to make CPU competitive with the faster generation SSD's. The upside is that most FreeNAS installations will be limited to 20Gbps or less.

Are you speaking strictly to the network connectivity between targets and initiators, or between disks/pools and controllers? My gut says network connectivity between hosts or between iSCSI targets/initiators.

Since it's slightly off topic of the original post which seems to suggest that removing L2ARC from a system without enough RAM was a possible solution for the FreeNAS hang problem, I won't go too far away from OP's repurposing of the L2ARC for vFRC. Since you brought up bandwidth though, I have seen some posts on here discussing SAS bandwidth for 6G SAS, being that with more than 12? SAS disks, it's possible to oversubscribe the bandwidth of the 4 channels in a 8088 to a SAS expander. Is there merit to that? Should it be researched when implementing an all SSD pool?

I feel even with having 72GB of RAM on my system, I still feel like blaming my similar system hang on my L2ARC, but I don't know how to prove it. I just try not to vStorage Motion more than two things at once.

 

[jgreco]

Well, look at it from a different angle: if you only have 10Gbps or 20Gbps of Ethernet connectivity, there may not be a pressing need to go beyond 24Gbps of SAS connectivity (which is only half an LSI 9211-8i).

Of course, from a certain point of view it'd be most ideal to have maximal bandwidth to each SSD, which would assist in scrub and resilver operations. However, there are all sorts of things inside a system that you need to look at, including PCIe lanes (and where they come from), etc., in order to make sure you're not just moving bottlenecks around.

If you merely look at 6Gbps SATA SSD's, the speeds are only 2-3x the speeds of a modern HDD, but the speeds of something like an NVMe SSD can be 10x or more the speeds of an HDD.

 

[jgreco]

vStorage Motion

*cough*

I am hoping I'm not the only one who hates all the artificial capitalization/decapitalization and letter-adding crap. Just finished seeing NVMe, PCIe, vFRC, etc. "Storage vMotion" is obviously correct but I hate the lowercase V. Does that abbreviate as SvM? etc.