VMWare iSCSI POOL structure

[mrstevemosher]

Hello

We currently have a VMware iSCSI POOL thats pretty critical to us. We learned a couple days ago of what not to do with our storage system the hard way so we turned to the boards. We think we're smarter at this point and trying to re architect it starting with this POOL.

For the first POOL we are going to use 8 4.3TB HGST drives. After reading "The Path to Success for Block Storage" and "Some differences between RAIDZ and mirrors, and why we use mirrors for block storage", we think 4 striped vdevs of 2 drives each should be a fit.

Are we on the right path ? Thoughts?

DISK8 1.32M 14.5T mirror 424K 3.62T gptid/7d2d03a4-45ff-11eb-9018-002590caf794 gptid/8dca85c8-45ff-11eb-9018-002590caf794 mirror 96K 3.62T gptid/782dd941-45ff-11eb-9018-002590caf794 gptid/823a5471-45ff-11eb-9018-002590caf794 mirror 368K 3.62T gptid/40cff3ff-45ff-11eb-9018-002590caf794 gptid/70d1a551-45ff-11eb-9018-002590caf794 mirror 460K 3.62T gptid/4a26bde8-45ff-11eb-9018-002590caf794 gptid/59b109f4-45ff-11eb-9018-002590caf794

@HoneyBadger and a couple other folks were generous with their time on another thread to point us towards a small success for us. So here we are. Thank you.
Current system build is below. The only focus right now is our iSCSI VMware pools.

Supermicro X9DRD-7LN4F-JBOD
128GB DDR3 ECC
2 E5-2643 CPUs
1 Intel X520-DA2 Ethernet
1 of 3 Supermicro CSE-836 w/BPN-SAS2-EL1 Cases holds the system board
Onboard LSI SAS 2008 controller
36 4.3 HGST drives
3 Supermicro AOC-SLG3-2M2
1 2TB NvME Samsung
1 512GB NvME Samsung
2 1TB NvME Sabrent
2 256GB NvME Sabrent
1 4.3 IBM SAS SSD

 

[mrstevemosher]

Saturday we went to the office and configured the pool for VMware as listed below and we found a nice improvement over our previous attempts.

Currently we are still using the Sabrent 256GB drives for cache and a second for SLOG. These are on their way out.

Current small POOL for engineering seems to be the right stuff. No longer laggy it seems with this solution over a huge RAIDz2 POOL of 8 drives.

Thank you.

DISK8 1.32M 14.5T mirror 424K 3.62T gptid/7d2d03a4-45ff-11eb-9018-002590caf794 gptid/8dca85c8-45ff-11eb-9018-002590caf794 mirror 96K 3.62T gptid/782dd941-45ff-11eb-9018-002590caf794 gptid/823a5471-45ff-11eb-9018-002590caf794 mirror 368K 3.62T gptid/40cff3ff-45ff-11eb-9018-002590caf794 gptid/70d1a551-45ff-11eb-9018-002590caf794 mirror 460K 3.62T gptid/4a26bde8-45ff-11eb-9018-002590caf794 gptid/59b109f4-45ff-11eb-9018-002590caf794

 

[Herr_Merlin]

What kind of VMs? Lots of the same? Like VDI? Just asking because of ARC and L2ARC size. Did you check the hit ration on with the previous setup?
Additionally once you replace the slog it should be a valid solution.
What strip size did you configure for your iscsi zvol?
How are you accessing it? Mipo?
Did you think about meta data vdevs ?

 

[mrstevemosher]

Herr_Merlin
All VDI 41 Win 10 machines 100GB each. We created 2 3TB zvols and split the vms 50/50 between the 2. We did not grab any stats from the last POOL. We do have engineering and research (slobs) still to contend with.
The connection we use is a single 10G link to one of 2 ports on a Intel 520 card the other port is network. Will be replaced with some sort of bonded 10G solution to our Cisco switches. The ESXi hosts are connected 10G for both iSCSI and network with the same Intel 520 based cards as the Freenas unit. Zero mpio atm. Open to suggestions.

We did not consider meta data vdevs. This is over our head atm. Did see someplace on these boards about it. We did not dive in to deep. Did see Trunas has an option for it. That is all we know.

Thanks again.

 

[mrstevemosher]

Pop back in for a simple thank you.

Nobody is complaining about speed any more.
We used a couple 8GB DDR3 Zeus drives for LOGS and a pair of 200GB Zeus SSD Drives for cache and all my hell is gone.

 

[jgreco]

Sorry, I was bruising my tailbone on the floor at a data center for large portions of December, so I missed this, but

After reading "The Path to Success for Block Storage" and "Some differences between RAIDZ and mirrors, and why we use mirrors for block storage", we think 4 striped vdevs of 2 drives each should be a fit.

Pop back in for a simple thank you.

Nobody is complaining about speed any more.

You just made my day, and you are most welcome.

 

[HoneyBadger]

Somehow I didn't get a notification of my tagging in the main post, so I apologize for being in absentia for this thread until now.

Very glad to see things have improved in the world of iSCSI. Configuring MPIO would be a good next step for added redundancy, but might require some downtime depending on the number of interfaces available and other access methods required (eg: do you need to install a second X520?)

I would also take the 12.0-U1.1 hotfix as soon as you can, since there's a high priority data integrity fix listed there for "VMs" - while I haven't seen any issues in my small Horizon test farm, it might be that I've just been lucky this far.

 

[mrstevemosher]

Somehow I didn't get a notification of my tagging in the main post, so I apologize for being in absentia for this thread until now.

Very glad to see things have improved in the world of iSCSI. Configuring MPIO would be a good next step for added redundancy, but might require some downtime depending on the number of interfaces available and other access methods required (eg: do you need to install a second X520?)

I would also take the 12.0-U1.1 hotfix as soon as you can, since there's a high priority data integrity fix listed there for "VMs" - while I haven't seen any issues in my small Horizon test farm, it might be that I've just been lucky this far.

Upgraded to Trunas latest and greatest. Currently have a 2 Port x520 in there. I'll take your advice and see if the team cant get that up during the next change window. Also we have a couple 4 Port 520 based cards here. Now Im looking at the metadata functionality.

Boss keeps asking about an EMC or HP solution for storage. I keep asking about wage increases instead :)

 

[mrstevemosher]

Ok we're back with slow performance on this ZVOL. Should not take an hour to copy 2 100GB VMs via vMotion from one iSCSI ZVOL to another on the same POOL.

We tried just running the 10G network cable right from the ESX host to the arrays so we feel its not network. We did try the 10G tunables and nothing worked out there.

We currently added a shelf to the VMware server until we can get over this issue. We see other folks here getting 450MB plus on their copies/moves using Freenas/Trunas with VMware and iSCSI.

Veeam backups are about 100MB for transfer so they suck too.

vMware 6.7
Trunas (latest update)
Drives are al HGST 4TB
Log drive is an 8GB DDR3 RAM Disk
Cache is a ZUES 200GB drive.
We also updated the server with dual 2670v2 CPUs and 192GB of ram ( we had random reboots and data corruption but that was a troubled Trunas release. )



capacity operations bandwidth pool alloc free read write read write ---------------------------------------------- ----- ----- ----- ----- ----- ----- DISK11 20.2T 8.82T 31 17 3.97M 2.31M mirror 2.93T 716G 3 0 412K 185K gptid/66e8d910-4ee1-11eb-9018-002590caf794 - - 1 0 210K 92.5K gptid/6bd6477b-4ee1-11eb-9018-002590caf794 - - 1 0 202K 92.5K mirror 2.92T 727G 3 1 394K 192K gptid/6c8c1d98-4ee1-11eb-9018-002590caf794 - - 1 0 193K 95.9K gptid/6c98ae33-4ee1-11eb-9018-002590caf794 - - 1 0 202K 95.9K mirror 2.91T 729G 3 1 421K 194K gptid/6bb52994-4ee1-11eb-9018-002590caf794 - - 1 0 213K 97.2K gptid/6b5cb253-4ee1-11eb-9018-002590caf794 - - 1 0 209K 97.2K mirror 2.90T 740G 3 1 395K 198K gptid/668f2f50-4ee1-11eb-9018-002590caf794 - - 1 0 201K 99.0K gptid/69861aa3-4ee1-11eb-9018-002590caf794 - - 1 0 194K 99.0K mirror 2.14T 1.48T 4 1 622K 219K gptid/b2967b5c-516a-11eb-9018-002590caf794 - - 2 0 305K 109K gptid/b32d22dc-516a-11eb-9018-002590caf794 - - 2 0 317K 109K mirror 2.12T 1.51T 4 1 620K 222K gptid/1893dfa5-516b-11eb-9018-002590caf794 - - 2 0 308K 111K gptid/1a917665-516b-11eb-9018-002590caf794 - - 2 0 312K 111K mirror 2.13T 1.49T 4 1 606K 213K gptid/15e9d324-516b-11eb-9018-002590caf794 - - 2 0 303K 107K gptid/16f88a19-516b-11eb-9018-002590caf794 - - 2 0 303K 107K mirror 2.13T 1.49T 4 1 591K 203K gptid/529c2fbd-516b-11eb-9018-002590caf794 - - 2 0 296K 102K gptid/53352285-516b-11eb-9018-002590caf794 - - 2 0 295K 102K logs - - - - - - gptid/bbd05b51-7917-11eb-97e4-00e0ed506482 37.1M 6.96G 0 6 7 743K cache - - - - - - gptid/4ec31152-7917-11eb-97e4-00e0ed506482 30.1G 156G 0 4 7.35K 481K ---------------------------------------------- ----- ----- ----- ----- ----- -----

 

[mrstevemosher]

Had to call in Dell today to pitch their EMC garbage. Meh.

 

[HoneyBadger]

I'm seeing a few reports of poor iSCSI performance recently, especially on people who were previously working fine. Kind of concerned myself.

What do the latency columns (ms/r, w, d) look like under gstat -dp ?

 

[sretalla]

Log drive is an 8GB DDR3 RAM Disk

I guess you understand the risks you're playing with here... you mentioned

We currently have a VMware iSCSI POOL thats pretty critical to us

How critical exactly? if you have an unclean shutdown, you will lose transactions.


You may be slowing down your pool with the L2ARC, since all reads from the pool potentially cause writes to it. Perhaps try without it.

 

[HoneyBadger]

I guess you understand the risks you're playing with here

@mrstevemosher is using HGST ZeusRAM devices, not an md ramdrive, so that would be "none"

You may be slowing down your pool with the L2ARC, since all reads from the pool potentially cause writes to it. Perhaps try without it.

The L2ARC isn't a new addition though, and the pool was reportedly running fine beforehand. Assuming the default tunables haven't been messed with (and I don't think they were here) it's only going to feed 8MB/s, and the ZeusIOPS SSDs should be able to handle that much while still returning good read latencies.

 

[sretalla]

@mrstevemosher is using HGST ZeusRAM devices, not an md ramdrive, so that would be "none"

OK, I missed that post (saw it now)... it wasn't in the initial HW list, which I re-checked before sharing the concern.

Anyway, all good.

 

[mrstevemosher]

OK, I missed that post (saw it now)... it wasn't in the initial HW list, which I re-checked before sharing the concern.

Anyway, all good.

I appreciate you jumping in man. All of you folks are incredible.

 

[mrstevemosher]

I'm seeing a few reports of poor iSCSI performance recently, especially on people who were previously working fine. Kind of concerned myself.

What do the latency columns (ms/r, w, d) look like under gstat -dp ?

I'll grab this info later today. Driving us nuts so we stuffed a couple drives in the esxi host. All VMs are on that datastore.

Thank you sir.

 

[HoneyBadger]

Are the writes still fast to the TrueNAS-based datastore? The other threads I've seen are primarily focused on poor read performance (especially during sequential/svMotion operations)

 

[kspare]

My iscsi writes and reads are now crawling...back to FN11 we go.....

 

[mrstevemosher]

Are the writes still fast to the TrueNAS-based datastore? The other threads I've seen are primarily focused on poor read performance (especially during sequential/svMotion operations)

Sorry we havent done any work yet in the DC due to change schedules. The team is really tired of messing with it during the nights so this weekend we are all going in to address it.

We had 11.x and we corrected issues with provided feedback from the boards here. Once we created the new iSCSI targets we saw that our issues vanished.
Then we did the 12.0 upgrade. That seemed to corrupt about 20% of the VMs here and we had to do the upgrade (hotfix) to prevent that from happening again. Currently on the latest 12.0-U2.1. Since we upgraded the POOLs we cannot go back to 11.x.
The execs are hard core on the EMC/DELL garbage. All they hear is 'oh thats freenas, its open source so its going to have issues.' Although Trunas has supported enterprise products to which again my execs don't see.

We'll be back in the dc this weekend.

1) We'll vMotion the VMs from the onboard disks back to Trunas
2) We'll create a second iSCSI target and do vMotion testing from the first target to the second using 2 POOLs
3) we will also test the Veeam backups as they suck too while on Trunas.

I'm pretty sure with these boards and the brains here plus my team we will get this sorted.

 

[HoneyBadger]

We'll be back in the dc this weekend.

In all honesty, if I were in your shoes, at this point I would roll back to 11.3-U5 - you have a demanding production workload and it was working well there. Let someone without the sword of Damocles (or senior management) hanging over them fight this battle.