Out of the blue vmware esx + iscsi issues

[daimi]

Poster might do OK since we were discussing SSD for pool, not L2ARC. It is not clear that lots of memory would be particularly advantageous especially if it is eating into an architecture-imposed limit like the E3's 32GB and reducing the availability to other VM's. It would at least be smart to try the SSD pool and see if the increased access speed and elimination of seek overhead overcame the small-memory performance hit; my best uninformed guess is "yes!"

As for SLOG, I still haven't found a better suggestion than abusing a BBU RAID card and hard drives. This works out neatly on an ESXi build for the local boot device and datastore and does double duty as SLOG. Cool.

- Test Data -
I have been testing zpool (zpool->NFS->ESXi->VM->windows->HD Tune) with various RAM level (from 5 GB to 25 GB) for FreeNAS VM:

For SSD zpool,
@5GB RAM = Avg Read:148.7MB/s, Avg Write:32MB/s
@25GB RAM = Avg Read:116MB/s, Avg Write:28MB/s

For HDD zpool,
@5GB RAM = Avg Read:152MB/s, Avg Write:4.3MB/s
@25GB RAM = Avg Read:120MB/s, Avg Write:4.8MB/s

For HDD zpool + SSD zil/slog,
@5GB RAM = Avg Read: 110MB/s, Avg Write: 29.7MB/s
@25GB RAM = Avg Read: 163.9MB/s, Avg Write: 34.2MB/s

- Conclusion -
You are right, as long as enough RAM is being given, more RAM does not necessary have dramatic effect on (sync) write performance. Also as cyberjock mentioned "sync write" it goes to disk VS "non-sync write" it goes to the write cache in RAM. At some later point when the write cache in RAM flushes to disk

- Questions -
Q1) As you have mentioned:

"...Such write performance even on a low end LSI RAID controller with BBU can easily run into the many hundreds of MBytes/sec...."
(from http://forums.freenas.org/threads/s...l-with-zfs-on-freenas.13633/page-3#post-81882)

"By default, FreeNAS will properly store data using sync mode for an ESXi client. That's why it is slow. You can make it faster with a SSD SLOG device. How much faster is basically a function of how fast the SLOG device is."
http://forums.freenas.org/threads/s...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

So you are saying....
(a) BBU RAID + SSD for zpool
- OR -
(b) in addition to my existing M1050 (flashed to IT mode and pass through to FreeNAS) + SSD for zpool, buy another BBU RAID + SSD for dedicated SLOG device?

Do I need to flash BBU RAID to IT mode as HBA?

Q2) How about PCIe SSD as SLOG device?

 

[jgreco]

No, you do not want to (and probably cannot) flash a BBU RAID unit to HBA mode.

Normally we do not encourage the use of a hardware RAID controller for ZFS, because there are numerous downsides and relatively few advantages. Hold that thought for a second.

The problem with SSD, even if PCIe, is that there's usually a controller and a need to commit the data to flash in order to do a proper sync write, and this is always limited by the speed of flash, even if PCIe. Over SAS/SATA there's more latency from that bus as well.

But there's nothing that's inherently wrong with the hardware RAID controller ... if used as a piece that fits into the big jigsaw puzzle in the right way.

And here's the thing. The RAM on the BBU RAID is writable at immense speed, very low latency, and once written to the RAM it is effectively stable storage. So if you have a pair of hard drives in RAID1 sitting behind a BBU RAID unit, you have a SLOG device with a bunch of desirable characteristics, including way low latency, massive endurance, and you should be able to pick drives that give you in excess of gigE speeds. The only downside is that a BBU RAID unit with a large cache is a bit pricey.

But if you're doing ESXi, you probably want a hardware RAID with BBU anyways so your local datastores aren't absolutely stinky slow. So you put ESXi on your RAID1 datastore on a BBU RAID controller, and make two virtual disks, one to hold the VM and one to hold the SLOG. You then pass through a regular controller to FreeNAS for the pool drives. It kind of seems like a waste of the hardware RAID controller, but you might not think so once you see it fly on those sync writes.

 

[daimi]

No, you do not want to (and probably cannot) flash a BBU RAID unit to HBA mode.

Normally we do not encourage the use of a hardware RAID controller for ZFS, because there are numerous downsides and relatively few advantages. Hold that thought for a second.

The problem with SSD, even if PCIe, is that there's usually a controller and a need to commit the data to flash in order to do a proper sync write, and this is always limited by the speed of flash, even if PCIe. Over SAS/SATA there's more latency from that bus as well.

But there's nothing that's inherently wrong with the hardware RAID controller ... if used as a piece that fits into the big jigsaw puzzle in the right way.

And here's the thing. The RAM on the BBU RAID is writable at immense speed, very low latency, and once written to the RAM it is effectively stable storage. So if you have a pair of hard drives in RAID1 sitting behind a BBU RAID unit, you have a SLOG device with a bunch of desirable characteristics, including way low latency, massive endurance, and you should be able to pick drives that give you in excess of gigE speeds. The only downside is that a BBU RAID unit with a large cache is a bit pricey.

But if you're doing ESXi, you probably want a hardware RAID with BBU anyways so your local datastores aren't absolutely stinky slow. So you put ESXi on your RAID1 datastore on a BBU RAID controller, and make two virtual disks, one to hold the VM and one to hold the SLOG. You then pass through a regular controller to FreeNAS for the pool drives. It kind of seems like a waste of the hardware RAID controller, but you might not think so once you see it fly on those sync writes.

Will move my question to a more relevant post

 

[cyberjock]

Can we get this thread back on-topic please? This thread is about esxi and iscsi issues from the OP....we shouldn't be working on someone else's setup in the same thread.

Thanks.

 

[jgreco]

Since I cannot even figure out what he's proposing, suits.