After assistance diagnosing VM real world performance on AIO box

[Stux]

Have you benchmarked your pool locally yet?

You need to create a test dataset with encryption off, then do a dd from /dev/zero to a file on the pool, set the block size to 1M, and the count to an equiv of 20GB or so.

And you can try it with and without sync on the dataset too

 

[Eds89]

Running

Code:

dd if=/dev/zero of=/mnt/VMs/DDTest/testfile bs=1M count=20000

On a dataset with sync disabled, gives me 2719437992 bytes/s, which unless I have my math wrong works out to 2719 Megabytes/s (2.7GB/s)?

If I then try with sync=always, I get 788765257 bytes/s, which I think is 788MB/s

I should note, this is with the SLOG still added to the pool.
Should I try again without the SLOG?

 

[Eds89]

SLOG removed, sync=always, basically 100MB/s
sync=disabled, 2.7GB/s

 

[Stux]

That's too fast... its a pair of mirrors right? So it should only be writing at a maximum of 600MB/s or so.

Did you set compression disabled on the dataset?

You can do it in the GUI, or https://www.jamescoyle.net/how-to/496-zfs-dataset-compression

 

[Eds89]

The 700MB/s figure is with the SSD SLOG still in the pool, and I could see IO on that drive of between 700MB/s and 800MB/s, which would explain that figure?

Isn't 100MB/s on target for what I should expect with sync=always, as that would be about the write speed of 1 drive?

As for the 2.7GB/s figures, I had assumed that this was because the writes were happening in RAM within the 5 second TXG period, so wasn't waiting on the disks?

I will check compressions settings in a little while and advise. I assume because we are using /dev/zero, compression has a big impact here?

 

[Eds89]

Compression was enabled.
If I disable compression, and then run same test with sync=disabled, I get about 200MB/s
sync=always results in 33MB/s

That seems horrendously low?!

 

[Chris Moore]

FreeNAS has pool of 2 mirrored vdevs consisiting of 4x 7200rpm 2TB Hitachi drives

The limit you are running into is based on the small number of spindles involved. More drives = more IOPS.

No further thoughts on whether it may be iSCSI specific?

Your iSCSI pool is starved for IOPS because there are not enough disks. I have 16 drives in my iSCSI pool and sometimes it can't keep up.

 

[Chris Moore]

If I disable compression, and then run same test with sync=disabled, I get about 200MB/s
sync=always results in 33MB/s

That seems horrendously low?!

If I recall correctly, this thread discusses the issue you are having:

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

 

[Eds89]

I just don't get it. This is saying that 4 disks in effectively raid 10 is slower than a single disk?
At this point, I'm either going to break the pool into 4 pools of single disk vdevs and put VM on each, or replace the whole lot with SSDs.

It just doesn't feel right to me.

 

[Chris Moore]

pool into 4 pools of single disk vdevs

That would be worse. It is the number of vdevs in the pool that give IOPS. More more disks gives you more vdevs which gives you more IOPS.
SSDs would always be faster, but then you have to pay for SSDs.

 

[Eds89]

But if I put one VM on each drive, and each drive is capable of 100MB/s writes, it would better wouldn't it?

It would be no different to me having a physical machine with Windows install on a single drive.

I think at this point, SSDs is my simplest solution to what I consider poor performance.

 

[Stux]

Compression was enabled.
If I disable compression, and then run same test with sync=disabled, I get about 200MB/s
sync=always results in 33MB/s

That seems horrendously low?!

Now add your SLOG back and try again

 

[Stux]

Compression was enabled.
If I disable compression, and then run same test with sync=disabled, I get about 200MB/s
sync=always results in 33MB/s

That seems horrendously low?!

It’s certainly in the ballpark.

I get circa 15MB/s direct on my single vdev, you have two, so double.

This is why you add a slog. It turns sync writes into async writes.

My 8TB 7200 rpm drives have high density and 6-way RaidZ2.

This means effectively I have 4 disks simultaneously reading/writing for sequential async streams.

My experience has been that HDs are capable of somewhere between 100% and 50% of their max performance, in ideal situations, depending on if you are writing to the inside or outside edge.

My HDs have a max write of 210MB/s
https://www.seagate.com/www-content/datasheets/pdfs/ironwolf-12tbDS1904-9-1707-GB-en_AU.pdf

So, I should get about 400-800MB/s for unfragmented async writes. And I do.

Throw in memory caching (I have 24GB of ram and can cache up to 16GB or so) and my system can offer burst performance of up to 1.7GB/s.

Not sure what your drives max speed is, but you have lower density, so less bits run under the drive head per rotation.

Maybe less platters too? Again less bits under the heads.

So, quite possible that your average drive speed is about 100MB/s.

Mirrors write at the speed of a single disk. And read at twice that.

Two mirrors. 200MB/s.

Add more mirrors and you will increase your speed.

And your IOPS.

Or just use some SSDs

HDs actually suck for anything except bulk storage.

 

[Eds89]

Thanks.

Again, this is all well and good, except for when I then compare copying the same file over the virtual switch to a similarly configured pool (2 mirrored vdevs) of slower disks, the writes are 100% consistent at 300MB/s. That's 5x faster than the speeds that the writes drop to when writing to the pool I am using for ESXi iSCSI.

I don't think the underlying disks are the problem here, as I would expect almost identical write performance from both.
I find it highly unlikely that one set of disks have 5x the write speed of another set of disks.

Even if there is a spike while some initial caching goes on, surely once the disks start to have to physically write, they should be capable of writing at at least 100MB/s?
If you're saying that with a disk write capability of about 100MB/s, when in this configuration I should expect to see a 33% drop in performance, then I don't think this realistically is the product for me without SSDs.

 

[Stux]

Your sync=always test with 33MB/s, that’s what I would expect without a slog. Did Yoh run the test with the slog?

200MB/s sync disabled.

And 171MB/s via iSCSI. Seems about right.

What block size do you have on the iSCSI zvol?

If you think the issue is with the ESXi then boot FreeNAS bare metal abs rebench it

 

[Eds89]

That's what I'm saying, I don't get 171MB/s via iSCSI, i'm only getting between 50 and 70MB/s after the initial spike.

Is there a way to check the block size via the GUI after the zvol has been created? Think I just left as defaults.

I don't think I said I thought ESXi was the issue, only that there was an issue somewhere, which is why I created the thread as a diagnosing thread.
Maybe the issue is with FreeNAS, I don't know, but it seems to only affect performance when the storage is done via iSCSI.