High iowait on TrueNAS Scale

[willtruenas111]

Hi

I experience consistently high iowait times as shown in the CPU chart below. I wanted to check if this is 1) normal? 2) how I could further debug this?

Any help would be appreciated, it's impacting the performance of a ubuntu VM I have running. All the disks I have are fairly new and listed below as well.

I've read something about HP gen8 microservers having a terrible raid controller. I don't know what a raid controller does exactly and if that might be the case in my setup.

OS Version:TrueNAS-SCALE-23.10.1.3
Product:ProLiant MicroServer Gen8
Model:Intel(R) Xeon(R) CPU E31265L @ 2.40GHz
Memory:16 GiB

2x SSDs (1 pool mirrored zfs) - a couple of VMs have zvols here.
2x HDDs (1 pool mirrored zfs) - media storage, not used much
USB boot-pool. (not ideal but I don't have a spare pool to dedicate to the boot-pool, or any more space for additional drives. I don't know enough about partitioning to carve up some of the HDD pool if thats possible)

 

[Arwen]

First, you should never use a RAID controller with ZFS, as ZFS is the RAID controller. Please run the following and put the output in CODE tags, (not screen shot).
lspci | egrep -i "SATA|SAS|RAID"

Next, USB boot devices do work okay. And it is not supported, (or recommended), to share data disks with boot pool.

 

[willtruenas111]

First, you should never use a RAID controller with ZFS, as ZFS is the RAID controller. Please run the following and put the output in CODE tags, (not screen shot).
lspci | egrep -i "SATA|SAS|RAID"

Next, USB boot devices do work okay. And it is not supported, (or recommended), to share data disks with boot pool.

Thanks for the guidance Arwen.

Output attached

 

[Arwen]

As long as you are using the built in 6 SATA ports from Intel, you should be good on that account.

I can't help further with the high I/O wait times... sorry. Perhaps someone else can assist.

Using zVols for VMs is tricky, and selecting the right block size, zVol parameters and such is not something I have done.

 

[willtruenas111]

As long as you are using the built in 6 SATA ports from Intel, you should be good on that account.

I can't help further with the high I/O wait times... sorry. Perhaps someone else can assist.

Using zVols for VMs is tricky, and selecting the right block size, zVol parameters and such is not something I have done.

No worries and thanks. Good to know the sata ports setup should be ok then.

Sometimes I wonder if could be setting this whole thing up in a better way. Like do I need the combination of zfs, vms, truenas scale. The most important part of the setup for me is turning out to be the VMs, though I like the redundancy and safety that truenas provides with the raid setup and pools.

 

[PK1048]

There have been lots of varying opinions on using zvol vs. zfs for VM storage. Overall each has pros and cons.

If you use a separate zvol per VM then you get theoretically better throughput than zfs, plus you can snapshot / clone / rollback per VM. But you get NO visibility into the underlying data.

If you use a separate zfs (dataset) per VM you get the same snapshot / clone / rollback advantages of a zvol, plus you get the visibility into the VM's files. You can use various tools to repair a .vmdk or .vdi or whatever virtual disk format you use. You do loose a very small amount of performance.

I assumed you were running the VM on the SCALE node. If you are not, the you need to share the zvol (via iSCSI) or zfs (via NFS) to the compte node where the VM is running.

In either case, adding a very fast SLOG device will help performance as VM writes are always SYNC, whether local or remote.

I am still running TruNAS CORE as a file server for 5 compute nodes (between 100 and 150 VMs) shared via NFS. We stick with the Conservative Tier ( https://www.truenas.com/software-status/ ) so have not migrated to SCALE yet.

 

[ChrisRJ]

Sometimes I wonder if could be setting this whole thing up in a better way. Like do I need the combination of zfs, vms, truenas scale. The most important part of the setup for me is turning out to be the VMs, though I like the redundancy and safety that truenas provides with the raid setup and pools.

In that case I would look at Proxmox or perhaps even XCP-ng.

Why did you choose TrueNAS in the first place?

 

[willtruenas111]

Thanks for the suggestion of adding a SLOG device, I will aim to do this in future. Though for now I think what I'm experiencing is far worse than expected:

Code:

sudo dd if=/dev/zero of=test bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 503.408 s, 2.1 MB/s

The poor speed is only on the SSD pool. I get normal speeds on the other pool.

As for using TrueNAS, thats just because its first use was as a Plex server with lots of media files. So it was more of a NAS utility back then. I would like to look at proxmox in future for sure, since my use case has moved over more to virtualisation.