TrueNAS -> ProxMox slow network and read/write performance

[AreUSirius_]

Hey there,

i tested and googled some settings and got a little push up for write/read performance but its not that good.

Pool-Settings:

Sync Disabled was for testing normally its set to "always"

Network/Interface at test:

ProxMox Server:

Uses a Dual Port 10Gbe-LAN SFP+ too

Storage is via NFS connected but tbh my write/read is pretty slow.At the TrueNAS Server is the whole VM saved. (VZDump,ISO,Disk Image, Container)


Switch:

UniFi Switch 16XG 10G 16-Port

https://www.notebooksbilliger.de/ubiquiti+unifiswitch+16+port

--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- ---

IOzone: https://pastebin.com/KLC1VvNV

TrueNAS dd test:

dd if=/dev/zero of=/mnt/ST-01-HWDEKA7/st1-proxmox-vms/test.dat bs=2048k count=1000010000+0 records in10000+0 records out20971520000 bytes transferred in 7.992925 secs (2623760479 bytes/sec)

[~]# dd of=/dev/null if=/mnt/ST-01-HWDEKA7/st1-proxmox-vms/test.dat bs=2048k count=1000010000+0 records in10000+0 records out20971520000 bytes transferred in 3.901468 secs (5375288884 bytes/sec)

iperf from ProxMox to TrueNAS:

iperf from TrueNAS to ProxMox:

Don't know why it is just 1.7 Mbits/sec!!

This was tested at a Windows VM:

 

[cyrus104]

I had a very similar setup awhile back and ran into a similar issue, same switch too.

Have you tested different DAC cables? I got some on Amazon and they worked fine for awhile but then some of them just died and I got horrible performance in one direction.

Keep us updated with the status and I hope others are able to offer more suggestions.

 

[AreUSirius_]

I had a very similar setup awhile back and ran into a similar issue, same switch too.

Have you tested different DAC cables? I got some on Amazon and they worked fine for awhile but then some of them just died and I got horrible performance in one direction.

Keep us updated with the status and I hope others are able to offer more suggestions.

switched them out but same result :(

 

[cyrus104]

Darn, sorry. I thought it was weird to with the one way slow down. What about different switch ports. The nic drivers should be pretty basic.
Any other issues with going from truenas to another device? What about doing the test from a proxmox Linux vm?

 

[AreUSirius_]

Darn, sorry. I thought it was weird to with the one way slow down. What about different switch ports. The nic drivers should be pretty basic.
Any other issues with going from truenas to another device? What about doing the test from a proxmox Linux vm?

Everything to the TrueNAS Storage has up to 8/9 Gbits/sec but from the TrueNAS Storage to ProxMox or the VM's its just 1.7 mbit/s

 

[fahadshery]

Everything to the TrueNAS Storage has up to 8/9 Gbits/sec but from the TrueNAS Storage to ProxMox or the VM's its just 1.7 mbit/s

How are you mounting these in Proxmox? NFS or iSCSI?

 

[WallyZ]

New to Free/True NAS and I'm not a TrueNAS administrator . I have noticed similar issues on a number of True/Free NAS servers in our organization when they are running in a VM (Hyper-V). The performance tends to vary from extremely good to very bad. When it becomes bad it will remain bad regardless of load and the only fix is to reboot the host server. Our section has three TrueNAS servers all hosted on separate hardware/networks and each of them suffers from this problem. The work load performed is predominantly copying large files 1GB-8GB.
When looking at the usage metrics on the server the CPU < 3%, Disk I/O <5%, network <5% and memory is mostly free. So I can't figure out where the bottleneck is! So one moment the 10Gb network is moving 150MB/S and then it falls to 1.5MB/S with no change in work load! It will just change half way through a job for no apparent reason.

 

[jgreco]

True/Free NAS servers in our organization when they are running in a VM (Hyper-V).

Hyper-V isn't an acceptable hypervisor. The only hypervisor known to work correctly is ESXi, and some people have reported success with Proxmox.

Please re-do your virtualization following the guidance in:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

for a better experience. TrueNAS is extremely demanding on virtualization platforms; it isn't like your typical low-stress web server or other services VM.

 

[WallyZ]

Hyper-V isn't an acceptable hypervisor. The only hypervisor known to work correctly is ESXi, and some people have reported success with Proxmox.

Please re-do your virtualization following the guidance in:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

for a better experience. TrueNAS is extremely demanding on virtualization platforms; it isn't like your typical low-stress web server or other services VM.

Thank you for the link/information. Much appreciated.

 

[WallyZ]

Hyper-V isn't an acceptable hypervisor. The only hypervisor known to work correctly is ESXi, and some people have reported success with Proxmox.

Please re-do your virtualization following the guidance in:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

for a better experience. TrueNAS is extremely demanding on virtualization platforms; it isn't like your typical low-stress web server or other services VM.

Thanks for the information. If ESXi is the only hypervisor known to work then that seems like important information.

Is there any official documentation TrueNAS documentation describing the supported virtualization configuration.

 

[jgreco]

Is there any official documentation TrueNAS documentation describing the supported virtualization configuration.

Yup, it's right here between the arrows: => <= Heh.

Put simply: There are NO officially "supported virtualization configurations".

Please do not run FreeNAS in production as a Virtual Machine!

[---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file sharing (small office, documents, scratch space). THIS post is aimed somewhat more at people wanting to use FreeNAS to manage lots of storage space. ----] [---- 2022/11/19: It's...

www.truenas.com

FreeNAS and TrueNAS support are provided by these forums. iXsystems will be happy to sell you TrueNAS, but it will be bare metal on a customized TrueNAS appliance. That is the only officially supported-by-iXsystems platform.

Informally, lots of people have tried lots of things to virtualize FreeNAS, and there are limited, carefully, hard-won paths to potential success. This was really VERY rough years ago on the Sandy Bridge and older CPU's with ESXi 4, and only worked on VERY limited hw/sw configurations. CPU, mainboard, and hypervisor support has gotten substantially better over the years, such that it's generally possible to get a modern server to virtualize correctly, but there's still a "golden path" to success:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Because TrueNAS is an extremely complex system pushing a ton of server features such as storage, networking, and memory all to their limits simultaneously, it's a rough workload to virtualize. Some of us do this and run a bunch of virtualized FreeNAS hosts. I've been posting on how to do this successfully for many years. However, even today, not every hardware platform capable of running ESXi is guaranteed to be able to virtualize FreeNAS successfully or well.

Proxmox is new to the game and in the last few years, has introduced experimental PCIe passthru. This seems to make it a promising-looking hypervisor option, but it isn't clear whether there's value to this, as TrueNAS SCALE also offers KVM-based hypervisor capabilities. This will probably become mature and reliable over the next few years, and, if it works, then it's probably good for people to act as guinea pigs.

Don't let this stop you from trying other things. I've discussed Hyper-V over the years a number of times. It's gone from effectively not supporting FreeBSD/Linux at all, to now at least making some attempt to do so. Unfortunately, as I often say, "it works" is not the measure of success, because "it works poorly" is still covered by that. Microsoft doesn't have a lot of incentive for making Hyper-V into an awesome class-leading hypervisor, while VMware has been the Cadillac of hypervisors for a decade and a half.

None of this translates to official support for virtualization in FreeNAS/TrueNAS. However, you can take some comfort that many people have followed the guidance above with good-to-great success. Modern post-Nehalem Supermicro platforms, ESXi, and PCIe passthru have a high rate of success.

 

[WallyZ]

Yup, it's right here between the arrows: => <= Heh.

Put simply: There are NO officially "supported virtualization configurations".

Thanks.

"=><="

I agree looking at the TrueNAS documentation (for virtualization) it goes from a high level recommendation in the getting started->code hardware guide to no further technical information.

Really appreciate the time you have taken to respond.

 

[jgreco]

I agree looking at the TrueNAS documentation (for virtualization) it goes from a high level recommendation in the getting started->code hardware guide to no further technical information.

Yeah. Back around 2013, the prospects for successful virtualization were very poor. Since the forum here is the primary method of support for iX's free product, I got really tired of seeing person after person coming in with, for example, unstable Nehalem platform, messed up pools destroyed by RDM, etc., and as a result I wrote the original "Please do not run FreeNAS in production as a VM" article. Which was quite a bit of "do as I say not as I do" because some of my filers WERE VM's even then. But, lots of sharp edges and pointy bits.

This apparently angered the powers-that-develop over at iXsystems, because they eventually put out a blog post that backchannels inform me was intended as a rebuttal, because they really did want people to feel free to virtualize. This came across here as more of a re-butt-head move because I had already posted several other articles discussing strategies to (more) safely virtualize for different scenarios (here, here). It's also worth noting that iX made a habit of pooh-poohing actual issues we ran into here in the forums, particularly including guidance on disk controllers, where the developers maintained an attitude of "anything that works with FreeBSD is fine". The problem is that they are not actually providing SUPPORT for it; that's my job, and your job, and the job of all the other people in the forums.

I'm not funded by iXsystems to be doing this work but nevertheless I have spent many hundreds of hours over the years discussing the topic with people on the forum. I maintain a small "virtualization lab" in one of our data centers to let me play with hypervisors and experiment with things at a level that most hobbyists can't afford to, and I'm pretty serious about knowing the ins and outs of this stuff, not for your benefit, but for mine and the benefit of my commercial customers; I do infrastructure design professionally. This means that the stuff that I say is generally stuff that I am comfortable with trusting. I do my best to explain my reasoning as well, so that you may choose to disregard as you see fit, from an informed position.

Unfortunately, Hyper-V remains a really annoying thing. It's got the positive end-user usability aspects to it of Windows, but it doesn't feel like a product which is technically excellent. It's an "also-ran" in the hypervisor race.