Network connection keeps dropping during outbound file copy

[Spifer]

Hi all - I've spent the last week searching and trying everything I can find on this, but I'm at a loss at this point. I'm 1.1Tb into a 1.4Tb data transfer and I can't get to the finish line.

I'm trying to copy all of my media onto an external HDD for offsite backup. It made it through the first 1.1Tb without issue before stalling with TeraCopy saying the files are offline. When it happens I lose TrueNAS both through Windows Explorer and through the web interface. The ESXi web interface also stops responding but I don't lose connection to game servers or the pi-hole that I have running in VM's, which I don't understand. I've been running TrueNAS core for about a year now without issue until this. Restarting the machine, TrueNAS, or just reseating the network cable reestablishes the connection.

So far what I've found online are (assuming these are still correct) that TrueNAS doesn't get along with Macs or non-intel NICs. I don't have any macs on my network. The machine does have one built in NIC with a Marvel driver, but the network cables (two for redundancy) were never connected to that one and it's now disabled in the bios. The network cables were connected to one of the Intel NICs but are now split between both of them. (something that I needed to do anyway for redundancy). Also, there's no messages in /var/log/messages as far as I can tell. (I'm definitely a novice and TrueNAS, FreeBSD, and CLI.)

I've tried;
Restarting all machines, including network equipment.
Disabling the non-intel nic that wasn't being used anyway in the BIOS
switch the TrueNAS VM to a new vNIC that's Intel (E1000e) instead of the default VMWare one (VMXNET3)
Moving one connection to the other Intel NIC to rule out bad NIC
Copying to other hard drives and other computers, all fail the same way.

I can copy large amounts of data to TrueNAS and I can copy large amounts of data onto or from the ESXi datastores and other VM's on the same machine. It will stay connected and act normally as long as I don't try to copy files out of TrueNAS, but it almost instantly goes offline again if I try to copy any files off of TrueNAS. If I'm really lucky it will go for a 1-2 Gb before losing the connection, or I'm not and it craps out at 20Mb. It's usually good for ~100-200Mb.

The machine:
R710 w/
ESXi running TrueNAS-12.0-U8 with 16Gb of ram allocated
1 integrated NIC - qlogic netxtreme ii bcm5709 w/ a Marvel driver
2 add-in NICs - both are intel 82576


I think that's everything, the bulk of the troubleshooting was in the first three days, the last three have mostly been researching in-between family and work hours.

Please let me know if I'm forgetting anything. Any/all help is appreciated. Thank you.

 

[jgreco]

You haven't really described what you're doing for TrueNAS as a VM. There are lots of ways to do it wrong, ranging from vaguely bad to catastrophically disastrous. Please make sure you've followed the guidance at

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

If problems persist, eliminate ESXi from the picture. Install TrueNAS on a USB thumb drive, download your configuration from the VM TrueNAS, shut down ESXi, boot the TrueNAS thumb drive in its place, and then upload the configuration to the bare metal TrueNAS. See if the problem persists. If it does, we can help with that. If not, you are probably doing something bad with ESXi networking. Stuff happens. I'm fighting with ESXi and i40gb driver/firmware issues on some hosts right now.

 

[Spifer]

I appreciate your fast response and trying to help.

I"m not exactly sure what you mean by "what you're doing for TrueNAS as a VM." If you mean why am I using a VM instead of installing it to bare metal, because I wanted ESXi has my hypervisor but I still wanted an simple to set up data store and to learn about TrueNAS.

It's not ESXi networking. That one's been tested thoroughly.

Judging by that link you posted it sounds like it could just be that TrueNAS is unstable as a VM and can run into random problems I guess?
If that's the case, I guess I can accept that it's a bad idea and I just didn't research enough to make it past the how to do it. It's just annoying that it worked so well for that long and then out of nowhere started doing this without changing anything to cause it.

So there's nothing else besides the Mac and non Intel NIC issues on the TrueNAS side of things that might cause that behavior?

 

[jgreco]

TrueNAS is designed to be installed on bare metal and to manage storage. ZFS is very demanding in a variety of ways, and can be unforgiving to naive virtualization strategies. TrueNAS can be completely stable as a VM when done correctly, but lots of TrueNAS VM's are not stable when they're done incorrectly. Your failure to provide any details is a commonality with many other users who have come with vague virtualization issues. I may be the person who's spent more time helping with virtualization of FreeNAS/TrueNAS than anyone else on the planet, but I tired of pulling teeth to get information years ago.

 

[Spifer]

Impressive response, it’s been a while since someone’s used that many words to tell me I’m too ignorant to be worth their time, lol, you sound more burned out than me dealing with the "pre-computer generation" at work sometimes. That aside though, your apparent expert opinion that it is a virtualization issue is both noted and appreciated. That being the problem, it’s not worth the effort to fix unfortunately. Especially since it sounds like I won’t be reusing TrueNAS now.

Just an friendly aside... My failure to provide the correct details was due to gross ignorance of the subject matter. You seem to have assumed that I posses a base level of knowledge allowing me to narrow it down to a virtualization issue despite obviously lacking the experience with TrueNAS to thoroughly rule out settings or hardware compatibility issues. I many not know TrueNAS, but I do know enough about troubleshooting to not assume anything. But who knows, maybe that was only as obvious to you as the problem being a virtualization issue was to me. Either way, have a nice life.

 

[jgreco]

Impressive response, it’s been a while since someone’s used that many words to tell me I’m too ignorant to be worth their time, l

If I felt you were too ignorant to be worth my time, I wouldn't even bother to reply.

Just an friendly aside... My failure to provide the correct details was due to gross ignorance of the subject matter.

If you decide that you'd like to isolate your issues, please do feel free to go back up to the post I linked above, and make sure your virtualization setup is compliant. That's a critical starting point. That's also why I linked to the article. I write these things so that people who do not have the information can learn the things that they need to be informed of. It's my mission to educate.

That aside though, your apparent expert opinion that it is a virtualization issue is both noted and appreciated. That being the problem, it’s not worth the effort to fix unfortunately.

It's been some time since someone was so determined to simultaneously fail to provide requested information, not read the provided material, not try the suggested course of action, and then manage to take offense while also putting an opinion in my mouth that I didn't say and jumping to conclusions like "it's not ESXi networking". I never said it was a virtualization issue. However, it very well could be. These things are complicated. That's why you were being asked for stuff and pointed at resources.

But I do learn from my mistakes. It appears to me that you're resistant to help. So, by all means, as you say,

have a nice life.

If you wish to constructively continue this conversation and work towards solving your problems, then please consider rewinding to the top, doing what was asked, and we can certainly still move forward. I cannot singlehandedly diagnose your problem without you doing the things that are asked. You are the eyes and ears here. You've presented some puzzling and contradictory issues such as the "ESXi web interface stops responding" which is suggestive of certain types of ESXi networking issues. This makes it extra important to start crossing t's and dotting i's.

 

[Spifer]

Ok, starting off, I'll apologize for reacting poorly to, what I at least read to be, a dismissive and condescending post in your second post. (I'm not accusing it of being that now, just giving context to my own poor behavior.)

tl'dr, as much as I'd love to dive into fixing this for the sake of learning and not letting it beat me, I just don't have the time to devote to it at this point that it sounds like it will need and I don't want to waste more of yours. I'll just move the files over a little bit at a time and maybe try TrueNAS later on when I can meet all the criteria in the article. I do truly thank you for your time and willingness to help though.


The rest is just trying to address a few points for understanding if you're interested. (also, any frustration in it is at me not you)


I still don't understand what details I failed to provide. Your second post sounds like frustration at somebody missing something so basic and obvious.

You haven't really described what you're doing for TrueNAS as a VM.

This sounded like you needed more information, but I wasn't sure what exactly. I led with stating my lack of understanding in case I was reading it wrong before trying to respond to that without text-walling every last detail of how it was installed (that was the only other thing I could think of) Maybe I should have finished with asking outright if that's what you meant or something, I don' know.

Maybe you meant that I didn't do and report back on this one?

If problems persist, eliminate ESXi from the picture. Install TrueNAS on a USB thumb drive, download your configuration from the VM TrueNAS, shut down ESXi, boot the TrueNAS thumb drive in its place, and then upload the configuration to the bare metal TrueNAS. See if the problem persists. If it does, we can help with that. If not, you are probably doing something bad with ESXi networking. Stuff happens. I'm fighting with ESXi and i40gb driver/firmware issues on some hosts right now.

That sounds like a really interesting thing that would be cool to do, but also complicated and risky without having everything backed up. Maybe it's not, and I realize now that I didn't speak to this paragraph directly, but that's why I was trying confirm that you definitely thought it was something in the virtualization process / VM and not a setting or hardware thing that might be simpler to address first.

To the articles you linked. I somehow missed that you wrote the one "Virtually FreeNAS ... an alternative for those seeking virtualization". (regardless, I'd meant to thank you for those) That one I found especially informative and wish that I'd seen it sooner. I saved it for if/when I do ever try TrueNAS again. I don't currently have two separate redundant datastores to create the virtual drives on to do it right.
Oh, if by details, you meant how my setup diverged from those links.
1 - no PCI-Passthrough storage
2 - VM files are on a non-redundant drive
3 - VM currently only has a single vdisk
4 - only have one redundant datastore ATM. (I know you wrote it will technically work, but I feel like it doesn't matter until I can fix points 2 & 3)

ESXi networking issues - yeah, I didn't want to text-wall you unnecessarily. I'm still pretty new at that too, but I have a friend that's been doing it as a profession and hobby for a lot of years back me up on triple checking everything and confirming it all worked right. He's doesn't know TrueNAS though. (and yes I do agree, it's still possible it's an ESXi network thing because nothing is 100%. It's just not nearly as likely because it's been checked by somebody that actually knows what there doing, where as the TrueNAS part has just been me and google).

Thanks again for your time.

Have a wonderful life
(I meant it the first time too, but this time I mean with a smile )

 

[Samuel Tai]

Actually, what we're looking for is the exact setup of your VM and the host hardware. More info is better, and it's usually the case folks asking for help don't provide enough info initially to help us pinpoint where the problem could be.

Now, how are you providing storage to your VM? The supported method is to passthrough a PCI HBA to the VM, so TrueNAS ZFS can directly control drives. If you're doing things like RDM, this is known to have problems losing data, as well as performance issues keeping up with transfers in/out. If you're attaching vdisk files, that's even less performant.

TrueNAS has the VMXNET3 driver compiled in-kernel, so that's going to be the best performing option.