Update Logs for 9.3?

[mattlach]

Hey all,

I installed three updates the other day using the update system in the web gui. In my rush I forgot to take notes of what they were.

I've had a detrimental impact to my virtual 10Gbit ethernet speeds since those updates, and I am trying to troubleshoot it.

Where might I find the logs of those updates so I can determine what they were, and what might have changed?

Thanks,
Matt

 

[cyberjock]

YOu'll have to check your boot environments to see what versions you were on and what you went to, and bugs.freenas.org (or git) should show you what was changed/updated.

 

[mattlach]

Ahh, right, because updates replace the partition, I wouldn't expect to find anything in /var/log, right?

My issues have been with VMXNET3 under ESXi 5.5 (I believe I read some testing you did there at one point, or maybe that was jgreco, can't remember. Early 2013 I think, can't find the thread right now)

iperf speed test last week before the update:

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 5] local xxx.xxx.xxx.xxx port 5001 connected with xxx.xxx.xxx.xxx port 27820
[ 5] 0.0-10.0 sec 21.4 GBytes 18.4 Gbits/sec

iperf this week after the update:

Code:

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.xxx.xxx port 36620 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.46 GBytes 2.97 Gbits/sec

I'm wondering if this new version has its own VMXNET3 driver, and it is using it instead of the one I installed from VMWARE's package.

 

[zambanini]

the toe kernel module was added the last week. but this should not hurt.

 

[mattlach]

the toe kernel module was added the last week. but this should not hurt.

Hmm, well that DOES have potential, as trying to do any kind of hardware offloading to a paravirtual network adapter could have wonky results. If compatible I would imagine it just moving the CPU load from the FreeNAS guest to the host, but the FreeNAS guest may be better at handling it on its own.

Do you off hand know the name of the module?

is it:

/boot/kernel/toecore.ko?

I don't see toecore in my kldstat, so if that is the correct module it probably isn't what is impacting me.

Thank you for the suggestion though!

--Matt

 

[mattlach]

Hmm, well that DOES have potential, as trying to do any kind of hardware offloading to a paravirtual network adapter could have wonky results. If compatible I would imagine it just moving the CPU load from the FreeNAS guest to the host, but the FreeNAS guest may be better at handling it on its own.

Do you off hand know the name of the module?

is it:

/boot/kernel/toecore.ko?

I don't see toecore in my kldstat, so if that is the correct module it probably isn't what is impacting me.

Thank you for the suggestion though!

--Matt

On that note, I see that the TSO4 and TSO6 options are enabled for the vmxnet3 devices in ifconfig. I presume these are the LSO/TSO options for IPv4 and IPv6 respectively, is that correct?

This could be a potential. I don't know if they were listed before or not.

Does anyone know how I can pass manual options to disable them and see if it helps?

Thanks,
Matt

 

[mattlach]

On that note, I see that the TSO4 and TSO6 options are enabled for the vmxnet3 devices in ifconfig. I presume these are the LSO/TSO options for IPv4 and IPv6 respectively, is that correct?

This could be a potential. I don't know if they were listed before or not.

Does anyone know how I can pass manual options to disable them and see if it helps?

Thanks,
Matt

Well, that was another solid theory, but I disabled it with "ifconfig <device> -tso4 -tso6" followed by another test, with no change.

I am open to any suggestions!

 

[zambanini]

boot with one of the old fn9.3 snapshots, check ifconfig and kernel modules. otherwise i would suggest open a bug report.

in the last week was nothing else on the changelog. really get sure which version you had before.

 

[mattlach]

boot with one of the old fn9.3 snapshots, check ifconfig and kernel modules. otherwise i would suggest open a bug report.

Thank you

Yep, that's what I plan on doing once I have the chance. I have my MythTV PVR/DVR backend recording to FreeNAS though, so I haven't been able to take it down lately for testing. (requires no one watching TV and no shows currently being recorded) so my downtime/maintenance time is limited.

in the last week was nothing else on the changelog. really get sure which version you had before.

yeah, unfortunately that's the problem, and was why I was hoping to find it in a log file somewhere :p

I should have copied and pasted the installation notes when I installed them. There were three packages, one FreeNAS base, on Freenasgui (I think) and I can't for the life of me remember what the third was.

I DID image my boot drive before upgrading (as I always do just in case) though, so I can always go back.

 

[mattlach]

Interesting discovery.

kldstat shows me that I have two vmxnet driver modules loaded at the same time.

One is vmxnet3.ko, (37k in size) one I loaded myself from the VMWare BSD driver package, and located in /boot/modules (where I remember putting it)

Another is just vmxnet.ko (20k in size) located in /usr/local/lib/vmware-tools/modules/drivers which I don't remember doing anything with.

I don't know if they are somehow conflicting with each other, or if both were even there before. Booting from the older image will probably tell me more.

Though, it is a little suspicious that /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko is dated March 17th at 5:30pm, the exact date and time I ran the last update.

I'm guessing this file was added during that update.

 

[mattlach]

Alright,

so I just reverted to the system the way it was before the upgrade. The conclusion? The issue has absolutely nothing to do with the upgrade. The problem was there before. But then how did I measure really high speeds before?

Well, I discovered a different issue:

Look at this:

Two Iperf tests with FreeNAS as a server (iperf -s) and Ubuntu Server as client (iperf -c):

Code:

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 33485 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.79 GBytes  3.25 Gbits/sec

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 43405 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.69 GBytes  3.17 Gbits/sec

Two Iperf tests with Ubuntu Server as a server (iperf -s) and FreeNAS as client (iperf -c):

Code:

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 53941 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  24.9 GBytes  21.4 Gbits/sec

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 64555 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  25.5 GBytes  21.9 Gbits/sec

So,
FreeNAS Server, Ubuntu Client: Terribly slow.
Ubuntu Server, FreeNAS Client: Blisteringly fast



And this is repeatable every time.

When I did the initial test before the upgrade, I must just have tested with Ubuntu as the server, and more recent tests I tested with FreeNAS as the server.

Sooo..... Can anyone explain this?

I'll send you a package of Jammie Dodgers!

 

[mattlach]

And what do you know, the same is true over the physical adapters with transceivers and fiber.

When my linux worstation is the server, I practically max out the 10GBASE-SR.

Code:

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 62948 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.8 GBytes  9.30 Gbits/sec

But when FreeNAS is the server I'm back down to slow speeds.

Code:

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 40512 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.64 GBytes  3.13 Gbits/sec

I wonder if this is an iperf anomaly, or if the same is evident with file sharing protocols...

 

[mattlach]

I just noticed the TCP window size difference.

Testing to see if that makes the difference.

 

[cyberjock]

iperf in VMs is horribly horribly unreliable. It's actually virtually useless.

The problem can stem from all sorts of things. For example, if the FreeNAS VM isn't doing enough work to make the ESXi hypervisor realize the VM needs more CPU power (maybe even clock up the CPU from some really low frequency) then the iperf test will suck.


You can mitigate some of the problems by forcing recources to the VM.

 

[mattlach]

I just noticed the TCP window size difference.

Testing to see if that makes the difference.

Alright. There is obviously something wrong with iperf. Trying to specify TCP window size with -w gives me numbers all over the place, not what I requested.

 

[mattlach]

iperf in VMs is horribly horribly unreliable. It's actually virtually useless.

The problem can stem from all sorts of things. For example, if the FreeNAS VM isn't doing enough work to make the ESXi hypervisor realize the VM needs more CPU power (maybe even clock up the CPU from some really low frequency) then the iperf test will suck.


You can mitigate some of the problems by forcing recources to the VM.

That's a little bit odd since Vmware actually recommends iperf as a diagnosis tool:
http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2001003

 

[cyberjock]

That it does. But they don't tell you all the possible caveats and other problems that can occur. They expect you to know all the ins and outs and understand what the tool actually does, how it accomplishes it, and understand the meaning besides it just being a short burst number. ;)