Performance Monitoring of files.

[kspare]

In windows you have the performance monitor. If you select disks you are able to quickly diagnose what apps are creating io on the local disk.

I'm looking to do the same thing for NFS so I can track down/monitor how much IO a vm is creating on my storage servers.

How would I go about doing this?

 

[toadman]

What did google say when you searched this topic?

Google told me to try top -m io -o write -s 1

 

[kspare]

Obviously I did not find the answer because i'm here.

That command doesn't help me.

 

[toadman]

Obviously I did not find the answer because i'm here.

That command doesn't help me.

Gotcha. Why? Not possible to tie a process to a specific VM? (I'm curious as I don't run VMs on freenas.)

 

[kdragon75]

Yeah, we need way more information about what your running (vmware/kvn/bhyve/zen/huper-v/etc) and exactly what your trying to do before we can offer any useful information. @toadman was doing the best he could given what you provided.

 

[toadman]

Yeah, we need way more information about what your running (vmware/kvn/bhyve/zen/huper-v/etc) and exactly what your trying to do before we can offer any useful information.

+1

Actually, I think the command given does do the requested "diagnose what apps are creating io on the local disk." At least on my system it shows I/O by process (command).

I assumed he was talking about VMs running on Freenas. If he is hosting say ESXi datastores then the command I gave would show the nfs or iscsi process. But who knows. :)

 

[kspare]

Gotcha. Why? Not possible to tie a process to a specific VM? (I'm curious as I don't run VMs on freenas.)

I have a cluster of 16 hosts, and use 3 freenas hosts as my shared storage. We will often see spikes in diskio and network io via the freenas graphical stats, we're looking for a way to track down the io to a specific virtual machine. We are running ESXI 6.0 with 6.5 around the corner for a scheduled upgrade.

 

[kspare]

+1

Actually, I think the command given does do the requested "diagnose what apps are creating io on the local disk." At least on my system it shows I/O by process (command).

I assumed he was talking about VMs running on Freenas. If he is hosting say ESXi datastores then the command I gave would show the nfs or iscsi process. But who knows. :)

Sorry for the lack of detail. It made sense in my head, heh. The command just shows the nfsd as the process using the io, i've found other commands that will show the individual disk io, and the different volumes. But I need a more granular report so I can see that file either read or write is generating IO.

 

[toadman]

Ah. Yes, that's what I'd expect then, nfsd showing up.

I would think you could view what you are after within vCenter or VCSA (whatever you are using) depending on what your operations tools look like? Or via some netmon tools if you have those instrumented on the network.

For example, I can get disk and network i/o from vsphere on a per VM basis. If you really have one that is dominating the usage it would probably show up. I suppose it depends on how long it's sustained, the granularity of the data collection, etc. So not sure it would work for your issue.

 

[Elliot Dierksen]

This didn't generate as much info as I hoped, but perhaps it is a start.

Code:

root@freenas2:/tmp # ps auxw | grep nfs
root		3096	0.0  0.0  55484   2252  -  Is	3Oct18	  0:00.02 nfsd: master (nfsd)
root		3097	0.0  0.0   6256   2084  -  S	 3Oct18	 78:10.30 nfsd: server (nfsd)
root	   92289	0.0  0.0   6696   2580  0  S+   15:09		0:00.00 grep nfs
root@freenas2:/tmp # procstat -f 3096 3097
  PID COMM				FD T V FLAGS	REF  OFFSET PRO NAME
 3096 nfsd			  text v r r-------   -	   - -   /usr/sbin/nfsd
 3096 nfsd			   cwd v d r-------   -	   - -   /
 3096 nfsd			  root v d r-------   -	   - -   /
 3096 nfsd				 0 v c rw------   6	   0 -   /dev/null
 3096 nfsd				 1 v c rw------   6	   0 -   /dev/null
 3096 nfsd				 2 v c rw------   6	   0 -   /dev/null
 3096 nfsd				 3 v r rw------   2	   0 -   /var/db/nfs-stablerestart
 3096 nfsd				 4 v r rw------   2	   0 -   /var/db/nfs-stablerestart.bak
 3096 nfsd				 5 s - rw------   1	   0 TCP 0.0.0.0:2049 0.0.0.0:0
 3096 nfsd				 6 s - rw------   1	   0 TCP ::.2049 ::.0
 3097 nfsd			  text v r r-------   -	   - -   /usr/sbin/nfsd
 3097 nfsd			   cwd v d r-------   -	   - -   /
 3097 nfsd			  root v d r-------   -	   - -   /
 3097 nfsd				 0 v c rw------   6	   0 -   /dev/null
 3097 nfsd				 1 v c rw------   6	   0 -   /dev/null
 3097 nfsd				 2 v c rw------   6	   0 -   /dev/null
 3097 nfsd				 3 v r rw------   2	   0 -   /var/db/nfs-stablerestart
 3097 nfsd				 4 v r rw------   2	   0 -   /var/db/nfs-stablerestart.bak

 

[Elliot Dierksen]

I had another idea, but it also didn't tell me as much as I had hoped.

Code:

root@freenas2:/mnt/RAIDZ2-I/VMWare/alfred-2015 # fstat -f /mnt/RAIDZ2-I/VMWare
USER	 CMD		  PID   FD MOUNT	  INUM MODE		 SZ|DV R/W
root	 fstat	  93601   wd /mnt/RAIDZ2-I/VMWare   3339 drwxr-xr-x	  25  r
root	 csh		25025   wd /mnt/RAIDZ2-I/VMWare   3339 drwxr-xr-x	  25  r
root@freenas2:/mnt/RAIDZ2-I/VMWare/alfred-2015 # cd /
root@freenas2:/ # fstat -f /mnt/RAIDZ2-I/VMWare
USER	 CMD		  PID   FD MOUNT	  INUM MODE		 SZ|DV R/W
root@freenas2:/ # fstat -f /var
USER	 CMD		  PID   FD MOUNT	  INUM MODE		 SZ|DV R/W
elliot   ksh93	  71307	3 -		   326 -rw-------	  10 rw
elliot   ksh93	  25006	3 -		   329 -rw-------	   6 rw
root	 collectd   96506   wd /var		 71 drwxr-xr-x	  64  r
root	 cron		5753   wd /var		 96 drwxr-x---	  64  r
root	 cron		5753	3 /var		322 -rw-------	   4  w
root	 zfsd		5737   10 /var		 56 -rw-------	   4  w
root	 consul	  4886	5 /var		315 -rw-------  25169920 rw
root	 consul	  4886	6 /var		351 -rwxr-xr-x   29270 rw
root	 consul	  4886   10 /var		321 -rwxr-xr-x   29274 rw
root	 daemon	  4885	4 /var		309 -rwxr-xr-x	   4  w
root	 daemon	  4878	3 /var		308 -rw-------	   4  w
root	 python3.6   4848	3 /var		328 -rw-rw-r--	   4 rw
ladvd	ladvd	   4810   wd /var		 43 dr-xr-xr-x	   0  r
ladvd	ladvd	   4810 root /var		 43 dr-xr-xr-x	   0  r
ladvd	ladvd	   4810 jail /var		 43 dr-xr-xr-x	   0  r
ladvd	ladvd	   4810	3 /var		305 -rw-r--r--	   5  w
root	 ladvd	   4809	3 /var		305 -rw-r--r--	   5  w
root	 daemon	  4585	3 /var		270 -rw-------	   4  w
root	 winbindd	3394	4 /var		264 -rw-r--r--	  21 rw
root	 winbindd	3394	8 /var		236 -rw-r-----   12288 rw
root	 winbindd	3394   13 /var		251 -rw-r--r--	   5  w
root	 winbindd	3394   18 /var		239 -rw-r--r--   12288 rw
root	 winbindd	3394   26 /var		237 -rw-r--r--  135168 rw
root	 winbindd	3391	4 /var		259 -rw-r--r--	  20 rw
root	 winbindd	3391	8 /var		236 -rw-r-----   12288 rw
root	 winbindd	3391   13 /var		251 -rw-r--r--	   5  w
root	 winbindd	3391   18 /var		239 -rw-r--r--   12288 rw
root	 winbindd	3391   26 /var		237 -rw-r--r--  135168 rw
root	 smbd		3390	4 /var		258 -rw-r--r--	  20 rw
root	 smbd		3390	8 /var		236 -rw-r-----   12288 rw
root	 smbd		3390   12 /var		243 -rw-r--r--	   5  w
root	 smbd		3390   17 /var		245 -rw-------   20480 rw
root	 smbd		3390   18 /var		246 -rw-------	4792 rw
root	 smbd		3390   19 /var		247 -rw-------	4792 rw
root	 smbd		3390   20 /var		248 -rw-r--r--  122120 rw
root	 smbd		3390   21 /var		249 -rw-r--r--  122120 rw
root	 smbd		3390   22 /var		256 -rw-r--r--	4792 rw
root	 smbd		3389	4 /var		257 -rw-r--r--	  20 rw
root	 smbd		3389	8 /var		236 -rw-r-----   12288 rw
root	 smbd		3389   12 /var		243 -rw-r--r--	   5  w
root	 smbd		3389   17 /var		245 -rw-------   20480 rw
root	 smbd		3389   18 /var		246 -rw-------	4792 rw
root	 smbd		3389   19 /var		247 -rw-------	4792 rw
root	 smbd		3389   20 /var		248 -rw-r--r--  122120 rw
root	 smbd		3389   21 /var		249 -rw-r--r--  122120 rw
root	 smbd		3389   22 /var		256 -rw-r--r--	4792 rw
root	 winbindd	3388	4 /var		255 -rw-r--r--	  21 rw
root	 winbindd	3388	8 /var		236 -rw-r-----   12288 rw
root	 winbindd	3388   13 /var		251 -rw-r--r--	   5  w
root	 winbindd	3388   18 /var		239 -rw-r--r--   12288 rw
root	 winbindd	3383	4 /var		252 -rw-r--r--	  20 rw
root	 winbindd	3383	8 /var		236 -rw-r-----   12288 rw
root	 winbindd	3383   13 /var		251 -rw-r--r--	   5  w
root	 winbindd	3383   18 /var		239 -rw-r--r--   12288 rw
root	 winbindd	3383   26 /var		237 -rw-r--r--  135168 rw
root	 smbd		3378	4 /var		244 -rw-r--r--	  20 rw
root	 smbd		3378	8 /var		236 -rw-r-----   12288 rw
root	 smbd		3378   12 /var		243 -rw-r--r--	   5  w
root	 smbd		3378   17 /var		245 -rw-------   20480 rw
root	 smbd		3378   18 /var		246 -rw-------	4792 rw
root	 smbd		3378   19 /var		247 -rw-------	4792 rw
root	 smbd		3378   20 /var		248 -rw-r--r--  122120 rw
root	 smbd		3378   21 /var		249 -rw-r--r--  122120 rw
root	 smbd		3378   22 /var		256 -rw-r--r--	4792 rw
root	 smbd		3378   23 /var		239 -rw-r--r--   12288 rw
root	 smbd		3378   26 /var		237 -rw-r--r--  135168 rw
root	 smbd		3378   31 /var		265 -rw-------	4792 rw
root	 nmbd		3374	5 /var		238 -rw-r--r--	  21 rw
root	 nmbd		3374	8 /var		236 -rw-r-----   12288 rw
root	 nmbd		3374   11 /var		235 -rw-r--r--	   5  w
root	 nmbd		3374   16 /var		239 -rw-r--r--   12288 rw
uucp	 upsd		3279   wd /var		 86 drwxrwx---	 320  r
uucp	 snmp-ups	3277   wd /var		 86 drwxrwx---	 320  r
root	 powerd	  3176	3 /var		221 -rw-------	   4  w
root	 rpc.statd   3100	8 /var		219 -rw-r--r--	 256 rw
root	 nfsd		3097	3 /var		217 -rw-------	   0 rw
root	 nfsd		3097	4 /var		218 -rw-------	   0 rw
root	 nfsd		3096	3 /var		217 -rw-------	   0 rw
root	 nfsd		3096	4 /var		218 -rw-------	   0 rw
root	 mountd	  3090	3 /var		216 -rw-------	   4  w
root	 watchdogd   2834	5 /var		213 -rw-------	   4  w
root	 rpcbind	 2711	3 /var		211 -r--r--r--	   0  r
root	 syslog-ng   2704   wd /var		 61 drwxr-xr-x	1984  r
root	 syslog-ng   2704   15 /var		207 -rw-------   16384 rw
root	 syslog-ng   2704   21 /var		349 -rw-------  22364308  w
root	 devd		2437	7 /var		196 -rw-------	   4  w
root	 moused	  2420	5 /var		195 -rw-------	   4  w
root	 moused	  2396	5 /var		194 -rw-------	   4  w
root	 moused	  2372	5 /var		193 -rw-------	   4  w
root	 daemon	   262	3 /var		117 -rw-------	   3  w
root@freenas2:/ #

Where I was going with all that was to see if the more vague fstat would give me that information. It didn't but I am beginning to suspect that this is because the FreeNAS box doesn't actually consider the NFS files to be open. It only showed that dataset as having files open when I was in that directory. I suspect that wouldn't be as much an issue from a CIFS share, but I that is just a guess.

 

[Elliot Dierksen]

I think this confirms that FreeNAS/FreeBSD doesn't see those NFS mounted files as open.

Code:

root@freenas2:/ # ps auxw | grep nfs
root		3096	0.0  0.0  55484   2252  -  Is	3Oct18	  0:00.02 nfsd: master (nfsd)
root		3097	0.0  0.0   6256   2084  -  S	 3Oct18	 78:12.84 nfsd: server (nfsd)
root	   93771	0.0  0.0   6696   2580  0  S+   15:39		0:00.00 grep nfs
root@freenas2:/ # lsof -p 3096 -p 3097
lsof: WARNING: compiled for FreeBSD release 11.0-RELEASE; this is 11.1-STABLE.
COMMAND  PID USER   FD   TYPE			 DEVICE SIZE/OFF   NODE NAME
nfsd	3096 root  cwd   VDIR	  86,1804206330	   28	  4 /
nfsd	3096 root  rtd   VDIR	  86,1804206330	   28	  4 /
nfsd	3096 root  txt   VREG	  86,1804206330	23760 139120 /usr/sbin/nfsd
nfsd	3096 root  txt   VREG	  86,1804206330   139832   5077 /libexec/ld-elf.so.1
nfsd	3096 root  txt   VREG	  86,1804206330  1671232   4999 /lib/libc.so.7
nfsd	3096 root  txt   VREG	  86,1804206330	27544  23551 /usr/local/lib/nss_mdns.so.1
nfsd	3096 root  txt   VREG	  86,1804206330	44296  11532 /usr/local/lib/libdns_sd.so.1
nfsd	3096 root	0u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3096 root	1u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3096 root	2u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3096 root	3u  VREG	 255,2264924163		0	217 /var/db/nfs-stablerestart
nfsd	3096 root	4u  VREG	 255,2264924163		0	218 /var/db/nfs-stablerestart.bak
nfsd	3096 root	5u  IPv4 0xfffff801606dc410	  0t0	TCP *:nfsd (LISTEN)
nfsd	3096 root	6u  IPv6 0xfffff801608f1410	  0t0	TCP *:nfsd (LISTEN)
nfsd	3097 root  cwd   VDIR	  86,1804206330	   28	  4 /
nfsd	3097 root  rtd   VDIR	  86,1804206330	   28	  4 /
nfsd	3097 root  txt   VREG	  86,1804206330	23760 139120 /usr/sbin/nfsd
nfsd	3097 root  txt   VREG	  86,1804206330   139832   5077 /libexec/ld-elf.so.1
nfsd	3097 root  txt   VREG	  86,1804206330  1671232   4999 /lib/libc.so.7
nfsd	3097 root	0u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3097 root	1u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3097 root	2u  VCHR			   0,20	  0t0	 20 /dev/null
nfsd	3097 root	3u  VREG	 255,2264924163		0	217 /var/db/nfs-stablerestart
nfsd	3097 root	4u  VREG	 255,2264924163		0	218 /var/db/nfs-stablerestart.bak

I am stumped at the moment.

 

[kdragon75]

What does vCenter tell you? This is a big part of whats its for.

 

[kspare]

What does vCenter tell you? This is a big part of whats its for.

I'm going to start looking at that, it doesn't look like I can get this from the storage side.

 

[kdragon75]

* face palm *

 

[kspare]

* face palm *

keep in mind we run about 150 vms. so looking at the storage would be a very logic way to track down the offender. trying to find it by looking at vm is very cumbersome. The fact that I can't just see what file has all the io is kind of odd....even windows can display this data. So that was my expectation to figure this out quickly....I guess thats not the case!

 

[kdragon75]

Connect to each host via SSH (you may need to enable this), run esxtop and press v. vRealize Operations may have an easier way of doing this.

 

[kdragon75]

You should also look into Veeam ONE Free, it wont do long term logging but its great for doing basic performance and configuration assessments.

 

[kspare]

Thanks! We have 16 hosts, so it's easier then looking at each vm, ultimately we just need to invest in some performance monitoring tools now that I know I can't do this via freenas.