FreeNAS slowing down with increased disk usage

[MelaGo]

Hi there,

I've been running FreeNAS without a problem for almost 2 years now, 6x3TB in RaidZ2. I run FreeNAS 9.3 with a couple of jails, including Plex and Transmission. A few weeks ago, I suddenly noticed that file transfers from and to the NAS started going slower, and Plex started acting 'sluggish' with the dashboard loading extremely slow and a long wait time before opening video files.

No noticeable changes in hardware or software took place, so I have no clue what's wrong. However, under the 'Reporting' tab I noticed my disk usage increased dramatically around the same time the problems started occuring:

I run SMART tests every night and Scrub once a month.

Is there anything I'm missing? How can I find out what is causing this? And how to fix it?

Thx


NB: Hardware used
Case: Fractal Node 304
Mobo/CPU: ASRock C2750D4I
RAM: Crucial CT2KIT102472BD1608B 16GB (2x8 GB)
PSU: Seasonic G-450
Storage: 6x WD Red 3TB. Plan on running Raidz2, as recommended here.
Boot Device: Integral 8GB Fusion USB 3.0

 

[joeschmuck]

You are missing one key piece of data... How full is your pool? At 10% or less free space the pool goes into a different mode of operation which slows things down. If you were at or below 10% recently then this could be the cause.

 

[MelaGo]

You are missing one key piece of data... How full is your pool? At 10% or less free space the pool goes into a different mode of operation which slows things down. If you were at or below 10% recently then this could be the cause.

pool is 76% full, been hovering around that for the last 4 months. It did peak above 80% once, but I quickly deleted some stuff (as I got an alert).

 

[joeschmuck]

I don't think the optimized writing happens at 80% but the alert was not a good sign. Did the problems start after the alert? Also have you rebooted since you have been below 80%?

Also, I don't want to assume so I'll ask, have you been performing SMART tests on your drives? A failing drive can slow things down as well but typically you will get error messages of dying hard drives.

 

[MelaGo]

I don't think the optimized writing happens at 80% but the alert was not a good sign. Did the problems start after the alert? Also have you rebooted since you have been below 80%?

Also, I don't want to assume so I'll ask, have you been performing SMART tests on your drives? A failing drive can slow things down as well but typically you will get error messages of dying hard drives.

Difficult to tell if it started around that time, it might have. I have rebooted before, but I'll give that another try. Thx for the help.

I run SMART tests every night and a scrub once a month (followed the guide here), no errors whatsoever.

 

[joeschmuck]

Good to know you run SMART tests, wish everyone did.

If rebooting does not speed things up then maybe it's time for a few tests...
1) Change the connectivity to a directly connected computer to the NAS and test the transfer speed for a large file (500MB or larger). Is it okay?
2) If you have some benchmark tool then use it as well. Is it okay?
3) If you still fail then it's time to do an internal hard drive transer test using the "dd" command (here is a link to help with that type of testing) What are the results?

Post all the results so we all can see them and if I'm not here to offer advice, someone else will jump in.

 

[MelaGo]

Reboot completed. It looked promising for a couple of minutes, but then the disk usage spiked up again...


Funny thing, during those few minutes of low usage, I noticed Plex loading faster. Transferring files (CIFS) went back to the old speeds as well (80-90 MB/s), but right now they're at 30-40MB/s again (as they were the last couple of weeks).

Something is using up my disks, and I have no idea what it is...

 

[joeschmuck]

Post the output of: (the first one will show the fragmentation)
1) zpool list
2) df
3) zfs list

It looked promising for a couple of minutes, but then the disk usage spiked up again...

This means that data is being read/write from/to the drives. Without the entire picture I couldn't tell the scale but that in itself is not a failure at all. The higher the better actually.

Also, do you use snapshots? If so, those are pretty well hidden and you should clean those up, they could very well be impacting your system

Since you have some transfer rate and maybe you don't know what that is, start to turn off your plugins and services until it stops. Something is requesting data.

 

[MelaGo]

Post the output of: (the first one will show the fragmentation)
1) zpool list
2) df
3) zfs list


This means that data is being read/write from/to the drives. Without the entire picture I couldn't tell the scale but that in itself is not a failure at all. The higher the better actually.

Also, do you use snapshots? If so, those are pretty well hidden and you should clean those up, they could very well be impacting your system

Since you have some transfer rate and maybe you don't know what that is, start to turn off your plugins and services until it stops. Something is requesting data.

As requested:

1)

Code:

[root@freenas] ~# zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  7.44G  1.84G  5.59G		 -	  -	24%  1.00x  ONLINE  -
vol1		  16.2T  12.1T  4.19T		 -	17%	74%  1.00x  ONLINE  /mnt

2)

Code:

[root@freenas] ~# df
Filesystem												  1K-blocks	   Used	  Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011			 6416666	 793753	5622913	12%	/
devfs															   1		  1		  0   100%	/dev
tmpfs														   32768	   6324	  26444	19%	/etc
tmpfs															4096		  8	   4088	 0%	/mnt
tmpfs														 5574600	 142688	5431912	 3%	/var
freenas-boot/grub											 5629867	   6953	5622913	 0%	/boot/grub
vol1													   5959061166 3329477626 2629583539	56%	/mnt/vol1
vol1/NAS_TM												2629583819		280 2629583539	 0%	/mnt/vol1/NAS_TM
vol1/jails												 2629584003		463 2629583539	 0%	/mnt/vol1/jails
vol1/jails/.warden-template-VirtualBox-4.3.12			  2630437656	 854117 2629583539	 0%	/mnt/vol1/jails/.warden-template-VirtualBox-4.3.12
vol1/jails/.warden-template-pluginjail					 2630173371	 589831 2629583539	 0%	/mnt/vol1/jails/.warden-template-pluginjail
vol1/jails/.warden-template-pluginjail--x64				2630173323	 589783 2629583539	 0%	/mnt/vol1/jails/.warden-template-pluginjail--x64
vol1/jails/.warden-template-pluginjail--x64-20150915183012 2630173355	 589815 2629583539	 0%	/mnt/vol1/jails/.warden-template-pluginjail--x64-20150915183012
vol1/jails/.warden-template-pluginjail--x64-20150915212945 2630173339	 589799 2629583539	 0%	/mnt/vol1/jails/.warden-template-pluginjail--x64-20150915212945
vol1/jails/.warden-template-standard					   2632328816	2745276 2629583539	 0%	/mnt/vol1/jails/.warden-template-standard
vol1/jails/couchpotato_1								   2630473861	 890321 2629583539	 0%	/mnt/vol1/jails/couchpotato_1
vol1/jails/firefly_1									   2630414367	 830828 2629583539	 0%	/mnt/vol1/jails/firefly_1
vol1/jails/headphones_1									2630752077	1168538 2629583539	 0%	/mnt/vol1/jails/headphones_1
vol1/jails/plexmediaserver_1							   2645446984   15863445 2629583539	 1%	/mnt/vol1/jails/plexmediaserver_1
vol1/jails/sickrage_1									  2630810020	1226481 2629583539	 0%	/mnt/vol1/jails/sickrage_1
vol1/jails/subsonic_1									  2632545979	2962440 2629583539	 0%	/mnt/vol1/jails/subsonic_1
vol1/jails/transmission_1								  2632099208	2515669 2629583539	 0%	/mnt/vol1/jails/transmission_1
vol1/jails/transmission_2								  2631961615	2378075 2629583539	 0%	/mnt/vol1/jails/transmission_2
vol1/nas												   7895031921 5265448382 2629583539	67%	/mnt/vol1/nas
vol1/.system											   2629714827	 131287 2629583539	 0%	/var/db/system
vol1/.system/cores										 2629585529	   1990 2629583539	 0%	/var/db/system/cores
vol1/.system/samba4										2629584474		935 2629583539	 0%	/var/db/system/samba4
vol1/.system/syslog-5dadcabebc7540c8a3adcfc974f2cd77	   2629587104	   3564 2629583539	 0%	/var/db/system/syslog-5dadcabebc7540c8a3adcfc974f2cd77
vol1/.system/rrd-5dadcabebc7540c8a3adcfc974f2cd77		  2629583731		192 2629583539	 0%	/var/db/system/rrd-5dadcabebc7540c8a3adcfc974f2cd77
vol1/.system/configs-5dadcabebc7540c8a3adcfc974f2cd77	  2629632811	  49272 2629583539	 0%	/var/db/system/configs-5dadcabebc7540c8a3adcfc974f2cd77
devfs															   1		  1		  0   100%	/mnt/vol1/jails/couchpotato_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/couchpotato_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/couchpotato_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/firefly_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/firefly_1/proc
/mnt/vol1/nas/Music										7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/firefly_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/headphones_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/headphones_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/headphones_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/plexmediaserver_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/plexmediaserver_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/plexmediaserver_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/sickrage_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/sickrage_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/sickrage_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/subsonic_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/subsonic_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/subsonic_1/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/transmission_2/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/transmission_2/proc
/mnt/vol1/server										   5959061166 3329477626 2629583539	56%	/mnt/vol1/jails/transmission_2/media
devfs															   1		  1		  0   100%	/mnt/vol1/jails/transmission_1/dev
procfs															  4		  4		  0   100%	/mnt/vol1/jails/transmission_1/proc
/mnt/vol1/nas											  7895031921 5265448382 2629583539	67%	/mnt/vol1/jails/transmission_1/media

3)

Code:

[root@freenas] ~# zfs list
NAME														 USED  AVAIL  REFER  MOUNTPOINT
freenas-boot												1.84G  5.36G	31K  none
freenas-boot/ROOT										   1.81G  5.36G	25K  none
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201506292332			 36K  5.36G   557M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011		   1.81G  5.36G   775M  /
freenas-boot/ROOT/Initial-Install							  1K  5.36G   507M  legacy
freenas-boot/ROOT/Wizard-2015-09-13_15:43:53				   1K  5.36G   555M  /
freenas-boot/ROOT/default								   1.86M  5.36G   509M  legacy
freenas-boot/grub										   20.4M  5.36G  6.79M  legacy
vol1														8.04T  2.45T  3.10T  /mnt/vol1
vol1/.system												 185M  2.45T   128M  legacy
vol1/.system/configs-26bec1b654b54f3489f57aa43ee342f9		192K  2.45T   192K  legacy
vol1/.system/configs-5dadcabebc7540c8a3adcfc974f2cd77	   48.1M  2.45T  48.1M  legacy
vol1/.system/cores										  1.94M  2.45T  1.94M  legacy
vol1/.system/rrd-26bec1b654b54f3489f57aa43ee342f9			192K  2.45T   192K  legacy
vol1/.system/rrd-5dadcabebc7540c8a3adcfc974f2cd77			192K  2.45T   192K  legacy
vol1/.system/samba4										  935K  2.45T   935K  legacy
vol1/.system/syslog-26bec1b654b54f3489f57aa43ee342f9		2.00M  2.45T  2.00M  legacy
vol1/.system/syslog-5dadcabebc7540c8a3adcfc974f2cd77		3.48M  2.45T  3.48M  legacy
vol1/NAS_TM												  280K  2.45T   280K  /mnt/vol1/NAS_TM
vol1/jails												  31.2G  2.45T   464K  /mnt/vol1/jails
vol1/jails/.warden-template-VirtualBox-4.3.12				834M  2.45T   834M  /mnt/vol1/jails/.warden-template-VirtualBox-4.3.12
vol1/jails/.warden-template-pluginjail					   576M  2.45T   576M  /mnt/vol1/jails/.warden-template-pluginjail
vol1/jails/.warden-template-pluginjail--x64				  576M  2.45T   576M  /mnt/vol1/jails/.warden-template-pluginjail--x64
vol1/jails/.warden-template-pluginjail--x64-20150915183012   576M  2.45T   576M  /mnt/vol1/jails/.warden-template-pluginjail--x64-20150915183012
vol1/jails/.warden-template-pluginjail--x64-20150915212945   576M  2.45T   576M  /mnt/vol1/jails/.warden-template-pluginjail--x64-20150915212945
vol1/jails/.warden-template-pluginjail--x64-20151015105212   576M  2.45T   576M  none
vol1/jails/.warden-template-standard						2.62G  2.45T  2.62G  /mnt/vol1/jails/.warden-template-standard
vol1/jails/.warden-template-standard--x64				   2.62G  2.45T  2.62G  none
vol1/jails/couchpotato_1									 334M  2.45T   869M  /mnt/vol1/jails/couchpotato_1
vol1/jails/firefly_1										 243M  2.45T   811M  /mnt/vol1/jails/firefly_1
vol1/jails/headphones_1									  573M  2.45T  1.11G  /mnt/vol1/jails/headphones_1
vol1/jails/plexmediaserver_1								14.7G  2.45T  15.1G  /mnt/vol1/jails/plexmediaserver_1
vol1/jails/sickrage_1										662M  2.45T  1.17G  /mnt/vol1/jails/sickrage_1
vol1/jails/subsonic_1									   2.30G  2.45T  2.83G  /mnt/vol1/jails/subsonic_1
vol1/jails/transmission_1								   1.88G  2.45T  2.40G  /mnt/vol1/jails/transmission_1
vol1/jails/transmission_2								   1.75G  2.45T  2.27G  /mnt/vol1/jails/transmission_2
vol1/nas													4.91T  2.45T  4.91T  /mnt/vol1/nas

And for the snapshots:

Code:

[root@freenas] ~# zfs list -t snapshot
NAME																	USED  AVAIL  REFER  MOUNTPOINT
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011@2015-05-23-02:44:51  2.92M	  -   507M  -
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011@2015-07-11-13:55:12  5.27M	  -   509M  -
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011@2015-09-13-15:43:53  12.1M	  -   555M  -
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011@2016-02-05-10:02:39  14.2M	  -   557M  -
freenas-boot/grub@Pre-Upgrade-FreeNAS-9.3-STABLE-201506292332		  6.77M	  -  6.78M  -
freenas-boot/grub@Pre-Upgrade-Wizard-2015-09-13_15:43:53				 20K	  -  6.79M  -
freenas-boot/grub@Pre-Upgrade-FreeNAS-9.3-STABLE-201602031011			20K	  -  6.79M  -
vol1/jails/.warden-template-VirtualBox-4.3.12@clean					 160K	  -   834M  -
vol1/jails/.warden-template-pluginjail@clean							320K	  -   576M  -
vol1/jails/.warden-template-pluginjail--x64@clean					   192K	  -   576M  -
vol1/jails/.warden-template-pluginjail--x64-20150915183012@clean		320K	  -   576M  -
vol1/jails/.warden-template-pluginjail--x64-20150915212945@clean		192K	  -   576M  -
vol1/jails/.warden-template-pluginjail--x64-20151015105212@clean		176K	  -   576M  -
vol1/jails/.warden-template-standard@clean							  288K	  -  2.62G  -
vol1/jails/.warden-template-standard--x64@clean						 176K	  -  2.62G  -

Should I clean those up? I don't really use periodic snapshots, how do I delete them?

I tried turning off the Plex jail, without any success. I'll start switching off the others one by one and update the thread. Thanks a ton for all the help!

 

[joeschmuck]

Those snapshots are normal and do not consume much space. Some people have snapshots enabled and consume thier entire drive by accident.

74% capacity should be fine, as well as 17% fragmentation should not be terrible. I see nothing else obvious either.

I think you will need to do some testing to find out what is causing your system to act up. You need to just start disabling some of these plugins to see what is causing the issue. How many files do you have being hosted by Plex?

I don't think you have a CPU issue, meaning that your CPU is fast enough to keep up.

One last thing is it could be your network. Lets say you are running everything over wireless and you added a few cell phones to the package, you would be overwelming that wifi network. Or maybe the router just got an upgrade last month and you didn't really notice an issue until recently. Remember that it could be almost anything and the problem could have be induced well before you noticed it. Keep an open mind.

Good luck troubleshooting.

 

[MelaGo]

It seems like I found the culprit.
As suggested, I was turning off plugins one by one in order hoping to find a decrease in disk usage. I had almost given up, but at the second to last suddenly it happened:

Sickrage

I've had this installed for ages (switched from sickbeard), and regularly updated. No clue why it sent so many read/write commands. Searching for it online didn't bring up anything, except for this:

https://github.com/SickRage/SickRage/issues/3577


Anyway, I'm gonna try a fresh jail install and see what happens.
Thanks @joeschmuck for all the help, if anybody else has similar problems with SickRage, feel free to reply.

 

[joeschmuck]

Glad you were able to identify the suspect jail and I hope a clean install fixes everything.