MiniDLNA Automatic Scan - Fix

[joeschmuck]

This fix has been tested on FreeNAS 8.3.0-RELEASE through 9.1.1-RELEASE however due to a change in the way the jails are operating in 9.1.1, it does have issues. See this posting for an update to these concerns about 9.1.1. http://forums.freenas.org/threads/minidlna-automatic-scan-fix.9312/page-5#post-76696
DO NOT USE THIS WITH FREENAS 9.2.x or later, I have not tested it.

What this does is automatically forces a rescan of your databases when you make a change to the database. Currently the MiniDLNA plugin does not support this so here is the fix and it has been working fine for me on a real physical machine.

Some background about what I was trying to do...

Some people resorted to restarting the MiniDLNA service via a CRON job which unfortunately happens at a predetermined time and always spins up the hard drives to scan the files. This is no way to do it as in my system I have my hard drives sleep after 2 hours of no use and I don't want them spun up just to scan, especially if I haven't made any changes. So I have been looking for a different way to trigger the rescan event and I discovered the wait_on function. This is not a perfect function but it does work.

So I've been working on this and go figure, it works pretty good. The only downside I have is if you make a change while you are streaming a video, it generally will stop the streaming. I really don't consider this a downside for my use but it will be something you have to factor in. You can try it to see if you like it, if not then it's easy to disable.

A few issues I found through some valuable feedback is there is a limitation on the length of the list of directories where your files reside. This is not a limitation on how many directories nor how many files you may have. Think of the limitation this way, if you start listing all the recursive directories from the default /media all in a single string (example) "/media /media/Movies_A_Z /media/Movies_A_M/Movies_A /media/Movies_A_Z/Movies_B", well you can see how this string can become very long especially with lots of music. The limitation is 261649 characters for this string. If you exceed this the wait_on function fails and the rescanning occurs constantly. The fix is to change your directory structure layout to reduce this string size. The work around was to put into place an alternate method to check for a change. This method is active and does poll the drives whereas wait_on does not. So this scanning will change from Normal to Alternate automatically and is checked during a detected changing event (added or deleted a file). If you prefer to sleep your drives then you will not like this alternate method if it kicks in because you have passed the threshold of wait_on capability.

I am looking into using Kqueue to replace the wait_on function but my programming skills are lacking so it will not happen overnight, however anyone who wants to collaborate with me, you can take the credit as I'm only looking for a proper solution for everyone.

I have incorporated variables for the ease of tailoring the script and I have picked default values that I use. Please change them to your liking.

NOTE: Steps 1 and 2 must be done and MiniDLNA MUST be working before you continue to step 3. If you have any problems it's easier to fix if you know MiniDLNA has already been running fine. If you have already installed MiniDLNA then you can skip to step 3. You also must be logged in as root or have root privileges.

Here are the steps to install it. (Note: single quotes around text in the steps are omitted when you actually type in the command)

1) Running FreeNAS 8.3.0-Release or later.
2) Install the MiniDLNA PBI:

a) For FreeNAS versions prior to 9.1.x, install the Jail PBI and MiniDLNA PBI and configured them, ensure Rescan on restart is checked. Make sure you add your mount points to /media in the jail and it works normally as expected.
b) For FreeNAS 9.1.1 you must install the MiniDLNA PBI from the GUI or as an upload. Configure MiniDLNA plugin and ensure Rescan on restart is checked. Make sure you add your mount points to /media in the jail and is works normally as expected. In FreeNAS 9.1.1 after you complete the modifications below the plugin will look like it's not operating in the GUI however it actually is running. Hopefully this will be fixed in a future release of FreeNAS.​

NOTE: MiniDLNA for 8.3.0-Release has a minor problem. It's not critical to install the fix and it has been corrected in FreeNAS version 8.3.1 and later. See posting #45 for the fix.​

3) Use SSH to get into the jail using the following command: jexec dlna_1 /bin/tcsh
4) Type 'pkg_add -r wait_on' and once it completes type 'rehash'.
5) Type 'cd /etc/rc.d'
6) Type 'ee scanmedia'
7) Cut and Paste the following code into the editor.

Code:

#!/bin/sh
# File name 'scanmedia'
# Place this file into /etc/rc.d
# Edit /etc/rc.conf to include scanmedia_enable="YES"

# Things to know...  The purpose of this script is to trigger a rescan of the
# media for the minidlna plugin.  This uses the "wait_on" command which
# although works, it has it's limitations:
#  1) It it rescans on the detection of a change, not the completion of a change.
#  2) It has a limitation of the directory string length so I have implemented
#    an alternate method for scanning however it does actively read the drives
#    so if you wanted your drives to sleep, this method will prevent sleeping
#    from occurring and I strongly recommend you work on your directory structure.


. /etc/rc.subr

name="scanmedia"
rcvar=scanmedia_enable

PATH="$PATH:/usr/local/bin"

start_cmd="${name}_start"
stop_cmd=":"

load_rc_config $name
eval "${rcvar}=\${${rcvar}:-'NO'}"

# Variables used in this script...

# RedAlert = Maximum string length wait_on can handle
RedAlert=261649

# YellowAlert = User selected warning point.  Looking to use this to enable the
#  flashing warning light in the GUI.
YellowAlert=255000

# TriggeredDelay = The amount of time in seconds to delay rescanning from the point of a
# change being detected.  The default is 300 seconds (5 minutes).  Change to your liking
# and think of this as how long it takes you to write a large file (movie) to your NAS.
TriggeredDelay=300

# AlternateScanRate = The amount of time in seconds between polling the media for changes.
# Default value is 1800 seconds (30 minutes).  Change to your liking.
AlternateScanRate=1800

# ScanMode = Normal or Alternate method.  Default is '0' for Normal Mode.  This changes
# in the script based on the actual directory string length vs the RedAlert value.
ScanMode=0

# MediaLocation = The location your want to scan for your media.  The default is "/media".
MediaLocation="/media"

scanmedia_normal()
{
while [ "$ScanMode" -eq "0" ] ; do

# First lets locate all the subdirectories.  Since the minidlna plugin only
# allows one path for media we will use the path defined above (default=/media)

directories=$(find $MediaLocation -type d -print0 | xargs -0 -I % echo -n '"%" ')

# Check on maximum length of directory string.

if [ ${#directories} -gt $YellowAlert ];
  then
  if [ ${#directories} -gt $RedAlert ];
  then
  date > /minidlnaerror.log
  echo "RED ALERT! You have exceeded the maximum directory string length of 261649 and you are currently at" >> /minidlnaerror.log
  echo ${#directories}".  The alternate method is in use and you need to shorten the directory names or rearange your folders" >> /minidlnaerror.log
  echo "to restore the normal scanning method." >> /minidlnaerror.log
# Change from normal scan mode to alternate scan mode.
  ScanMode=1
  else
  date > /minidlnaerror.log
  echo "YELLOW ALERT! The maximum directory string length is 261649 and you are currently at" >> /minidlnaerror.log
  echo ${#directories}".  You should shorten the directory names or rearange your folders" >> /minidlnaerror.log
  echo "to remove this warning." >> /minidlnaerror.log
  fi
fi

if [ "$ScanMode" -eq "0" ]
then

# Wait_on will trigger on any event which writes to the directories listed.

csh -c "wait_on -w $directories"

# Now let's call the rescanning routine
scanmedia_rescan

fi
done
}

scanmedia_rescan()
{
# This routine pauses for the user specified delay (default 5 mins) before killing
# the minidlna service and restarting it.

  sleep $TriggeredDelay

# We must stop the minidlna service in the GUI first, then wait 10 seconds (probably overkill)
# and then kill the service.

  service minidlna onestop
  sleep 10
  pkill minidlna

# We wait 10 seconds to allow the service lots of time to wrap up.

  sleep 10

# And now to start up the service again.

  service minidlna onestart
}

scanmedia_alternate()
{
while [ "$ScanMode" -eq "1" ] ; do

#Lets check to see if we can go back to normal_scan mode.
directories=$(find $MediaLocation -type d -print0 | xargs -0 -I % echo -n '"%" ')
if [ ${#directories} -lt $RedAlert ]
then
ScanMode=0
fi

# We list all the files and save as a file in memory, hopefully the file doesn't get too
# big to cause an issue.  If it does then remove the '/var/' from the below lines.

ls -RlT $MediaLocation > /var/altscan1.dat
sleep $AlternateScanRate
ls -RlT $MediaLocation > /var/altscan2.dat

# diff return 0 if files are identical, 1 if they are different, 2 if one of the files cannot open.
diff -q /var/altscan1.dat /var/altscan2.dat

# We don't really care about cases 0 and 2 but it's nice to have for troubleshooting. They
# are commented out right now.
case "$?" in
#"0") echo "Files Identical";;
"1") scanmedia_rescan;;
#"2") echo "Failed to open one of the files";;
esac

done
}

scanmedia_start()
{
while :; do
if [ "$ScanMode" -eq "0" ]
then
scanmedia_normal
else
scanmedia_alternate
fi
done
}

scanmedia_start

run_rc_command "$1"

8) If you are using the default path /media then you need do nothing to the script, otherwise you will need to adjust it some.
9) The default delay for rescanning your database from when a change starts is 5 minutes (300 seconds) and is called TriggeredDelay in the script. If you want this delay to be longer (I use 600 seconds for 10 minutes) then I recommend you change it now. There are two reasons you would change this value:

a) If a single rescan of your database takes longer than 5 minutes then TriggeredDelay should be larger than a single rescan event.
b) If you typically transfer files into the area which take longer than the TriggeredDelay setting. This will prevent slowing down the transfer due to unnecessary rescanning which is the reason my personal setting is 600 seconds.​

10) Now press ESCape and save the file.
11) Type 'chmod +x scanmedia' to make it executable.
12) Type 'cd ..'
13) Type 'ee rc.conf'
14) Add 'scanmedia_enable="YES" '
15) Press ESCape and save the file.
16) Type 'exit'
17) Type 'reboot'

After rebooting the script should be running.

And once again, in FreeNAS 9.1.1 after you complete the modifications the plugin will look like it's not operating in the GUI however it actually is running. Hopefully this will be fixed in a future release of FreeNAS.

To Disable this feature (does not remove it until you get to step 8):
1) SSH (you could use the shell) to get into the jail using the following command: jexec `jls jid` /bin/tcsh
2) Type 'cd /etc'
3) Type 'ee rc.conf'
4) Locate the line we added in step 13 above and place a hash '#' mark in front of it to disable running of the scanmedia script on next reboot. Press ESCape and save it.
5) Type 'exit'
6) Type 'reboot'
7) You could stop here because you might want to try it again later, however if you want to completely remove it... Do the remaining steps
8) After the reboot get back into the jail as in step 1
9) Type 'rm /etc/rc.d/scanmedia'
10) Type 'pkg_info' and locate wait_on. You need all of what it's name is (this example is wait_on-1.1)
11) Type 'pkg_delete wait_on-1.1'

That makes a complete removal of the changes in this posting.

If you notice any quirks please let me know.

My goal would be to have this functionality built in to the minidlna plugin with the following selections: Enable, Delay (in seconds), File Path. FreeNAS Ticket http://support.freenas.org/ticket/1789

-Mark

10/20/2012 Updated code to automatically scan sub-directories. Does not automatically scan if a sub-directory has a space character in it.
10/23/2012 Updated code to automatically scan sub-directories including those with spaces in the names. Also the stopping and starting of the minidlna service has been changed to work properly now.
10/30/2012 WDTV apparently writes temporary files to the device which causes continual restarting of minidlna. Not sure if this is from DLNA or CIFS sharing.
10/30/2012 Subsonic may be causing continual restarting of minidlna, waiting on confirmation. Not sure if this is from DLNA or CIFS sharing.
11/09/2012 Due to a limitation with 'wait_on' (described above) I have implemented and automatic alternate scanning method but there is a downside to it, the NAS drives will be actively accessed to check for changes which will impact a system if you wanted to sleep the drives. I hope to find a better way around this but this does work and if you leave your drives on all the time, it will not be an issue for you.
11/19/2012 Fix for MiniDLNA Plugin Start/Stop (8.3.0-Release), see post #45 in this thread.
11/28/2012 Added that you must be logged in as root or have root privileges.
3/30/2013 The MiniDLNA plugin fix referenced in step 2A above and the 11/19/2012 note has been verified fixed in the 8.3.1-Release plugin.
7/27/2013 Verified script works as written on FreeNAS 9.1.0RC2. Cleaned up some of the text.
9/1/2013 Verified script works with FreeNAS 9.1.1-RELEASE (with a minor GUI issue) and updated the text a little to address this.
7/25/2014 Added warning message that this has not been tested with FreeNAS 9.2.x.

 

[durdur]

Excellent : Thank you very much Mark, it works fine for me, I just had to change 'minidlna_enable' from NO to YES in rc.conf otherwise the 'service minidlna start' command answered something like "minidlna won't start as minidlna_enable is set to no".

I was using the Cron method every night and was thinking about a script easily usable by users to restart on demand via ssh, but your method is much better.

I don't understand why there is no other comments as it's a frequently asked question

 

[joeschmuck]

I'm glad it works for you and thanks for letting me know. I agree, not sure why there has been no interest except by you. Maybe most folks are content stopping and restarting the plugin. I know having it automated for me is the best way to go. I would appreciate any opinion you have on how this works and if you think it should be tweaked. You know a few months ago I was so close to having this work but I had to discover the PKILL command in order to make this work properly. I'd love it if I could simply just force a rescan but I haven't had any success with that.

 

[hervon]

Hi joeschmuck. I'm really interested in your tweak. I installed it and i'm in the troubleshooting process. My movies are mainly in subfolfers so I added some of them to the wait_on command. Do I have to do something specific if my folder name has spaces in it ?

 

[joeschmuck]

Hi joeschmuck. I'm really interested in your tweak. I installed it and i'm in the troubleshooting process. My movies are mainly in subfolfers so I added some of them to the wait_on command. Do I have to do something specific if my folder name has spaces in it ?

I believe you need to place quotes around anything that has a space in the directory name. Ex: wait_on -w "/media/Movies A" "/media/Movies B"

Please post if this works or not.

I'm working on something new which would grab all the subdirectories automatically but I only just started on it a few minutes ago and I'm not a lunix programmer at all so it could be a few days before I even find out if it works.

 

[hervon]

It doesn't work for now. The service scanmedia is activated but when I change a subfolder targeted with the wait_on command, nothing happens. Here is how I change this line :

wait_on -w "/media" "/media/Movies" "/media/Movies/1 New Movies!" "/media/Music"

Any clue ?

Thanks for your effort to correct the 'rescan on change' issue with minidlna!

 

[joeschmuck]

It doesn't work for now. The service scanmedia is activated but when I change a subfolder targeted with the wait_on command, nothing happens. Here is how I change this line :

wait_on -w "/media" "/media/Movies" "/media/Movies/1 New Movies!" "/media/Music"

Any clue ?

Thanks for your effort to correct the 'rescan on change' issue with minidlna!

If I don't understand this, please explain what you are doing in detail.

Here is what I understand...

I either "/media" or "/media/Movies" or "/media/Movies/1 New Movies!" or "/media/Music" you have changed a file. Let's say you added the file BlockBuster.mkv to the directory "/media/Movies". This should have triggered a scan.

 

[joeschmuck]

Updated Code in Post#1.

This automatically scans for sub-directories but there is a severe limitation right now... The sub-directories cannot have any spaces in the names. So you cannot have "/media/Movies A-M" but you could have "/media/Movies_A-M".

Please post if this works, don't assume that if I don't hear from anyone that it must be working.

If someone knows what I need to do with xargs to get the quotes around the sub-directory names and wait_on accepts it, please let me know. I've been racking my brain and because I'm not a programmer I'm just over my head here.

 

[hervon]

I moved my lazy ass and looked around the web. I changed the double quotes with singles quotes, according to this link and it seems to works! Thanks joeschmuck for your support with this tool. ;-) Now it looks like this:

wait_on -w 'media' '/media/Movies' '/media/Movies/1 New Movies!' '/media/Music'

 

[garfunkel]

I guess you could use this:

find . -type d | sed "s/^/'/g" | sed "s/$/'/g"

in the find command to search for sub dirs and allow for spaces in the anme.

 

[joeschmuck]

I must run the following command in a C shell but I'm not having much luck inside the script.

Must run as "csh test.sh" to actually work properly.

Code:

#!/bin/sh
# File Name = test.sh

echo "Waiting for a change"
wait_on -w "`find /media -type d -print `"
echo "Change happened"

So the question is, how do I run the larger script, which as far as I know must be 'sh' and still run 'csh' within this script? I have tried a few things which failed. If someone comes up with the fix, please post it. My brain hurts!

 

[paleoN]

Must run as "csh test.sh" to actually work properly.

I'm confused. What doesn't work properly? Also, you're saying the script is a Bourne shell script and not a C shell script in the first line, #!/bin/sh.

I would change service minidlna start to:

Code:

service minidlna onestart

This should keep minidlna synched with the GUI status.

It may be worthwhile to try a:

Code:

service minidlna onestop

Before your pkill command. I don't think it will stop minidlna if it's playing a video though. After the pkill you should clean up the old PID file. I believe that will be sufficient to update the GUI status as well. Perhaps someone who has actually done a plugin could comment.

 

[joeschmuck]

Fixed Scanning Sub-Directories with Spaces

The updated code is again in posting #1 of this thread...

http://forums.freenas.org/showthrea...tomatic-Scan-Fix&p=40023&viewfull=1#post40023

I have fixed the problem with spaces in the directory names and added the two items paleoN suggested. The GUI now changes for MiniDLNA from/to ON/OFF.

@paleoN,
Not sure how to clean up PIDs. Where would I look for these pesky guys and any suggestions on what to do would be appreciated. Also I tried to stop the minidlna service without pkill but it just doesn't cut it.

Again, my goal is to have this code somehow integrated into the MiniDLNA plugin. I did most of the work, it would be nice to see it used.

-Mark

 

[hervon]

I installed the updated code. Ill let you know how it works on my setup with 8.2 in a few days. I completely agree with your suggestion to integrate this tool directly into MiniDLNA in the furture. Thanks again for your hard work.

 

[joeschmuck]

No problem. It was suggested that I might try my hand at integrating the fix into the MiniDLNA plugin. That is probably outside my current skill level. Also, if I were to tackle this type of project, I'd rather just fix MiniDLNA properly vice this patch. I don't like the idea of the stream dropping just because you added a file. There must be a better way.

 

[paleoN]

@paleoN,
Not sure how to clean up PIDs. Where would I look for these pesky guys and any suggestions on what to do would be appreciated.

I'm not sure myself. Maybe /var/run/php-fpm-minidlna.pid inside the jail if I'm looking at the right file.

Also I tried to stop the minidlna service without pkill but it just doesn't cut it.

Looks like they use pkill in nanobsd/plugins/minidlna_pbi/resources/control file. I'm not entirely sure they actually use that file though.


Line 3 can be simplified a bit to:

Code:

3) I used SSH (you could use the shell) to get into the jail using the following command: 'jexec `jls jid` /bin/tcsh'

I assume this will also work with a normal csh shell.

I don't like the idea of the stream dropping just because you added a file. There must be a better way.

Have you tried posting over on the minidlna SourceForge forums? Perhaps someone there might have some idea.

 

[hervon]

Well, I can't get the updated code to work properly. I get this error when I start the jail : freenas notifier: /etc/rc.d/scanmedia: !/bin/sh: not found

 

[joeschmuck]

Well, I can't get the updated code to work properly. I get this error when I start the jail : freenas notifier: /etc/rc.d/scanmedia: !/bin/sh: not found

Somehow I dropped the hash in front of the first line when I cut and pasted the code over since you last used it. It's fixed now. Thanks for letting me know.

 

[hervon]

Your updated code works perfectly! Actually, it works too good for me. ;-) Let me explain. I stream my movies from my FreeNAS server to a WDTVLive. The problem is that the WDTVLive creates an hidden folder with information about the current movie to allows resume at the current time in the movie. So I played a movie tonight. The folder change was detected and MiniDLNA restarted after 300 seconds! If I could exclude a folder, it would fix my problem. For now, I reverted to the previous version withe selected folder to watch for changes.

 

[Zendaug]

Your updated code works perfectly! Actually, it works too good for me. ;-) Let me explain. I stream my movies from my FreeNAS server to a WDTVLive. The problem is that the WDTVLive creates an hidden folder with information about the current movie to allows resume at the current time in the movie. So I played a movie tonight. The folder change was detected and MiniDLNA restarted after 300 seconds! If I could exclude a folder, it would fix my problem. For now, I reverted to the previous version withe selected folder to watch for changes.

On that note, would you recommend setting the Transmission download folder to a different folder to "/media"? I concluded that was my problem after minidlna only seemed to be showing a fraction of my movie files (presumably because it was restarting every minute or so).