My first cronjob; scripting difficulties.

[Sokonomi]

Are you sure your cameras won't work with ZoneMinder?

You can create a FreeBSD 13.2-RELEASE jail (Basejail), switch to the "latest" repository, and then install and configure zoneminder in the jail.

Then just add the proper "mountpoint" for the jail (/mnt/Tank2/Surveillance), and go from there.

ZoneMinder can automatically prune old video files, among other things.

Seems more pragmatic than playing this dance of scripting.

Zoneminder is a whole other beast. It requires bucketloads of compute power to 24/7 record five 4mp cameras to x265. Something the cameras themselves can do natively without stressing my server. It's just that juggling the files seems a bit tricky. I wonder if Dahua made their own kind of DVR recording management software..

The cameras are also capable of depositing files using a NFS, but that turned out unreliable for some reason. They occasionally drop connection to that somehow, recordings went missing. Maybe the cams own pruning system would work with that, but if recordings don't reliably make it to the NAS thats a no-go.

Its likely I will end up having to rely on some kind of cronjob to keep this thing alive..

The first and the last line:
```bash
and
```
don't belong in the script, delete them.

That's markdown formatting instructions - how did they end up in your script? When reading your initial post I thought you put them in there to format for the forum ...

Every script should start with a line starting with #! followed by the interpreter for the script language.

Likely a remnant from when I had openAI write a basic script for me so I could have a frame to work with. Me being completely new to bash scripting, I just left it in thinking it just belonged there. :')

 

[winnielinnie]

UPDATE: See my post below this.

Its likely I will end up having to rely on some kind of cronjob to keep this thing alive..

This is as far as I'm willing to go, since you're dealing with a script that deletes entire directories.

You can use my "latest updated" script as a template. It uses two rudimentary "safeguards" to make sure the pool is valid and the directory exists, before cd'ing into the correct parent directory to execute the remainder of the script.

It needs to be run as root (or any user that has permission to execute the script and delete the relevant directories.) When you add it as a Cron Job in the GUI, you can invoke it with "sh", such as entering this for the "command" input field: sh /path/to/pruning-script.sh

Keep in mind this doesn't take into account that each camera has its own parent folder.


Updated version with rudimentary "safeguards" and explanations, as well as exit codes:

Code:

#!/usr/bin/env sh

# Examples of variables that _must_ be manually filled out.
# Some are in later sections after the safeugards.
# The following 3 variables all must be manually configured.
# MIN_FREE_SPACE=1073741824
# STORAGE_POOL=mypool
# MEDIA_DIR=$STORAGE_POOL_ROOT/Media


# DECLARE VARIABLES

MIN_FREE_SPACE=1073741824
# Hint: 1073741824 bytes = 1 gibibyte

STORAGE_POOL=Tank2



# SAFETY CHECKS

# Confirm $STORAGE_POOL is a valid pool, or else just abort
if ! zpool status $STORAGE_POOL > /dev/null; then
    echo "$STORAGE_POOL is an invalid pool name!"
    exit 1
fi

STORAGE_POOL_ROOT=/mnt/$STORAGE_POOL
MEDIA_DIR=$STORAGE_POOL_ROOT/Surveillance

# Confirm $MEDIA_DIR is a valid path, or else just abort
if ! stat $MEDIA_DIR > /dev/null; then
    echo "$MEDIA_DIR is an invalid path!"
    exit 1
fi



# CONTINUE WITH SCRIPT IF SAFETY CHECKS PASS

# Get and declare the pool's free space in bytes
POOL_FREE_SPACE=$(zpool get -H -p free $STORAGE_POOL | cut -f3)

# Change into media directory
cd $MEDIA_DIR

# Get and declare the oldest directory in current working directory
OLDEST_DIR=$(ls -rt1d */ | head -n 1)

# Check if the free space is below the minimum threshold
if [ $POOL_FREE_SPACE -lt $MIN_FREE_SPACE ]; then
    # Find the oldest directory and delete it
    # rm -rf $OLDEST_DIR
    echo "Deleted $OLDEST_DIR as $STORAGE_POOL free space was below $MIN_FREE_SPACE bytes."
    exit 0
else
    echo "Sufficient free space available on $STORAGE_POOL. Otherwise, $OLDEST_DIR would have been deleted."
    exit 0
fi

 

[winnielinnie]

Alternatively, you can create a script that combs through the entire "Surveillance" folder and deletes all video files that are older than 30 days (as an example.)

Perhaps it could run every midnight, as a typical pruning job, and in such a way that only videos which are younger than 30 days will remain saved.

This could be accomplished with the "find" command, which will scrape the entire directory and subdirectories, finding all files older than X days, and issue the "-delete" option to delete them. ("find" allows you to consolidate several requirements into a single command. No need to calculate free space, or such.)


This finds all .mp4 files that are older than 30 days.

Code:

find /mnt/Tank2/Surveillance -type f -name "*.mp4" -mtime +30

This finds all .mp4 files that are older than 30 days, and deletes them.

Code:

find /mnt/Tank2/Surveillance -type f -name "*.mp4" -mtime +30 -delete

EDIT: Before you try or test anything, create a recursive snapshot of your Surveillance dataset right now.

 

[Sokonomi]

Thank you for your assistance in this, and I fully understand you don't want proverbial blood on your hands, so don't worry; I would not blame you for anything if I have a mishap, that's my sole responsibility.

Destroying the videofiles would work, with the small caveat that it would leave a bit of a trail of leftovers, as these cameras arent simply pooping out a videofile; they produce a .dav (video) .idx (index) and .jpg (iFrame snapshot).

There's also the thing of relying on an age gate; Id have to proactively calculate how far back my pool can handle, size wise. Despite the cameras running on static bitrate, the total file size of a day seems to vary by a couple GB. If I stick to the biggest it has gotten so far, id be on 90gb per day per camera. With 3 cameras and a 3Tb pool id have roughly 10 days before it saturates. So if im not mistaken, the command should look like this;

Code:

find /mnt/Tank2/Surveillance -type f \( -name "*.dav" -o -name "*.idx" -o -name "*.jpg" \) -mtime +9 -delete

This should keep it relatively clean, but if something decides to overload the capacity it would still seize up I think.

 

[winnielinnie]

That "could" maybe work.

It's also possible to use the "depth" option in find, and have it delete the oldest directory (which assumes it will be in the naming format of 2023-10-05, etc., under each camera's parent folder.)

This can work as an "emergency" delete, that destroys all the oldest video subfolders when free space is les than 100GB. (Waiting until 1 GB is way to risky. You shouldn't even be allowing your pool to get that full. 100GB assures that you have enough for an "extra day".)


This gives you an idea, by printing out and sorting all the "day-by-day" subfolders in each camera's parent folder:

Code:

for CAMDIR in /mnt/Tank2/Surveillance/*/; do find $CAMDIR -type d -depth 1 | sort | xargs echo ; done

This gives you an idea, by printing out only the oldest "day-by-day" subfolders in each camera's parent folder. (These results are what would be deleted.)

Code:

for CAMDIR in /mnt/Tank2/Surveillance/*/; do find $CAMDIR -type d -depth 1 | sort | head -n1 | xargs echo ; done

This will actually destroy all the folders from the above example:

Code:

for CAMDIR in /mnt/Tank2/Surveillance/*/; do find $CAMDIR -type d -depth 1 | sort | head -n1 | xargs rm -rf ; done

This relies on many assumptions:

• The folder naming schema will be in the predictable YYYY-MM-DD format for every camera
• The folder naming schema will never change
• The depth will remain consistent
• The cameras will not change any of the above criteria after a firmware update
• The ownership and permissions will remain consistent
• ...and whatever else we might have overlooked

This. Is. Why. Such. Things. Are. Left. To. DVR. Or. NVR. Software.