My first cronjob; scripting difficulties.

[Sokonomi]

I'm a complete novice to the nix verse, so all of this is some pretty hairy territory for me;

What I'm trying to do, is to get a cronjob to occasionally delete the oldest folder in a dataset, whenever it detects free space dropping below a threshold. I believe pruning is the word for this function. I've struggled through some google hits and tutorials and managed to write a bash script.. I think.

Code:

```bash
#!/usr/bin/env sh

# Set the minimum free disk space in gigabytes
MIN_FREE_SPACE_GB=1

# Get the current free disk space in bytes
free_space=$(df -Pk . | awk 'END{print $4}')

# Convert the free space from kilobytes to gigabytes
free_space_gb=$((free_space / 1024 / 1024))

# Check if the free space is below the minimum threshold
if [ "$free_space_gb" -lt "$MIN_FREE_SPACE_GB" ]; then
    # Find the oldest directory and delete it
    oldest_dir=$(ls -t1d */ | head -n 1)
    # rm -rf "$oldest_dir"
    echo "Deleted $oldest_dir as free space was below $MIN_FREE_SPACE_GB GB."
else
    echo "Sufficient free space available."
fi
```

I've commented out the actual delete command since I am not sure how to point this thing in the right folder yet, but it should be spitting out some info when you run it, I think. But here's the issue; it doesn't.

From what I can gather, bash should be installed out of the box. I'm trying to execute the script from Shell, as root. Whenever I run the script it just drops back to command without echoing anything, zero feedback. Does anyone know what could be the issue?

I'm running TrueNAS-12.0-U8.1

 

[Patrick M. Hausen]

I don't see anything fundamentally wrong by just reading your script. For debugging shell scripts the silver bullet is sh -x /path/to/script.

HTH, Patrick

 

[winnielinnie]

What I'm trying to do, is to get a cronjob to occasionally delete the oldest folder in a dataset, whenever it detects free space dropping below a threshold. I believe pruning is the word for this function.

Keep in mind, this won't free up any space being held by a snapshot(s).

So in theory, you might think this script will "free up space in an emergency", when in fact no space is freed on your pool.

 

[Sokonomi]

I don't see anything fundamentally wrong by just reading your script. For debugging shell scripts the silver bullet is sh -x /path/to/script.

HTH, Patrick

All that returned was '+bash'
And I unfortunately don't know what that means. :')

Keep in mind, this won't free up any space being held by a snapshot(s).

So in theory, you might think this script will "free up space in an emergency", when in fact no space is freed on your pool.

The dataset in question is just a surveillance camera dumptank, so no snapshots are involved. I just want to prevent the pool from filling up.

 

[winnielinnie]

The dataset in question is just a surveillance camera dumptank,

Doesn't software used for surveillance footage usually have its own built-in pruning mechanism?

 

[Sokonomi]

Doesn't software used for surveillance footage usually have its own built-in pruning mechanism?

The cameras do have an 'overwrite' feature, unfortunately this only applies to their own SDcard, not the FTP server it offloads to. They also have a 'delete older than x days' feature which also doesn't seem to work through FTP.. A bit inconvenient, but im hoping I can get the NAS to auto maintain enough disk space for cameras to offload to.

 

[winnielinnie]

Code:

oldest_dir=$(ls -t1d */ | head -n 1)

Heads up (no pun intended), this will declare your most recently modified directory.

You either need to invoke -r in your "ls" portion (to reverse the order), or pipe it into "tail" (instead of "head".)

One of these changes is needed. Not both.

 

[winnielinnie]

This seems to work for me.

Keep in mind the "rm" part is commented out. I also included redundant echos near to the end as a test. (Those can be removed.)

I used "bytes" instead of GB to keep things simple. I also use a "zpool" command, to keep the code cleaner and more practical to ZFS in general.

Code:

#!/usr/bin/env sh

STORAGE_POOL=mypool
MIN_FREE_SPACE=1073741824
# Hint: 1073741824 bytes = 1 gibibyte

# Confirm $STORAGE_POOL is a valid pool, or else just abort
if ! zpool status $STORAGE_POOL > /dev/null; then
    echo "$STORAGE_POOL is an invalid pool name!"
    exit 1
fi

# Get and declare the pool's free space in bytes
POOL_FREE_SPACE=$(zpool get -H -p free $STORAGE_POOL | cut -f3)

# Get and declare the oldest directory in current working directory
OLDEST_DIR=$(ls -rt1d */ | head -n 1)

# Check if the free space is below the minimum threshold
if [ $POOL_FREE_SPACE -lt $MIN_FREE_SPACE ]; then
    # Find the oldest directory and delete it
    # rm -rf $OLDEST_DIR
    echo "Deleted $OLDEST_DIR as $STORAGE_POOL free space was below $MIN_FREE_SPACE bytes."
else
    echo "Sufficient free space available on $STORAGE_POOL. Otherwise, $OLDEST_DIR would have been deleted."
fi


# YOU CAN DELETE THESE
echo $POOL_FREE_SPACE
echo $OLDEST_DIR
echo $STORAGE_POOL
echo $MIN_FREE_SPACE

 

[Sokonomi]

This seems to work for me.

Keep in mind the "rm" part is commented out. I also included redundant echos near to the end as a test. (Those can be removed.)

I used "bytes" instead of GB to keep things simple. I also use a "zpool" command, to keep the code cleaner and more practical to ZFS in general.

Code:

#!/usr/bin/env sh

STORAGE_POOL=mypool
MIN_FREE_SPACE=1073741824
# Hint: 1073741824 bytes = 1 gibibyte

# Confirm $STORAGE_POOL is a valid pool, or else just abort
if ! zpool status $STORAGE_POOL; then
    echo "$STORAGE_POOL is an invalid pool name!"
    exit 1
fi

# Get and declare the pool's free space in bytes
POOL_FREE_SPACE=$(zpool get -H -p free $STORAGE_POOL | cut -f3)

# Check if the free space is below the minimum threshold
if [ $POOL_FREE_SPACE -lt $MIN_FREE_SPACE ]; then
    # Find the oldest directory and delete it
    OLDEST_DIR=$(ls -rt1d */ | head -n 1)
    # rm -rf $OLDEST_DIR
    echo "Deleted $OLDEST_DIR as $STORAGE_POOL free space was below $MIN_FREE_SPACE bytes."
else
    OLDEST_DIR=$(ls -rt1d */ | head -n 1)
    echo "Sufficient free space available on $STORAGE_POOL."
fi

echo $POOL_FREE_SPACE
echo $OLDEST_DIR
echo $STORAGE_POOL
echo $MIN_FREE_SPACE

Oh thats a nice cleanup job, thank you!

I'm going over the code to try and understand what is happening, but one thing kinda bugs me; Will ls look only at $STORAGE_POOL to find the oldest directory, or will it go over everything, since cronjob will run it system wide? Its the thing I was kinda struggling with in my initial post as well, how do I "point it at" the correct pool? Id hate for it to start eating the wrong thing. It should only ever look at and manipulate /mnt/Tank2/

Also, why is OLDEST_DIR=$(ls -rt1d */ | head -n 1) also in the 'else' statement?

 

[winnielinnie]

Also, why is OLDEST_DIR=$(ls -rt1d */ | head -n 1) also in the 'else' statement?

I'm going over the code to try and understand what is happening

I had edited it since you quoted the post, just a heads up! (I modified "line 8" to suppress excessive output, as well as added a note near the bottom Other minor changes too.)

 

[winnielinnie]

I also suggest you add a line that will change into the absolute directory, to minimize the chance of this script accidentally running in the wrong directory. (It's up to you. Just keep in mind you have nothing in the script that specifies what directory to change into before removing anything...)

 

[Sokonomi]

I had edited it since you quoted the post, just a heads up! (I modified "line 8" to suppress excessive output.)

Ah, now it makes more sense.

It still seems to point to the working directory, but chronjob would be launching the script from somewhere else. Id hate for this to start eating my TrueNAS install.

I also suggest you add a line that will change into the absolute directory, to minimize the chance of this script accidentally running in the wrong directory. (It's up to you. Just keep in mind you have nothing in the script that specifies what directory to change into before removing anything...)

And then you post to point that out. ;-)

So then, can I just change
OLDEST_DIR=$(ls -rt1d */ | head -n 1)
to say
OLDEST_DIR=$(ls -rt1d /mnt/Tank2/ | head -n 1)
and have it pointed at my cam pool by definition?

 

[winnielinnie]

Here is an "updated" version to add a further level of safety:

Code:

#!/usr/bin/env sh

STORAGE_POOL=Tank2

# Confirm $STORAGE_POOL is a valid pool, or else just abort
if ! zpool status $STORAGE_POOL > /dev/null; then
    echo "$STORAGE_POOL is an invalid pool name!"
    exit 1
fi

STORAGE_POOL_ROOT=/mnt/$STORAGE_POOL

MEDIA_DIR=$STORAGE_POOL_ROOT/Surveillance

# Confirm $MEDIA_DIR is a valid path, or else just abort
if ! stat $MEDIA_DIR > /dev/null; then
    echo "$MEDIA_DIR is an invalid path!"
    exit 1
fi

MIN_FREE_SPACE=1073741824
# Hint: 1073741824 bytes = 1 gibibyte


# Get and declare the pool's free space in bytes
POOL_FREE_SPACE=$(zpool get -H -p free $STORAGE_POOL | cut -f3)

# Change into media directory
cd $MEDIA_DIR

# Get and declare the oldest directory in current working directory
OLDEST_DIR=$(ls -rt1d */ | head -n 1)

# Check if the free space is below the minimum threshold
if [ $POOL_FREE_SPACE -lt $MIN_FREE_SPACE ]; then
    # Find the oldest directory and delete it
    # rm -rf $OLDEST_DIR
    echo "Deleted $OLDEST_DIR as $STORAGE_POOL free space was below $MIN_FREE_SPACE bytes."
else
    echo "Sufficient free space available on $STORAGE_POOL. Otherwise, $OLDEST_DIR would have been deleted."
fi

 

[Sokonomi]

I also just realized, the parent directory will probably be older than any other directory within, and each camera has its own parent directory..

Tank2
- Surveillance
-- Counter
--- 16_10_2023
--- 17_10_2023
--- 18_10_2023
-- Workshop
--- 16_10_2023
--- 17_10_2023
--- 18_10_2023
-- Parking
--- 16_10_2023
--- 17_10_2023
--- 18_10_2023

This thing just got a whole lot more problematic, I think.. How should I approach this?

Or does modified date on directories automatically get updated by the contents changing? This doesnt appear to be the case.

 

[winnielinnie]

You're skating on a plank between two skyscrapers at this point. Trying to get "scripty" might end up destroying data where you couldn't foresee a future situation in which a small change in setup ends up destroying important data.

I also noticed you're using the pool's root dataset to save files directly within? That's a big "no-no"! I'm not sure how you managed, since there's a safeguard in the TrueNAS GUI that disallows changing permissions in the top-level root dataset.

 

[Sokonomi]

You're skating on a plank between two skyscrapers at this point. Trying to get "scripty" might end up destroying data where you couldn't foresee a future situation in which a small change in setup ends up destroying important data.

I also noticed you're using the pool's root dataset to save files directly within? That's a big "no-no"! I'm not sure how you managed, since there's a safeguard in the TrueNAS GUI that disallows changing permissions in the top-level root dataset.

Sorry, forgot there was a 'Surveillance' dataset wrapping that. Its 1:30am here. :')

 

[winnielinnie]

Sorry, forgot there was a 'Surveillance' dataset wrapping that. Its 1:30am here. :')

That's more reassuring. One issue out of the way.

Or does modified date on directories automatically get updated by the contents changing? This doesnt appear to be the case.

That's the scary part. You might inadvertently do something in one of those parent folders, and mess up the "order" of oldest/newest.



Another scary thing: Your cameras each have their own folder. The script (even the "safer" one I modified) will destroy an entire camera's folder.

You could add an "-R" flag to the "ls" command. But jeez this is getting precarious...

 

[Sokonomi]

That's more reassuring. One issue out of the way.




That's the scary part. You might inadvertently do something in one of those parent folders, and mess up the "order" of oldest/newest.



Another scary thing: Your cameras each have their own folder. The script (even the "safer" one I modified) will destroy an entire camera's folder.

You could add an "-R" flag to the "ls" command. But jeez this is getting precarious...

I guess I will have to run separate cronjobs for each camera to prevent it from eating a cameras root directory. Unless you can somehow work folder depth into this so it wont 'touch' anything that's less than 1 deep?

 

[winnielinnie]

Are you sure your cameras won't work with ZoneMinder?

You can create a FreeBSD 13.2-RELEASE jail (Basejail), switch to the "latest" repository, and then install and configure zoneminder in the jail.

Then just add the proper "mountpoint" for the jail (/mnt/Tank2/Surveillance), and go from there.

ZoneMinder can automatically prune old video files, among other things.

Seems more pragmatic than playing this dance of scripting.

 

[Patrick M. Hausen]

All that returned was '+bash'
And I unfortunately don't know what that means. :')

The first and the last line:
```bash
and
```
don't belong in the script, delete them.

That's markdown formatting instructions - how did they end up in your script? When reading your initial post I thought you put them in there to format for the forum ...

Every script should start with a line starting with #! followed by the interpreter for the script language.