script to move some files with a timer.

[cyberjock]

So here's the situation: I have 700000 files in one folder(all *.db). To minimize loading on the server I'd like to move 10000 at a time, then pause for 2 minutes. I don't need any particular order. I just want to prevent moving them all at once. All of the files in this folder are .db so no need to figure out which files are db and which aren't.

Basically, I want to be able to do this:

mv /mnt/tank/folder1/*.db to /mnt/tank/folder2 (but only 10000 from the source folder).
wait 2 minutes
repeat above 2 steps until /mnt/tank/folder1 is empty.

I have a program that will import the .dbs into our new bigger database and it monitors the folder for changes, but I don't want to dump all of the databases at the same time or the new database will go crazy for days. So I need to feed it small amounts at a time.

So how do I build a script to do this? I've tried alot of Google searches but I'm getting nowhere fast. I figured I'd ask here since there's lots of helpful smart people that probably can provide the answer off the top of their head.

 

[dbanck]

Quick bash script I just wrote:

Code:

#!/bin/bash

cnt=0

for i in *.db; do
    mv $i folder/

    cnt=$((cnt+1))
    if [[ $((cnt % 10000)) -eq 0 ]]; then
        sleep 120
    fi
done

I moves all *.db files in the current folder into `folder/`.
And pauses every 10000 items for 120 seconds.

I haven't tried it for 70000 items, but for 50 items it just worked fine.

 

[cyberjock]

Wow. You are awesome. It ran great for 1 iteration. Then I found out there's subfolders. I was told there was no subfolders. Haha. Gonna do some more googling to see if I can figure out how to make it traverse folders automatically.


Edit: Haha. I just tried to do a mv *.db /mnt/tank/folder2 and I got the error "/bin/mv: Argument list too long." I'm in deep now.

Another edit: I just found out there's over 1.8 million db files in a bajillion folders and subfolders. What a darn mess!

I will say that trying to do a searh for "freebsd for command" really gives results that have nothing to do with the 'for' command. I'm definitely having fun learning this stuff!

 

[dbanck]

Edit: Haha. I just tried to do a mv *.db /mnt/tank/folder2 and I got the error "/bin/mv: Argument list too long." I'm in deep now.

This error occours becase it are too many files for mv to move. Doing this with a loop will prevent this error, because we're moving the files one by one.

The script is bash, nothing freebsd specific. Here is some basic information about bash loops: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-7.html
If you want to include sub directories you have to modify the script like this:

Code:

#!/bin/bash

cnt=0

for i in $(find . -name '*.db'); do
    mv $i /home/daniel/folder/
    cnt=$((cnt+1))

    if [[ $((cnt % 10000)) -eq 0 ]]; then
        sleep 120
    fi
done

`find .` will search in the current directory, including all sub directories for files matching '*.db'.

 

[cyberjock]

Just wanted to post back that your code worked brilliantly. I also learned alot by getting to see some bash commands in action.