Quickly search index of all files on nas?

[Jens]

Hi, sometimes i need to find that one file, and it would be great if the nas could build up a index every night of all the files so i could search and file a file instant.
does that exist?

 

[Neil Whitworth]

Look into the locate command. According to google it should be updating its index weekly, however on my system (FreeNAS 8.3.1) it does appear to not be working. My guess would be because the DB it is attempting to update is on the USB drive which is read only.

I'm not familiar enough with freeBSD (more used to linux) to say what need changing to make this work, but I am sure it can be done.

Another option would be to use cron to run a script each night to create a simple text file with details of each file. Something like this shoudl do the trick:-

ls -laR /mnt/tank > /mnt/tank/index.txt

 

[Dusan]

Edit /etc/periodic.conf and set weekly_locate_enable="YES" (the last line in the file). This will schedule weekly update of the locate database. After you do the edit you can also trigger immediate update of the database by running /etc/periodic/weekly/310.locate
You can then use the locate command to search for files. Please note that the DB upgrade runs as user nobody, so files that are not accessible by user/group nobody will not be in the DB. Also remember that the database will be wiped on reboot as it is stored in /var/db (ramdisk).

 

[Jens]

ok, cool, thanks.

 

[cyberjock]

I can't find any info on this, but if you have millions of files, won't that database be REALLY big? To me this has "fail" written all over it. Am I out to lunch with this? I can't find much information on how big a locate database can get per file entry or even rough estimates. Anyone know?

 

[Dusan]

You can run locate -S to get statistics about the DB. Numbers for one of my build VMs:
[PANEL]Compression: Front: 17.66%, Bigram: 57.21%, Total: 12.37%
Filenames: 449791, Characters: 24021892, Database size: 2971845
Bigram characters: 1271671, Integers: 18637, 8-Bit characters: 0[/PANEL]So, less than 7 bytes per file (it uses some kind of compression). It is also possible to move the DB to a new location if needed.

 

[evilandy]

Thanks, I found this to be really useful thread. I cd'd into a music folder and then could find all of my cue files in that directory and recursively (to include sub directories) via

find . -name '*.cue'

Spent awhile trying to do this on OSX spotlight using AFP and direct location searching but to no avail.

The output is really useful. I had a full path from the music directory with a new line per entry. I found this very fast and I'm only using a basic AMD processor.

 

[diskdiddler]

Bump to this thread (for good reason)

Has 9.2.1.6 "broken" the updatedb command and the locate indexing service?
I'm new to FreeNAS but my locate / updatedb simply don't work ,I figured they were deliberately broken for the sake of performance or something?

[root@freenas] ~# updatedb
updatedb: Command not found.
[root@freenas] ~#

[root@freenas] /etc/periodic/weekly# ls -l
total 5
drwxr-xr-x 2 root wheel 512 Jul 4 08:59 ./
drwxr-xr-x 6 root wheel 512 Jul 4 07:08 ../
-rwxr-xr-x 1 root wheel 548 Jul 4 08:55 310.locate*
-rwxr-xr-x 1 root wheel 550 Jul 4 08:55 340.noid*
-rwxr-xr-x 1 root wheel 886 Jul 4 08:55 400.status-pkg*
-rwxr-xr-x 1 root wheel 532 Jul 4 08:55 999.local*
[root@freenas] /etc/periodic/weekly# 310.locate
310.locate: Command not found.
[root@freenas] /etc/periodic/weekly#

[root@freenas] /etc/periodic/weekly# locate updatedb
locate: database too small: /var/db/locate.database
[root@freenas] /etc/periodic/weekly#

 

[cyberjock]

Umm.. there is no updatedb on my 9.2.0 box.

I'm thinking you are confusing something somewhere...

And if you wanted to use locate you have to setup the database and such. And then on any update you have to redo the changes if I remember correctly.

 

[diskdiddler]

Well look, I've found this "find" command and you know - the only difference is, it'll thrash the entire filesystem but do a live search.
Locate and update will keep a cached copy for me which is much easier / faster to search. I guess I kind of have too much faith in what a USB key can do - but the develeopers have obviously done their best to reduce unnecessary writes to the USB key (and this would be one)

So all said and done, find will do the job I think - thanks anyhow.
(I'm about to make a big post as to WHY I needed to find something... elsewhere)

 

[cyberjock]

I use the find command and it definitely can suck when you have millions of files. Some people I've worked with have had billions of files.

If you *do* setup the locate database and such it does work very well. I've used it. The problem is you basically have to handle maintaining it yourself on every update to FreeNAS. Obviously doing cronjobs to update the database every night and such is a *very* good idea.

This is just one aspect of the "OS as an appliance" problem you inherit by choosing to use an "appliance".

 

[diskdiddler]

That's reasonable, I'm yet to deal with an upgrade and don't know what exactly it wipes / doesn't wipe on the system when you change the version.
I'm going to assume your data on the disks stays in tact the jails, the jail configuration but non standard stuff (not in the FreeNAS UI?) configured in the FreeBSD shell are a goner?

 

[cyberjock]

The update does secret stuff in the background.

FreeNAS' boot device has 4 partitions:

One is /data, two are OS partitions, and one is bootstrap crap I believe. (note two OS partitions)

When you first install FreeNAS it will install to partition #2. When you do an upgrade the upgrade installs on partition #3 and the boot partition is changed from #2 to #3. Next update partition #2 is overwritten and made the boot partition.

On an OS update the only stuff that is overwritten is said partition above. So any changes you need to make to the OS must be handled in the WebGUI and the update process will reapply those changes per your config file that is stored in /data. If you make changes yourself then on update they will naturally be MIA.

 

[diskdiddler]

Edit /etc/periodic.conf and set weekly_locate_enable="YES" (the last line in the file). This will schedule weekly update of the locate database. After you do the edit you can also trigger immediate update of the database by running /etc/periodic/weekly/310.locate
You can then use the locate command to search for files. Please note that the DB upgrade runs as user nobody, so files that are not accessible by user/group nobody will not be in the DB. Also remember that the database will be wiped on reboot as it is stored in /var/db (ramdisk).

Actually I did this yesterday and it looks like it's "fixed" stuff now. Locate command seems to be working (although manually typing "updatedb" doesn't work)

 

[diskdiddler]

Actually I did this yesterday and it looks like it's "fixed" stuff now. Locate command seems to be working (although manually typing "updatedb" doesn't work)

Ok I've thought a bit more about this and I gotta be honest, it would be nice if I could turn it on permanently.
Anyone know some way I can manage to do this for my build? I realise many changes are discarded on reboot but surely it's possible to make a script, even if it's a jail with permission to write back to the root somehow??


It really is as simple as

Edit /etc/periodic.conf and set weekly_locate_enable="YES" (the last line in the file). This will schedule weekly update of the locate database. After you do the edit you can also trigger immediate update of the database by running /etc/periodic/weekly/310.locate

 

[diskdiddler]

Sorry to bump but surely I'm not the only person who would like to make changes to the fundamental parts of FreeNAS which aren't in a jail?
This is possible right?

 

[Knowltey]

Sorry to bump but surely I'm not the only person who would like to make changes to the fundamental parts of FreeNAS which aren't in a jail?
This is possible right?

Yes.

Don't.

 

[diskdiddler]

I'm not understanding your reply, can someone helpful please respond?

 

[Knowltey]

I'm not understanding your reply, can someone helpful please respond?

Yes, it is technically possible to change the fundamentals. However this is highly not recommended to do.

 

[diskdiddler]

Have you actually read what I'm proposing to change, or are you just deciding to make a snap, steadfast decision just to feel better about weighing in on a topic? There's consistently a staunch attitude of "this way or the highway" on the forums and frankly, it's tiring.
(Exactly the same issue when I had 1/3 the performance from my machine as I should have, but people insisted "it's probably normal")


I'm wanting to enable a single service, which indexes the filesystem once a week. I'm not running a huge cluster of 940 drives, I don't need 99.999999999999999% uptime, nor do I need the upper echelon of performance endlessly.

This is what I believe to be a cron job, that's (to my knowledge) enabled as per standard on most linux builds. I'm willing to take the risk or running a feature which has been in for I'd say at least a decade.
I don't even care if I lose the functionality on update of FreeNAS, but I would like it to remain on after a reboot.

Can someone please advise how to do this.
Thank you.