Faster Searching from Mac Spotlight or Finder of Freenas Files

[kamal juck]

[ Before anyone yells at me: I’VE tried to Search for this and haven’t found anything useful in RECENT posts, if there is an answer could you point me to the solution; OR for Posting this in the wrong place ]

Hello All,

Over xMass, I helped my brother get his Synology NAS up running. I love my FreeNas for cost and expandability, but Synology web graphical user interface, included services and apps was impressive. The BIGGEST thing that blew me away was the speed of folder refresh and search from my MacBook. I noticed a service called “Universal Search”. Once I enabled it for one of his Disk Pools, the NAS seemed index all the files, and Presto Magic fast search! (See attached images)

My Mac was able to access Universal Search index from the finder. To give you example of speed of the Synology, I had loaded up a folder (40 GB or 30,000 files) with family photos. Opening the folder from my Mac with:

Universal Search Opening the folder
———————— —————————
OFF > +2 min
ON ~ 5 sec.

Wow, Search from the Mac was also very very fast! I tried this several times to make sure that the Mac hadn’t done something in the background, both my brother and I had similar results from our Macs..

I WANT THIS FEATURE on my FreeNas! I can’t be the only one out there.

I want to store more of my files on the Freenas, but speed of search and folder opening has stopped me in the past. I currently use a very old MacPro (2006) as a file server, it works, but not ideal, and it’s getting old, I’m worried if it fails.

Can someone help or point me in the right direction with my FreeNas

- is there feature or something included with FreeNas (11.2-U7) that I should use?
- Is there a PlugIn that exists that would allow me to do this?
- Is there some setting with my AFP shares that I might not have enabled?
- Do SMB shares have this ability?
- Is there an external package that I can install like Synology’s Universal Search, hopefully with step-by-step instruction for Freenas?
- OR just give-up and buy a Synology [when I win the lottery] :)

I’ve been using FreeNas for several years and only use -
- Plex Server
- AFP shares
- TimeMachine backup on some of the AFP pools
- I just upgraded from FreeNas 9 to 11 recently.

I’m a Mac user, and most the devices in my household are running iOS or OSX. The main day-to-day Macs are running 10.14 (Mojave) or 10.11 (El Cap) that need to get at files (fast)

My knowledge level: just enough to be dangerous and screw-up things :). I learn by following step-by-step tutorials

Thanks,
Kamal

 

[anodos]

I have this on my radar for the future. No earlier than FreeNAS 12.0. Will basically allow a separate lucene instance (like diskover) to index our shares and then have Samba query against the lucene instance.

 

[kamal juck]

Great, I think? What is the timeline for FreeNAS 12.0?

 

[anodos]

Great, I think? What is the timeline for FreeNAS 12.0?

We've not released 11.3 yet, and so there's a long road ahead.

 

[kamal juck]

Anodos, to your knowledge , has anyone been able to run lucene (or something else) on on FreeNAS at this time to get the job done?

 

[anodos]

Anodos, to your knowledge , has anyone been able to run lucene (or something else) on on FreeNAS at this time to get the job done?

The problem is that you need three components to this: (1) parser (2) indexer (3) metadata server. The parser converts Spotlight protocol queries into queries on the remote (or local) metadata server and the indexer updates the metadata server. Support for lucene as a metadata server was not added to Samba until 4.11 (11.3 is on 4.10). I may upgrade 11.3 to Samba 4.11 at a later date, but this will not work still on FreeNAS because Samba in FreeNAS isn't compiled with spotlight support.

 

[diedrichg]

Opening the folder from my Mac with:

Universal Search Opening the folder
———————— —————————
OFF > +2 min
ON ~ 5 sec.

I see the same issue when I'm using Linux. It's painfully slow over SMB, so much so that I will boot into my Windows partition when I know I'm going to be doing some heavy browsing.

 

[gegtor]

Now that elasticsearch support is here

Any guide on setting this up?
@anodos

 

[TravisT]

I know this thread is dated, but we're into TrueNAS version 12, and I'm hoping this is on the horizon? Spotlight search would be great for us Mac users out there.

If not available natively, is there any resources to guide setting a 3rd party solution up on TrueNAS 12 U2?

 

[Patrick M. Hausen]

That feature was already discussed here: https://www.truenas.com/community/threads/how-to-enable-smb-spotlight-support.88491/
And the documentation is right here: https://wiki.samba.org/index.php/Spotlight_with_Elasticsearch_Backend

So to me it seems this will just take

• create a jail
• pkg install elasticsearch7
• sysrc elasticsearch_enable=YES
• service elasticsearch start
• configure Samba according to the documentation linked above

Am I missing anything?

Edit: yes, fscrawler needs to be installed, too. But it's Java so nothing that would prevent it from running in a jail from the outset.
Documentation here: https://fscrawler.readthedocs.io/en/latest/

 

[TravisT]

Awesome - thanks for those links. Hopefully they are beginner friendly. I am not familiar with any of those packages, and am a novice at setting up the jails. I'll read through the documentation and give it a go.

 

[Henning Kessler]

Just gave this a try today.

I installed Elasticsearch7 in one jail and FSCrawler in another. You have to install this version (https://oss.sonatype.org/content/re...HOT/fscrawler-es7-2.7-20210104.212634-153.zip) from this Github issue conversation (https://github.com/dadoonet/fscrawler/issues/1065) as the current latest fails to run on FreeBSD.

I got FSCrawler to run and to push data to Elasticsearch but I could not see any requests from Samba to Elasticsearch after configuring it as in the documentation on wiki.samba.org.

Does anybody has made it work completely yet?

 

[TravisT]

I'd be interested in any steps you followed to get FSCrawler installed. I was able to get Elasticsearch7 installed in a jail, and setup the global and share based options on SMB, but couldn't figure out where to even start with fscrawler. Most of what I found was geared towards windows.

 

[Henning Kessler]

I created a vnet jail (maybe vnet isn't necessary)

Code:

sudo iocage create --name "crawler" -r 12.2-RELEASE ip4_addr="vnet0|192.168.192.28/24" vnet="on" boot="on" host_hostname="crawler"

installed openjdk15 and wget via packages

Code:

sudo iocage pkg crawler install -y openjdk15 wget

logged in to the jail

Code:

 sudo iocage console crawler 

downloaded the specific Snapshot of FSCrawler

Code:

wget https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7-SNAPSHOT/fscrawler-es7-2.7-20210104.212634-153.zip

unzip

Code:

unzip fscrawler-es7-2.7-20210104.212634-153.zip 

cd into the dir

Code:

cd fscrawler-es7-2.7-SNAPSHOT/

run

Code:

bin/fscrawler JOBNAME

the command asks you that there is no job named like this and if it should create such a job
After that you edit the job configfile in ~/.fscrawler/JOBNAME/_settings.yaml

Code:

---
name: "JOBNAME"
fs:
  url: "PATH/TO/SHARE"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: false
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: false
  index_content: true
  attributes_support: false
  raw_metadata: false
  xml_support: false
  index_folders: true
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: false
elasticsearch:
  nodes:
  - url: "http://IPofELASTIC:9200"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: true

then back in to the FsCrawler dir and

Code:

bin/fscrawler JOBNAME

You should see

Code:

[root@crawler ~/fscrawler-es7-2.7-SNAPSHOT]# bin/fscrawler test
,----------------------------------------------------------------------------------------------------.
|       ,---,.  .--.--.     ,----..                                     ,--,            2.7-SNAPSHOT |
|     ,'  .' | /  /    '.  /   /   \                                  ,--.'|                         |
|   ,---.'   ||  :  /`. / |   :     :  __  ,-.                   .---.|  | :               __  ,-.   |
|   |   |   .';  |  |--`  .   |  ;. /,' ,'/ /|                  /. ./|:  : '             ,' ,'/ /|   |
|   :   :  :  |  :  ;_    .   ; /--` '  | |' | ,--.--.       .-'-. ' ||  ' |      ,---.  '  | |' |   |
|   :   |  |-, \  \    `. ;   | ;    |  |   ,'/       \     /___/ \: |'  | |     /     \ |  |   ,'   |
|   |   :  ;/|  `----.   \|   : |    '  :  / .--.  .-. | .-'.. '   ' .|  | :    /    /  |'  :  /     |
|   |   |   .'  __ \  \  |.   | '___ |  | '   \__\/: . ./___/ \:     ''  : |__ .    ' / ||  | '      |
|   '   :  '   /  /`--'  /'   ; : .'|;  : |   ," .--.; |.   \  ' .\   |  | '.'|'   ;   /|;  : |      |
|   |   |  |  '--'.     / '   | '/  :|  , ;  /  /  ,.  | \   \   ' \ |;  :    ;'   |  / ||  , ;      |
|   |   :  \    `--'---'  |   :    /  ---'  ;  :   .'   \ \   \  |--" |  ,   / |   :    | ---'       |
|   |   | ,'               \   \ .'         |  ,     .-./  \   \ |     ---`-'   \   \  /             |
|   `----'                  `---`            `--`---'       '---"                `----'              |
+----------------------------------------------------------------------------------------------------+
|                                        You know, for Files!                                        |
|                                     Made from France with Love                                     |
|                           Source: https://github.com/dadoonet/fscrawler/                           |
|                          Documentation: https://fscrawler.readthedocs.io/                          |
`----------------------------------------------------------------------------------------------------'

 

[TravisT]

Awesome writeup. I made a minor modification, hopefully simplifying things.

Instead of installing elasticsearch7 in a separate jail, I just installed it in this same jail and kept the loopback address in the config file above.

I also used "/mnt/POOLNAME" as the url, which hopefully works. Since this is running in a jail, I'm not sure that syntax is correct. Would I need smb//:FQDN or IP/mnt/POOLNAME instead?

 

[Henning Kessler]

Awesome writeup. I made a minor modification, hopefully simplifying things.

Instead of installing elasticsearch7 in a separate jail, I just installed it in this same jail and kept the loopback address in the config file above.

I also used "/mnt/POOLNAME" as the url, which hopefully works. Since this is running in a jail, I'm not sure that syntax is correct. Would I need smb//:FQDN or IP/mnt/POOLNAME instead?

I tried installing both into the same jail in the beginning as well but FSCrawler had issues with the old openjdk8 version which is installed as a dependency for Elasticsearch.

If you want to index folders from the TrueNAS host I think the best way would be to mount them via fstab into the jail.

something like:

Code:

sudo mkdir -p /mnt/POOLNAME/iocage/jails/JAILNAME/root/mnt/SHARENAME
sudo iocage fstab -a JAILNAME "/mnt/POOLNAME/.../SHARENAME  /mnt/SHARENAME nullfs  ro  0  0"
sudo iocage restart JAILNAME

you can now access the SHARENAME dir and its subdirectories from jail in read-only mode. But beware that the paths above are for a default TrueNAS installation. If you moved your default iocage datasets the paths will be different. You can do the same from GUI as well...

 

[Patrick M. Hausen]

FreeBSD explicitly permits to have multiple JDK versions installed. You could make openjdk15 the default and configure Elasticsearch to explicitly use openjdk8.

The mounts can all conveniently be managed in the UI

 

[TravisT]

FreeBSD explicitly permits to have multiple JDK versions installed. You could make openjdk15 the default and configure Elasticsearch to explicitly use openjdk8.

The mounts can all conveniently be managed in the UI

I'll have to dig into the JDK version. I haven't seen any issues yet, but I'm not sure I know what I'm even looking for. If you care to share how to do what you're suggesting, that would be great. Otherwise I'll head to google shortly.

On the mount points, I think I'm up and running. What's the best way to verify everything is working?

 

[Patrick M. Hausen]

pkg install openjdk8; pkg install openjdk15

If you install 8 first and 15 second, 15 should be the default if you just invoke e.g. java -version.
Elasticsearch has a startup parameter named elasticsearch_java_home which defaults to /usr/local/openjdk8.

So everything should work out of the box if you just install both Java versions.

If FSCrawler does not pick up Java 15, you can look at the manpage for javavm or set JAVA_HOME manually.

HTH,
Patrick

 

[Henning Kessler]

T

pkg install openjdk8; pkg install openjdk15

If you install 8 first and 15 second, 15 should be the default if you just invoke e.g. java -version.
Elasticsearch has a startup parameter named elasticsearch_java_home which defaults to /usr/local/openjdk8.

So everything should work out of the box if you just install both Java versions.

If FSCrawler does not pick up Java 15, you can look at the manpage for javavm or set JAVA_HOME manually.

HTH,
Patrick

thanks for sharing this