Faster Searching from Mac Spotlight or Finder of Freenas Files

[kapitainsky]

Long story short:
-There are no plans for native Spotlight support (only spotlight backed enabled in latest versions of TrueNAS Core or Scale)
-This outlines what you should do https://wiki.samba.org/index.php/Spotlight_with_Elasticsearch_Backend

1) Prepare a Linux VM for fscrawler + ElasticSearch
2) Follow fscrawler docs and point it to your storage (via SSH for example)
3) Point fscrawler to your ElasticSearch install
4) Let it run and verify thats it's working via ElasticSearch web ui
5) Point your Samba Service under TrueNAS to your ElasticSearch instance
6) Verify with mdfind

Voilà!

Short story made even shorter - I think now only issue is with your point 5 - otherwise it is easy and works either in jail or VM.

 

[phradr]

OK could not wait for weekend and did all tests today.

Debian with Samba 4.15.9 compiled from source (the same version TrueNAS is using) works perfectly - so it is not problem with this particular Samba version.

Maybe problem is with FreeBSD? I realised that their pkg Samba (the latest available is 4.13.17) does not have Spotlight enabled so compiled from ports with Spotlight ON. All works perfectly - I can search files by their content. It is not FreeBSD issue.

These results more and more point into problem with TrueNAS Samba - they use their own fork.

Errors sequence we see in TrueNAS smb.log when ES response is received starts with:

Code:

[2022/09/29 15:56:50.745574,  0, pid=93402, effective(1000, 1000), real(0, 0)] ../../libcli/http/http.c:199(http_parse_response_line)
  http_parse_response_line: Error parsing header

and this is where it gets real interesting. FreeBSD samba port patches exactly this file - http.c - and exactly in place where this error is thrown:

patch-linuxisms « files « samba413 « net - ports - FreeBSD ports tree

cgit.freebsd.org

in vanilla Samba:

Code:

    n = sscanf(line, "%m[^:]: %m[^\r\n]\r\n", &key, &value);
    if (n != 2) {
        DEBUG(0, ("%s: Error parsing header '%s'\n", __func__, line));
        status = HTTP_DATA_CORRUPTED;
        goto error;
    }

but in FreeBSD:

Code:

#ifdef FREEBSD
    int s0, s1, s2, s3; s0 = s1 = s2 = s3 = 0;
    n = sscanf(line, "%n%*[^/]%n/%c.%c %d %n%*[^\r\n]%n\r\n",
           &s0, &s1, &major, &minor, &code, &s2, &s3);

    if(n == 3) {
        protocol = calloc(sizeof(char), s1-s0+1);
        msg = calloc(sizeof(char), s3-s2+1);

        n = sscanf(line, "%[^/]/%c.%c %d %[^\r\n]\r\n",
            protocol, &major, &minor, &code, msg);
    }
#else
     n = sscanf(line, "%m[^/]/%c.%c %d %m[^\r\n]\r\n",
            &protocol, &major, &minor, &code, &msg);
#endif
     if (n != 2) {
         DEBUG(0, ("%s: Error parsing header '%s'\n", __func__, line));
         status = HTTP_DATA_CORRUPTED;
         goto error;
    }

Could not find out TrueNAS Core Samba source code (I only started using TrueNAS and FreeBSD last week and still learning) to see what is there but I think that now we have enough info to create meaningful bug report. For this your traces would be extremely helpful - unfortunately I am not sure how to get them.

Hopefully some TrueNAS dev can look into it and figure out what is wrong with their Samba 4.15.9.

ES search over Samba really makes difference - works very well. Of course then it will be yet another story how to optimise it etc. Already can see that it needs good amount of RAM - I tried with 30k files and used like 5GB.

That, my friend, is the real issue! Great work and thanks for your analyzation! That strengthens my supposition :)

What I didn't think about - until you made it come up in my mind - is, that TrueNAS uses its own Samba fork.
The sources are available on GitHub (TrueNAS: LINK, Samba fork: LINK - attention: the branch is not quiet clear - at least for me)

Though you pointed to

Code:

    n = sscanf(line, "%m[^:]: %m[^\r\n]\r\n", &key, &value);
    if (n != 2) {
        DEBUG(0, ("%s: Error parsing header '%s'\n", __func__, line));
        status = HTTP_DATA_CORRUPTED;
        goto error;
    }

this is not perfectly correct. The real issues is in line 199 of "our sources" (within our running Samba stack). That looks a bit different and is only a few lines underneath your code (LINK, SCALE-v4-17-stable branch, =default):

Code:

165
/**
166
* Parses the first line of a HTTP response
167
*/
168
static bool http_parse_response_line(struct http_read_response_state *state)
169
{
170
    bool    status = true;
...
198
    if (n != 5) {
199
    DEBUG(0, ("%s: Error parsing header\n",    __func__));
200
    status = false;
201
    goto error;
202
    }
...
233
}

Anyway... As I don't know how the ES response should look like for (TrueNAS') Samba, I could not patch that on my own (pullrequest for ex.).

But you now have pointed to the exact position where we stand and we can derive, that this issue has to be resolved within TrueNAS - and not within the "vanilla" Samba sources (though it could also be possible, if TrueNAS pulls the master brancher from time to time...

So, any suggestions where to open a ticket? maybe at TrueNAS AND Samba?[/CODE]

 

[phradr]

@kapitainsky : What do you mean by that?

Short story made even shorter - I think now only issue is with your point 5 - otherwise it is easy and works either in jail or VM.

Targeting the ES API from the actual TrueNAS Samba fork settings is no big deal. Go to Services > SMB (pencil) > Auxiliary Parameters and activate:

Code:

spotlight backend = elasticsearch
elasticsearch:address = NAT IP (eg. 172.16.0.22) or DHCP IP (eg. 10.0.0.112)
elasticsearch:port = 9200
elasticsearch:index = scans (This is a need though it is marked as optional in Samba documentation!)

And in your share go to: Sharing > Windows Shares (SMB) > (sharename's) menu "Edit" > Auxiliary Parameters and set:

Code:

spotlight = yes

That's it.


- or did I get you wrong?

 

[kapitainsky]

What I didn't think about - until you made it come up in my mind - is, that TrueNAS uses its own Samba fork.
The sources are available on GitHub (TrueNAS: LINK, Samba fork: LINK - attention: the branch is not quiet clear - at least for me)

I got there but could not find Core 13 - Samba 4.15.9 code

this is not perfectly correct. The real issues is in line 199 of "our sources" (within our running Samba stack). That looks a bit different and is only a few lines underneath your code (LINK, SCALE-v4-17-stable branch, =default):

good catch - it was late for me when writing - included wrong part. Neverthless FreeBSD uses the same patched logic to calculate n variable in place you pointed too.

Code:

+#ifdef FREEBSD
+    int s0, s1, s2, s3; s0 = s1 = s2 = s3 = 0;
+    n = sscanf(line, "%n%*[^/]%n/%c.%c %d %n%*[^\r\n]%n\r\n",
+           &s0, &s1, &major, &minor, &code, &s2, &s3);
+
+    if(n == 3) {
+        protocol = calloc(sizeof(char), s1-s0+1);
+        msg = calloc(sizeof(char), s3-s2+1);
+
+        n = sscanf(line, "%[^/]/%c.%c %d %[^\r\n]\r\n",
+            protocol, &major, &minor, &code, msg);
+    }
+#else
     n = sscanf(line, "%m[^/]/%c.%c %d %m[^\r\n]\r\n",
            &protocol, &major, &minor, &code, &msg);
+#endif
 
-    DEBUG(11, ("%s: Header parsed(%i): protocol->%s, major->%c, minor->%c, "
-           "code->%d, message->%s\n", __func__, n, protocol, major, minor,
-           code, msg));
-
     if (n != 5) {
         DEBUG(0, ("%s: Error parsing header\n",    __func__));
         status = false;
         goto error;
     }

 

[kapitainsky]

@kapitainsk: What do you mean by that?


Targeting the ES API from the actual TrueNAS Samba fork settings is no big deal. Go to Services > SMB (pencil) > Auxiliary Parameters and activate:

Code:

spotlight backend = elasticsearch
elasticsearch:address = NAT IP (eg. 172.16.0.22) or DHCP IP (eg. 10.0.0.112)
elasticsearch:port = 9200
elasticsearch:index = scans (This is a need though it is marked as optional in Samba documentation!)

And in your share go to: Sharing > Windows Shares (SMB) > (sharename's) menu "Edit" > Auxiliary Parameters and set:

Code:

spotlight = yes

That's it.


- or did I get you wrong?

Ok maybe was not clear. What I meant was that point 5 does not work. Of course pointing Samba to ES is easy - but we want it to work.

 

[kapitainsky]

spotlight backend = elasticsearch elasticsearch:address = NAT IP (eg. 172.16.0.22) or DHCP IP (eg. 10.0.0.112) elasticsearch:port = 9200 elasticsearch:index = scans (This is a need though it is marked as optional in Samba documentation!)

I always use:

Code:

spotlight backend = elasticsearch
elasticsearch:address = 172.16.0.2
elasticsearch:port = 9200
elasticsearch:ignore unknown attribute = yes
elasticsearch:ignore unknown type = yes
elasticsearch:use tls = no

Re elasticsearch:index = scans - it might be required indeed in some more complex ES setup. One day when everything works we can start another thread to share best practices and findings:)

 

[kapitainsky]

So, any suggestions where to open a ticket? maybe at TrueNAS AND Samba?

They have fixed some issues in Spotlight implementation - looks like it would fix @seanm issue if TrueNAS ports it to their fork

Samba 4.15.10 - Release Notes

but otherwise I do not think there is any issue with Samba and ES from our perspective.

I would do TrueNAS ticket only -> and then see what some dev comes with

 

[phradr]

I got there but could not find Core 13 - Samba 4.15.9 code

Ah okay, well, I suspect they use the same source for CORE and SCALE.

Neverthless FreeBSD uses the same patched logic to calculate n variable

Damn right! And you might already have found the solution :)

Ok maybe was not clear. What I meant was that point 5 does not work. Of course pointing Samba to ES is easy - but we want it to work

Oh sorry, didn't want to be impolite.

One day when everything works we can start another thread to share best practices and findings

Great idea! I would participate :D

looks like it would fix @seanm issue if TrueNAS ports it to their fork

yes, could be. I don't have access to his mentioned ticket (LINK) - maybe @seanm could refer that?

 

[kapitainsky]

FYI:

MacOS support search in SMB shares

I have TrueNAS 13 CORE I have 1 Pool of 10TB I have 2 dataset in the Pool I have 2 shares of SMB User.. Groups.. Done I can navigate in the shares, the permisions are correct, BUT.... MacOS users cannot search for content within shared folders, but, if they can browse the files Does anyone...

www.truenas.com

some progress but it wont fix ES issues IMHO yet. They rather go oposite direction by setting:

rpc_daemon:mdssd = disabled
rpc_server:mdssvc = disabled

NAS-118498 / 13.0 / Disable the mdssd rpc daemon by anodos325 · Pull Request #9953 · truenas/middleware

Spotlight support via elasticsearch is currently broken and users can override this via auxiliary parameters if they're particularly ambitious.

github.com

and disabling Spotlight.

I hope it can be overwritten by auxillary parameters. [confirmed - it can]

PS. @phradr - somebody explained me why some jirra tickers are private - does not really help wiht community work -
"Issues are locked while users have private info uploaded to ticket."

 

[seanm]

yes, could be. I don't have access to his mentioned ticket (LINK) - maybe @seanm could refer that?

Oh, I didn't realize those links are not publicly visible without logging into jira. I didn't mark the ticket private when I created it. Anyway, it doesn't say much beyond what I wrote here and their fix is in the github link @kapitainsky linked above.

I went searching in jira for tickets with the word "spotlight" and there is this one:

[NAS-107345] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[kapitainsky]

Oh, I didn't realize those links are not publicly visible without logging into jira. I didn't mark the ticket private when I created it. Anyway, it doesn't say much beyond what I wrote here and their fix is in the github link @kapitainsky linked above.

I went searching in jira for tickets with the word "spotlight" and there is this one:

[NAS-107345] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

as I was told "Issues are locked while users have private info uploaded to ticket"

hahaha - so it means they decide which ticket is secret and which one is not

together with your comment it tells me that there are tones of tickets in jirra we can't see - so only iXsystems knows what they are working on with community kept in the dark.

ohh well - open community might have many meanings

 

[seanm]

as I was told "Issues are locked while users have private info uploaded to ticket"

I didn't add any private info to the ticket. Hiding such tickets make sense, but shouldn't be the case for my ticket.

hahaha - so it means they decide which ticket is secret and which one is not

It could just be a bug... Jira is well known to be a big steaming mess, from what I've heard. :) Can you not see any tickets without an account, or only some?

together with your comment it tells me that there are tones of tickets in jirra we can't see - so only iXsystems knows what they are working on with community kept in the dark.

I'm not with iXsystems. You can create an account on their jira for free, just requires giving them an email address.

 

[kapitainsky]

I didn't add any private info to the ticket. Hiding such tickets make sense, but shouldn't be the case for my ticket.


It could just be a bug... Jira is well known to be a big steaming mess, from what I've heard. :) Can you not see any tickets without an account, or only some?


I'm not with iXsystems. You can create an account on their jira for free, just requires giving them an email address.

I have account - still can not see your ticket. And not only yours. Experienced it already few times - very annoying.

 

[seanm]

I have account - still can not see your ticket. And not only yours. Experienced it already few times - very annoying.

I'm pretty sure that's a bug, and not some deliberate limitation. I'm able to see tickets created by many different people. I don't recall ever being unable to access a ticket. And I'm nobody, just a user. This thread can hopefully help you:

Jira Migration - 7/1/2022

TrueNAS Users, A quick heads up that this Friday 07/01/2022, iX IT staff will be performing a migration from our current on-prem Jira instance to our new Jira Cloud instance. Our existing Jira instance will be marked "Read-Only" at this point, while the open ticket data is migrated over. We...

www.truenas.com

 

[phradr]

Oh, I didn't realize those links are not publicly visible without logging into jira. I didn't mark the ticket private when I created it. Anyway, it doesn't say much beyond what I wrote here and their fix is in the github link @kapitainsky linked above.

I went searching in jira for tickets with the word "spotlight" and there is this one:

[NAS-107345] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Right today I installed SCALE on my esxi to play around with that.

It‘s different but with at least the same functionality PLUS utilizing docker instead of jails (obviously that makes sense as scale is based on debian and not bsd).

Maybe I choose the easy way and just move to scale…

Furthermore I know debian way better than freebsd

 

[kapitainsky]

Right today I installed SCALE on my esxi to play around with that.

It‘s different but with at least the same functionality PLUS utilizing docker instead of jails (obviously that makes sense as scale is based on debian and not bsd).

Maybe I choose the easy way and just move to scale…

Furthermore I know debian way better than freebsd

looking forward for your ES/Spotlight integration results.

 

[kapitainsky]

@phradr - are you going to open TrueNAS ticket re ES/Samba? I feel we should keep it rollling otherwise nothing happens. If not could I ask you to provide traces from your investigation? then I will open it.

I also noticed that Spotlight will be OFF in Scale

NAS-118498 / 22.12 / Disable spotlight-related RPC server by anodos325 · Pull Request #9955 · truenas/middleware

Until such a time as we officially support spotlight searches, disable the relevant RPC server. End-users can override this setting via auxiliary parameters.

github.com

 

[seanm]

@phradr - are you going to open TrueNAS ticket re ES/Samba?

Maybe https://ixsystems.atlassian.net/browse/NAS-107345 is already the ticket we need, and info should just be added to it?

Since you can't access jira, the ticket is:

---
Title: Add diskover / elasticsearch plugin to scale

Description: Cluster-wide search / indexing will probably be a good feature to have. Samba in 4.12 and later has ability to use elasticsearch as a backend for spotlight queries. There has also been some upstream work to support MS-WSP through same.
---

(Not sure why it refers only to Scale though.)

 

[phradr]

are you going to open TrueNAS ticket re ES/Samba?

I'll take care of that tomorrow as I still need to regenerate the debug logs and had a busy day.

Maybe https://ixsystems.atlassian.net/browse/NAS-107345 is already the ticket we need, and info should just be added to it?

Since you can't access jira, the ticket is:

---
Title: Add diskover / elasticsearch plugin to scale

Description: Cluster-wide search / indexing will probably be a good feature to have. Samba in 4.12 and later has ability to use elasticsearch as a backend for spotlight queries. There has also been some upstream work to support MS-WSP through same.
---

(Not sure why it refers only to Scale though.)

Thanks @seanm . As far as I undertand the ticket it targets to add a general ES functionality to SCALE. I doubt it will impact CORE as the basis of both differ extremely.

What you can't know: I wrote a Mod in the morning to have a look at this thread and maybe move it to the TrueNAS forum as it is in FreeNAS (legacy) actually. I also asked him whom to address regarding the ticket. If I don't get an answer until tomorrow night I'll create a ticket at iXsystem AND Samba, as it might be still a general issue.

 

[kapitainsky]

Maybe https://ixsystems.atlassian.net/browse/NAS-107345 is already the ticket we need, and info should just be added to it?

Since you can't access jira, the ticket is:

---
Title: Add diskover / elasticsearch plugin to scale

Description: Cluster-wide search / indexing will probably be a good feature to have. Samba in 4.12 and later has ability to use elasticsearch as a backend for spotlight queries. There has also been some upstream work to support MS-WSP through same.
---

(Not sure why it refers only to Scale though.)

Looks like future story - elasticsearch plugin - so you just click,click and samba share can be searched. First Samba/ES integration has to work - which at the moment is not.