SMB Pool is acting as case sensitive

[JBK]

TrueNAS (purchased from IXsystems)
TRUENAS-MINI-3.0-X
TrueNAS-12.0-U8.1

The POOL "Production" was created as an SMB pool (case insensitive) and it says insensitive in the pool settings.

However, I have an application that scans images and creates .jpg files from them. It runs on Windows and according to the author it uses standard Windows calls. When it scans the images it save them as IMAGE.JPG, however when it tries to read them back (same software) it looks for *.jpg and fails because it cannot fin them. I finally traced this down using procmon.exe and hours of testing. It is acting case-insensitive as that is how it was written (specifically for Windows).

TrueNAS is acting CASE-SENSITIVE even though I set the pool (and the share) to case-insensitive.

On a command prompt:

If Ido a:

dir *.JPG (uppercase ".JPG") all images are listed

If I do a:

dir *.jpg (lowercase ".jpg") I get "File Not Found"

Am I missing something or losing my mind? Any input would be appreciated.

 

[JBK]

UPDATE:

I just created a new dataset and set it to case SENSITIVE and now Windows ignores the case, dir *.JPG and dir *.jpg act as if there was no difference.

 

[anodos]

TrueNAS (purchased from IXsystems)
TRUENAS-MINI-3.0-X
TrueNAS-12.0-U8.1

The POOL "Production" was created as an SMB pool (case insensitive) and it says insensitive in the pool settings.
View attachment 56102

However, I have an application that scans images and creates .jpg files from them. It runs on Windows and according to the author it uses standard Windows calls. When it scans the images it save them as IMAGE.JPG, however when it tries to read them back (same software) it looks for *.jpg and fails because it cannot fin them. I finally traced this down using procmon.exe and hours of testing. It is acting case-insensitive as that is how it was written (specifically for Windows).

TrueNAS is acting CASE-SENSITIVE even though I set the pool (and the share) to case-insensitive.

On a command prompt:

If Ido a:

dir *.JPG (uppercase ".JPG") all images are listed
View attachment 56104

If I do a:

dir *.jpg (lowercase ".jpg") I get "File Not Found"
View attachment 56105

Am I missing something or losing my mind? Any input would be appreciated.

What were your share settings? Are these nested datasets? Case-sensitivity = insensitive means we're case-insensitive, but case-preserving, which is what Windows is.

 

[JBK]

What do you mean by share settings? I can get you any information you need. If I change it to not case preserve will it change the entire dataset (which I want).

I never use nested datasets so this is just a dataset from the root pool.

Thank you.

 

[JBK]

This dataset is encrypted, does that make a difference?

 

[anodos]

Code:

Try "help" to get a list of possible commands.
smb: \> ls
  .                                   D        0  Thu Mar 24 07:56:54 2022
  ..                                  D        0  Tue May 17 13:03:58 2022
  testfile2                           A       63  Thu Mar 24 07:57:04 2022

        970496 blocks of size 1024. 970368 blocks available
smb: \> allinfo Testfile2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes
smb: \> allinfo testfile2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes

smb: \> allinfo TESTFILE2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes

Code:

root@truenas[~]# zfs get casesensitivity dozer/smb-vss/sub1
NAME                PROPERTY         VALUE        SOURCE
dozer/smb-vss/sub1  casesensitivity  insensitive  -

 

[anodos]

I don't see problems with case-insensitive name lookups. I need clear repro case.

 

[JBK]

I do not see the allinfo command on the Tnas command line, (command not found). Here is the zfs output:

Code:

NAME             PROPERTY         VALUE        SOURCE
tank/Production  casesensitivity  insensitive  -

Let me inject something:

I had to do a STRIP-ACL on this volume while deploying and then create a new ACL and apply to children. It took an hour as there are over 14 million files in here.

Could that have affected the case issue?

 

[JBK]

Looks like I am going to end up having to migrate all of this data to a new dataset one way or another, that will take about 10 days but worth it if I can get it to work. I am creating several datasets right now (SMB/GENERIC/SENS/INSENS/MIXED) to test.

 

[JBK]

I ask that question because when I do and ls from the cli I get ... permissions are different from all other data sets

 

[JBK]

Ok, I have completed testing.

All datasets created with "insensitive" setting on case (SMB/Generic insensitive) - FAILED. They distinguish between case.
All datasets created with "sensitive" setting on case (Generic Mixed/Sensitive) - PASS. They ignore the case when being accessed.

The funny thing is this:

I have 13 TrueNAS units deployed so far as I am slowly trying to replace all legacy Windows storage.
I have also tested this on my other units and they do NOT exhibit this behavior. (They are all SMB/insensitive pools/sahres)

I am now concerned that if this pops up on one of the other units that are about to go into production (they all have SMB/insensitive - pools/shares) then I am going to be dead in production as this is around 500TB of data that I would have to migrate in order to return to work.

Is it a good idea to move to TrueNAS after all? I really would like and answer to this and can provide you with all of the information that you need to dig deep (from multiple systems).

 

[anodos]

I need a clear way to reproduce what you are seeing.

 

[JBK]

Unfortunately I do not know how. Any other system I use and it seems to work like is should. So even if I gave you specific steps that I am doing it would probably work for you on your system.

The ONLY difference is that this system is a TrueNAS IXsystems Mini-X. I own three of these. At almost $4000 each for a 4 core PC with 5 drive bays I would have expected more. Something has gone haywire on this one, I have already had to replace one of the three as DOA so this makes 2 out of 3 that now worry me. This is serious data that could make or break the company.

Otherwise I am going to keep experimenting and hope I come to a conclusion before the Windows system runs out of space, at which time I will probably have to move back to Windows storage.

 

[anodos]

Unfortunately I do not know how. Any other system I use and it seems to work like is should. So even if I gave you specific steps that I am doing it would probably work for you on your system.

The ONLY difference is that this system is a TrueNAS IXsystems Mini-X. I own three of these. At almost $4000 each for a 4 core PC with 5 drive bays I would have expected more. Something has gone haywire on this one, I have already had to replace one of the three as DOA so this makes 2 out of 3 that now worry me. This is serious data that could make or break the company.

Otherwise I am going to keep experimenting and hope I come to a conclusion before the Windows system runs out of space, at which time I will probably have to move back to Windows storage.

Your example has an auxiliary parameter set. Perhaps that's causing the issue?

 

[JBK]

I tried with and without the aux. No change.

Is it possible to create a new dataset and then use the CLI to migrate the data? Or do I have to migrate with robocopy over the network?

MY MAIN CONCERN: The next TrueNAS update may fix it, and I will have to migrate back again and that takes about 8 days.

 

[JBK]

Your example has an auxiliary parameter set. Perhaps that's causing the issue?

Is there any FILE attribute that could cause this case sensitive behavior? For example - could a file migration (copy/sync) application cause this?

 

[NugentS]

You can just ssh in, run screen and then cp (or indeed mv) the files from a to b. A lot quicker than going via the network and another machine.

You run screen to avoid the ssh session vanishing/closing/timing out on you and taking the cp/mv process with it.

I can't comment on the SMB case sensitive bit

 

[JBK]

You can just ssh in, run screen and then cp (or indeed mv) the files from a to b. A lot quicker than going via the network and another machine.

You run screen to avoid the ssh session vanishing/closing/timing out on you and taking the cp/mv process with it.

I can't comment on the SMB case sensitive bit

I really appreciate the information. I did not want to have to transfer 8TB over a 1gig link.

 

[JBK]

Code:

Try "help" to get a list of possible commands.
smb: \> ls
  .                                   D        0  Thu Mar 24 07:56:54 2022
  ..                                  D        0  Tue May 17 13:03:58 2022
  testfile2                           A       63  Thu Mar 24 07:57:04 2022

        970496 blocks of size 1024. 970368 blocks available
smb: \> allinfo Testfile2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes
smb: \> allinfo testfile2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes

smb: \> allinfo TESTFILE2
altname: TJ9RF0~N
create_time:    Thu Mar 24 07:56:54 2022 PDT
access_time:    Thu Mar 24 07:56:54 2022 PDT
write_time:     Thu Mar 24 07:57:05 2022 PDT
change_time:    Thu Mar 24 07:57:05 2022 PDT
attributes: A (20)
stream: [:smb2_stream:$DATA], 63 bytes
stream: [::$DATA], 63 bytes

Code:

root@truenas[~]# zfs get casesensitivity dozer/smb-vss/sub1
NAME                PROPERTY         VALUE        SOURCE
dozer/smb-vss/sub1  casesensitivity  insensitive  -

FINALLY, I have been able to narrow it down.

The issue is only happening when you use WILDCARDS. The application is doing a file scan for *.jpg,*.tif etc .. when it starts.

Code:

smb: \*******\Scan\DR\DR269\0\Test\> ls
  .                                   D        0  Tue Jun 21 10:31:14 2022
  ..                                  D        0  Tue Jun 21 10:18:45 2022
  Test.JPG                            A  2184496  Fri Mar 25 07:58:26 2022

                22045214362 blocks of size 1024. 12787236119 blocks available

smb: \*******\Scan\DR\DR269\0\Test\> dir *jpg
NT_STATUS_NO_SUCH_FILE listing \*******\Scan\DR\DR269\0\Test\*jpg


smb: \*******\Scan\DR\DR269\0\Test\> dir *JPG
  Test.JPG                            A  2184496  Fri Mar 25 07:58:26 2022

                22045214348 blocks of size 1024. 12787236105 blocks available
smb: \*******\Scan\DR\DR269\0\Test\>

 

[JBK]

@anodos UPDATE:

I was able to reproduce the issue. I built a new TrueNAS server fresh drives/pool.

I took a small dataset from the problem Mini-X and used ZFS replication to mirror it over and I ran into exactly the same issue.

Further, I took the same small dataset and used RSYNC to migrate to the new server and the problem goes away, which is consistent with copy/robocopy/any_tool that I have tried.

It appears that the issue is in the attributes of the dataset/zfs structure itself.

Tomorrow I will try another test using RSYNC and set the "preserve all attributes" flag and see what happens.