NTLM auth over network fails for new accounts, works with old accounts.

[MkIVMark]

Hi Everyone,


TrueNAS-SCALE-22.12.4.2 installation that has been running fine for a couple of months. Existing dataset with UNIX ACLs and an SMB share that I need to access from an iPad. Accessing it from the iPad did not work, so I created a test dataset + share using NFSv4 ACLs and SMB sharing which worked well with the iPad. Changed the existing dataset to NFSv4 ACLs, re-created the SMB share and tested these from MacOS 14.1 (my desktop). Connections to SMB shares using existing accounts where the password has been changed, or newly created accounts, all fail with a wrong password error. There is one account that existed before the ACL switch where the password has not been changed - accessing SMB shares using this account still works. Using smbclient on the TrueNAS server to make local SMB connections (127.0.0.1) authenticates successfully with new accounts, but only after a passdb sync.


So in summary:

• Network access over IPv4/6 to SMB shares using new accounts or where the password on an existing account has been changed fails with a NT_STATUS_WRONG_PASSWORD.
• Network access over IPv4/6 to SMB shares using an account setup before the ACL switch, that has not had the password changed, succeeds.
• Local access on the TrueNAS server using smbclient works for all accounts but only after I sync'd the passdb.
• Even after the passdb is sync'd, the network IPv4/6 access still fails for new accounts and existing accounts that had the password changed.

Below is the results of the various commands that I ran during diagnosis of this issue which might be helpful.


I have spent quite a few hours trying to figure this out but I am really at my wits end. Can anyone suggest some more diagnosis options? Or is there something in the network auth that needs to be reset? Any help would be most welcome.


Cheers
Mark


"sudo pdbedit -Lwv" shows all the accounts it should, including the new test account that fails and the lisbeth account (the older one with an unchanged password) that still works. Note that they both have the same NT hash as they both use the same password, so I know the password is being entered correctly on the MacOS client (the only difference between connection attempts is the account name).


Unix username: test
NT username:
Account Flags: [U ]
User SID: S-1-5-21-2813400636-2121464641-797741664-20072
Primary Group SID: S-1-5-21-2813400636-2121464641-797741664-513
Full Name: test
Home Directory: \\TRUENAS\test
HomeDir Drive:
Logon Script:
Profile Path: \\TRUENAS\test\profile
Domain: TRUENAS
Account desc:
Workstations:
Munged dial:
Logon time: 0
Logoff time: Wed, 06 Feb 2036 16:06:39 CET
Kickoff time: Wed, 06 Feb 2036 16:06:39 CET
Password last set: Sun, 29 Oct 2023 18:57:52 CET
Password can change: Sun, 29 Oct 2023 18:57:52 CET
Password must change: never
Last bad password : 0
Bad password count : 0
Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
LM hash : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
NT hash : C4E66191EE03F8BF1B618866CE5E748F


Unix username: lisbeth
NT username:
Account Flags: [U ]
User SID: S-1-5-21-2813400636-2121464641-797741664-20068
Primary Group SID: S-1-5-21-2813400636-2121464641-797741664-513
Full Name: Lisbeth
Home Directory: \\TRUENAS\lisbeth
HomeDir Drive:
Logon Script:
Profile Path: \\TRUENAS\lisbeth\profile
Domain: TRUENAS
Account desc:
Workstations:
Munged dial:
Logon time: 0
Logoff time: Wed, 06 Feb 2036 16:06:39 CET
Kickoff time: Wed, 06 Feb 2036 16:06:39 CET
Password last set: Sun, 29 Oct 2023 15:37:17 CET
Password can change: Sun, 29 Oct 2023 15:37:17 CET
Password must change: never
Last bad password : 0
Bad password count : 0
Logon hours : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
LM hash : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
NT hash : C4E66191EE03F8BF1B618866CE5E748F




------------------------------------------------------------------------------------------------------


"sudo midclt call smb.status AUTH_LOG | jq" shows two auth events, one success using the lisbeth account and a failure using the newly created test account (I have X'd out the IPv6 network ID, but the rest is unchanged).


{
"timestamp": "2023-10-29T19:00:22.343841+0100",
"type": "Authentication",
"Authentication": {
"version": {
"major": 1,
"minor": 2
},
"eventId": 4624,
"logonId": "0",
"logonType": 3,
"status": "NT_STATUS_OK",
"localAddress": "ipv6:XXXX:XXXX:XXXX:XXXX:7c8b:d5ff:fedb:7077:445",
"remoteAddress": "ipv6:XXXX:XXXX:XXXX:XXXX:6d3a:f06c:bcbf:7cb7:51353",
"serviceDescription": "SMB2",
"authDescription": null,
"clientDomain": "TRUENAS",
"clientAccount": "lisbeth",
"workstation": "MERCURY",
"becameAccount": "lisbeth",
"becameDomain": "TRUENAS",
"becameSid": "S-1-5-21-2813400636-2121464641-797741664-20068",
"mappedAccount": "lisbeth",
"mappedDomain": "TRUENAS",
"netlogonComputer": null,
"netlogonTrustAccount": null,
"netlogonNegotiateFlags": "0x00000000",
"netlogonSecureChannelType": 0,
"netlogonTrustAccountSid": null,
"passwordType": "NTLMv2",
"duration": 2443
},
"timestamp_tval": {
"tv_sec": 1698602422,
"tv_usec": 343841
}
},
{
"timestamp": "2023-10-29T19:00:47.934725+0100",
"type": "Authentication",
"Authentication": {
"version": {
"major": 1,
"minor": 2
},
"eventId": 4625,
"logonId": "0",
"logonType": 3,
"status": "NT_STATUS_WRONG_PASSWORD",
"localAddress": "ipv6:XXXX:XXXX:XXXX:XXXX:7c8b:d5ff:fedb:7077:445",
"remoteAddress": "ipv6:XXXX:XXXX:XXXX:XXXX:6d3a:f06c:bcbf:7cb7:51361",
"serviceDescription": "SMB2",
"authDescription": null,
"clientDomain": "TRUENAS",
"clientAccount": "test",
"workstation": "MERCURY",
"becameAccount": null,
"becameDomain": null,
"becameSid": null,
"mappedAccount": "test",
"mappedDomain": "TRUENAS",
"netlogonComputer": null,
"netlogonTrustAccount": null,
"netlogonNegotiateFlags": "0x00000000",
"netlogonSecureChannelType": 0,
"netlogonTrustAccountSid": null,
"passwordType": "NTLMv2",
"duration": 3373
},
"timestamp_tval": {
"tv_sec": 1698602447,
"tv_usec": 934725
}
},


------------------------------------------------------------------------------------------------------


"sudo midclt call smb.passdb_list" shows the correct list of user accounts including lisbeth and test with the correct UID numbers.


------------------------------------------------------------------------------------------------------


"sudo smbclient //127.0.0.1/MEDIA -U test" shows an "smb: \>" prompt but in /var/log/samba4/log.smbd I can see the failed authentication attempts like this one:


Auth: [SMB2,(null)] user [WORKGROUP]\[test] at [Sun, 29 Oct 2023 22:40:16.538365 CET] with [NTLMv2] status [NT_STATUS_WRONG_PASSWORD] workstation [SATURN] remote host [ipv4:127.0.0.1:59058] mapped to [WORKGROUP]\[test]. local host [ipv4:127.0.0.1:445]


and the successful attempts after a "sudo midclt call smb.synchronize_passdb" like this one:
[2023/10/29 23:13:28.304279, 2] ../../source3/auth/auth.c:324(auth_check_ntlm_password)
check_ntlm_password: authentication for user [test] -> [test] -> [test] succeeded

 

[anodos]

NT hashes are basically password-equivalent strings and should not be posted unredacted on internet.

 

[anodos]

What is output of testparm -s?

 

[MkIVMark]

NT hashes are basically password-equivalent strings and should not be posted unredacted on internet.

Good info and pointer for other people but I already know this and it isn't a problem. Anyone remember l0phtcrack? If anyone wants to try and get the password from this hash they are most welcome, it will not be a massive surprise

 

[MkIVMark]

Output of testparm -s

Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

Server role: ROLE_STANDALONE

# Global parameters
[global]
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
load printers = No
logging = file
max log size = 5120
netbios name = TRUENAS
ntlm auth = ntlmv1-permitted
passdb backend = tdbsam:/var/run/samba-cache/passdb.tdb
printcap name = /dev/null
registry shares = Yes
restrict anonymous = 2
server min protocol = NT1
server multi channel support = No
server string = TrueNAS Server
winbind request timeout = 2
idmap config * : range = 90000001 - 100000000
fruit:zero_file_id = false
fruit:nfs_aces = false
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
create mask = 0775
directory mask = 0775


[media-kids]
ea support = No
path = /mnt/zpool1/media-kids
posix locking = No
read only = No
smbd max xattr size = 2097152
vfs objects = fruit streams_xattr shadow_copy_zfs ixnas zfs_core io_uring
tn:vuid =
fruit:time machine max size = 0
fruit:time machine = False
fruit:resource = stream
fruit:metadata = stream
nfs4:chown = True
tn:home = False
tn:path_suffix =
tn:purpose = DEFAULT_SHARE


[test]
ea support = No
path = /mnt/zpool1/test
posix locking = No
read only = No
smbd max xattr size = 2097152
vfs objects = fruit streams_xattr shadow_copy_zfs ixnas zfs_core io_uring
tn:vuid =
fruit:time machine max size = 0
fruit:time machine = False
fruit:resource = stream
fruit:metadata = stream
nfs4:chown = True
tn:home = False
tn:path_suffix =
tn:purpose = DEFAULT_SHARE


[shared]
ea support = No
path = /mnt/zpool1/shared
posix locking = No
read only = No
smbd max xattr size = 2097152
vfs objects = fruit streams_xattr shadow_copy_zfs acl_xattr zfs_core io_uring
tn:vuid =
fruit:time machine max size = 0
fruit:time machine = False
fruit:resource = stream
fruit:metadata = stream
tn:home = False
tn:path_suffix =
tn:purpose = DEFAULT_SHARE


[media]
ea support = No
path = /mnt/zpool1/media
posix locking = No
read only = No
smbd max xattr size = 2097152
vfs objects = fruit streams_xattr shadow_copy_zfs ixnas zfs_core io_uring
tn:vuid =
fruit:time machine max size = 0
fruit:time machine = False
fruit:resource = stream
fruit:metadata = stream
nfs4:chown = True
tn:home = False
tn:path_suffix =
tn:purpose = DEFAULT_SHARE


[virtual-machines]
ea support = No
path = /mnt/zpool1/virtual-machines
posix locking = No
read only = No
smbd max xattr size = 2097152
vfs objects = fruit streams_xattr shadow_copy_zfs ixnas zfs_core io_uring
tn:vuid =
fruit:time machine max size = 0
fruit:time machine = False
fruit:resource = stream
fruit:metadata = stream
nfs4:chown = True
tn:home = False
tn:path_suffix =
tn:purpose = DEFAULT_SHARE

 

[MkIVMark]

Update: Seems to be working again. I have connected to two of the SMB shares with a new account successfully (that previously failed NTLM Auth).

I decided to have a second go at the MacOS Keychain manager to remove any stored UIDs/PWs - I searched for the TrueNAS server name, IPv4/6 addresses and account names used for SMB and deleted everything (as in the first go). My suspicion was that MacOS was sending the correct UID but not sending the password I typed. I was just about to install wireshark to see if that could provide any illumination, but when I rebooted after the second go at the Keychain and tried to connect using a new account to a TrueNAS SMB share, it worked.

In MacOS 14.1 Finder I have been observing odd behavior when using multiple accounts to access SMB shares. Connecting (and disconnecting) multiple SMB shares using multiple accounts has resulted in mounts not showing in "Locations", even though the mounts were in a Finder tab and fully navigable. Connections to SMB shares defaulted to previously used accounts, even when specifying a different account to be used (smb://account-name:*@server/share). This even happens when those previously used accounts have been removed from the Keychain manager, so perhaps cached in memory? But as this behavior survived a reboot, it might be that I was not as thorough with my Keychain cleaning as needed and the second attempt was better? But, actually I didn't find any additional accounts on the second Keychain cleaning compared to the first. Maybe MacOS is caching UID/PW info for SMB connections somewhere other than the Keychain?

I suspect that the local smbclient auth failing and then succeeding after a "sudo midclt call smb.synchronize_passdb" is not related to the issue with the MacOS connections, but instead resulted from switching from POSIX ACLs to NFSv4 ACLs. This means I actually had (have?) two different issues with mixed-up symptoms, making it hard to see cause and effect during fault finding.

I will continue to check SMB connections and see what is and isn't working, but I am going to stop using multiple accounts for SMB connections on MacOS.

 

[anodos]

Update: Seems to be working again. I have connected to two of the SMB shares with a new account successfully (that previously failed NTLM Auth).

I decided to have a second go at the MacOS Keychain manager to remove any stored UIDs/PWs - I searched for the TrueNAS server name, IPv4/6 addresses and account names used for SMB and deleted everything (as in the first go). My suspicion was that MacOS was sending the correct UID but not sending the password I typed. I was just about to install wireshark to see if that could provide any illumination, but when I rebooted after the second go at the Keychain and tried to connect using a new account to a TrueNAS SMB share, it worked.

In MacOS 14.1 Finder I have been observing odd behavior when using multiple accounts to access SMB shares. Connecting (and disconnecting) multiple SMB shares using multiple accounts has resulted in mounts not showing in "Locations", even though the mounts were in a Finder tab and fully navigable. Connections to SMB shares defaulted to previously used accounts, even when specifying a different account to be used (smb://account-name:*@server/share). This even happens when those previously used accounts have been removed from the Keychain manager, so perhaps cached in memory? But as this behavior survived a reboot, it might be that I was not as thorough with my Keychain cleaning as needed and the second attempt was better? But, actually I didn't find any additional accounts on the second Keychain cleaning compared to the first. Maybe MacOS is caching UID/PW info for SMB connections somewhere other than the Keychain?

I suspect that the local smbclient auth failing and then succeeding after a "sudo midclt call smb.synchronize_passdb" is not related to the issue with the MacOS connections, but instead resulted from switching from POSIX ACLs to NFSv4 ACLs. This means I actually had (have?) two different issues with mixed-up symptoms, making it hard to see cause and effect during fault finding.

I will continue to check SMB connections and see what is and isn't working, but I am going to stop using multiple accounts for SMB connections on MacOS.

Most of that sounds like MacOS issues. Possibly cached credentials. There are only two cases I've seen where auth fails with NT_STATUS_WRONG_PASSWORD:
1. wrong password used
2. NTLMv1 auth attempt when v1 is disabled.

Changing acltype is expected to potentially impact access shares. This is because any POSIX ACL information stored in the xattr will no longer be used when doing permissions checks in kernel / ZFS.