SOLVED Unable to login after upgrading Truenas (2FA is invalid)

[sdealmeida]

I recently upgraded my server from TrueNAS-12.0-U6.1 to TrueNAS-12.0-U7 and after that, my 2FA stopped working correctly. Everytime I try to login, it states that my username/password or 2FA is incorrect. If I keep re-trying, I'm eventualy able to login (which tells me my username/password is correct). I use a password manager to store my username/password and 2FA code (I use 1Password).
The clock on my server is up to date and I've even tried waiting the 30 seconds for a new 2FA code to be generated, but I'm still unable to login at times. I've even reset my root password and generated a new 2FA code, but this issue still persists.

I haven't seen any other user report about this issue, so I don't know if its just me. I have a feeling this is a bug in the software since I've been using Truenas for many years now and this is the first time I'm running into this problem.

What should I do? Thanks!

 

[Heracles]

I haven't seen any other user report about this issue, so I don't know if its just me

There have been many posts about 2FA...

The clock on my server is up to date

And how did you identified that ? To eyeball the clock is not enough. Are you sure that your NTP config is working and that your server is in sync ?

Also, it is just as important for the client to be perfectly sync. The use of a smartphone often with that because they are naturally sync by the cellular network itself. If you are using a desktop as a client, be sure to get NTP working on that one too.

 

[Samuel Tai]

Are you sure your clock is up to date? What does ntpq -p show? Also, does the clock drift? What does sysctl kern.timecounter.choice show?

 

[sdealmeida]

Are you sure your clock is up to date? What does ntpq -p show? Also, does the clock drift? What does sysctl kern.timecounter.choice show?

This is what I see.

ntpq -p.

Code:

root@freenas[~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp.zaf.ca      .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
 speedtest.switc .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
 ntp-cov-1.lewis .INIT.          16 u    - 1024    0    0.000   +0.000   0.000

sysctl kern.timecounter.choice

Code:

root@freenas[~]# sysctl kern.timecounter.choice
kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC-low(1000) dummy(-1000000)

PS: I'm using Freebsd's default NTP servers (x.freebsd.pool.ntp.org, where x is 0 to 2).

 

[Heracles]

Stratum 16 and all zeroes mean that it does not sync.

Also, how about the sync client side ?

 

[Samuel Tai]

You're NOT sync'ed to NTP. Is your ISP blocking NTP? Try setting a sysctl tunable kern.timecounter.hardware="HPET" (in case your TSC drifts), and then running ntpdate 0.freebsd.pool.ntp.org.

A sync'ed system will have output like this:

Code:

root@raven:~ # ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-a08.alphasrv.ne 124.216.164.14   2 u  291 1024  377  117.519   -2.190   0.612
*ntp2.your.org   .GPS.            1 u  189 1024  377   32.188   +1.334   0.456
+251.228.185.35. 169.254.169.254  3 u  258 1024  377   79.331   -2.848   0.606
+ntp2.as200552.n 202.70.69.81     2 u  302 1024  377   92.726   -0.678   0.490

 

[sdealmeida]

Also, how about the sync client side ?

I'm getting the OTP code from my phone, so I think its synced

You're NOT sync'ed to NTP. Is your ISP blocking NTP? Try setting a sysctl tunable kern.timecounter.hardware="HPET" (in case your TSC drifts), and then running ntpdate 0.freebsd.pool.ntp.org.

A sync'ed system will have output like this:

Code:

root@raven:~ # ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-a08.alphasrv.ne 124.216.164.14   2 u  291 1024  377  117.519   -2.190   0.612
*ntp2.your.org   .GPS.            1 u  189 1024  377   32.188   +1.334   0.456
+251.228.185.35. 169.254.169.254  3 u  258 1024  377   79.331   -2.848   0.606
+ntp2.as200552.n 202.70.69.81     2 u  302 1024  377   92.726   -0.678   0.490

Odd, nothing about my setup changed recently (except updating to 12.0-U7).
I've added those tunnables to my config but I see the exact same results.

Code:

root@freenas[~]# ntpdate 0.freebsd.pool.ntp.org
13 Dec 23:13:48 ntpdate[84327]: the NTP socket is in use, exiting

Also, I saw a tunnable thats new to me (I haven't seen this before "hint.isp.0.role").

 

[Samuel Tai]

The hint.isp.x.role tunables are for the Enterprise edition's Fibre Channel support.

Try service ntpd stop, then ntpdate 0.freebsd.pool.ntp.org, and finally service ntpd start.

 

[sdealmeida]

Oh, I did the following and its giving me a different output now

Code:

root@freenas[~]# service ntpd stop
Stopping ntpd.
Waiting for PIDS: 1365.

root@freenas[~]# ntpdate 0.freebsd.pool.ntp.org
13 Dec 23:29:56 ntpdate[94236]: step time server 192.241.146.233 offset +24.433896 sec

root@freenas[~]# service ntpd start
Starting ntpd.

root@freenas[~]# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 5.9.124.124     .INIT.          16 u    -   64    0    0.000   +0.000   0.000
*194.0.5.123     209.51.161.238   2 u   11   64    1   14.234   -4.031   0.154
 54.39.23.64     .INIT.          16 u    -   64    0    0.000   +0.000   0.000

Odd, not really sure why this is happening though.

PS: This now shows in my web console

Code:

Dec 13 23:29:20 freenas 1 2021-12-13T23:29:20.119999-05:00 freenas.local ntpd 1365 - - ntpd exiting on signal 15 (Terminated)
Dec 13 23:29:56 freenas 1 2021-12-13T23:29:56.827256-05:00 freenas.local upsd 1387 - - Data for UPS [ups] is stale - check driver
Dec 13 23:29:56 freenas 1 2021-12-13T23:29:56.827908-05:00 freenas.local upsd 1387 - - UPS [ups] data is no longer stale
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004502-05:00 freenas.local ntpd 94259 - - ntpd 4.2.8p15-a (1): Starting
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004646-05:00 freenas.local ntpd 94259 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004701-05:00 freenas.local ntpd 94259 - - ----------------------------------------------------
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004723-05:00 freenas.local ntpd 94259 - - ntp-4 is maintained by Network Time Foundation,
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004746-05:00 freenas.local ntpd 94259 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004780-05:00 freenas.local ntpd 94259 - - corporation.  Support and training for ntp-4 are
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004814-05:00 freenas.local ntpd 94259 - - available at https://www.nwtime.org/support
Dec 13 23:30:03 freenas 1 2021-12-13T23:30:03.004893-05:00 freenas.local ntpd 94259 - - ----------------------------------------------------

 

[Samuel Tai]

OK, you're starting to get sync'ed. One NTP server is locked in. You were 24 seconds behind NTP. Give it about 30 minutes, and 3-4 NTP servers will be locked in. Your 2FA should be more reliable now, but you should redo 2FA just to be safe.

 

[sdealmeida]

OK, you're starting to get sync'ed. One NTP server is locked in. You were 24 seconds behind NTP. Give it about 30 minutes, and 3-4 NTP servers will be locked in. Your 2FA should be more reliable now, but you should redo 2FA just to be safe.

Looks like my issue has been resolved. I ended up changing the NTP servers to other NTP servers closer to me (using ntp.org servers).

Thanks!