Upgraded from 11.2-U8 to 11.3-U1, now system crashes regulary

[rdvanbuuren]

So I've upgraded from 11.2-U8 to 11.3-U1 last weekend, and since then I'm having regular system crashes / freezes.
The system will run normal for about 8-12 hours and then completely freeze again. There's also no display output anymore.
Only a hard reboot can restart the system.

I never had these freezes on 11.2-U8, running on new hardware I purchased a couple of months ago.

System specifications:

CPU: AMD Ryzen 3 1300x
Mainboard: ASRock AB350 Gaming-ITX/AC
Memory: 16GB Corsair Vengeance LPX (CMK16GX4M2A2666C16)
ZFS Pool 1: 4x Seagate SkyHawk Surveillance, 4TB HDD
ZFS Pool 2: ADATA XPG SX8200 Pro 256 GB M.2 SSD

What I've tried so far:

• Disabled S.M.A.R.T service, described as a solution to crashes in another post
• Saved configs and completely re-installed the system and imported configs.

I've already updated the ZFS pools when I was on 11.3. So I'm not sure if I can even go back to the previous 11.2-U8 version, without having to create new pools.

Would anyone know what to do from here on?
I'd be happy to provide any data needed, as long as I know how to obtain it :)

Best regards,
Rick

 

[ornias]

You run ryzen, this is a known ryzen flaw thats already reported and not fixed yet:

https://jira.ixsystems.com/projects/NAS/issues/NAS-104597

There is a workaround that MIGHT work in some cases, but it's a ryzen related problem:
you could try setting rc.conf variable microcode_update_enable="NO" via WebUI and power cycling the system

I just trashed my ryzen install and moved to intel, due to it being the actual platform FreeNAS and Truenas run on, so more chance its stable.

 

[blanchet]

You should stay with FreeNAS 11.2u8 until the release of TrueNAS Core 12, which will support AMD Ryzen.

FreeNAS and TrueNAS are Unifying

Today, we announce the next evolution for FreeNAS and TrueNAS: with the 12.0 release, we are not only bringing more features and improvements than any release that has come before it, we are also unifying both products into a single software image and brand. We present to you: TrueNAS Open Storage.

www.ixsystems.com

 

[ornias]

You should stay with FreeNAS 11.2u8 until the release of TrueNAS Core 12, which will support AMD Ryzen.

FreeNAS and TrueNAS are Unifying

Today, we announce the next evolution for FreeNAS and TrueNAS: with the 12.0 release, we are not only bringing more features and improvements than any release that has come before it, we are also unifying both products into a single software image and brand. We present to you: TrueNAS Open Storage.

www.ixsystems.com

FreeNAS and FreeBSD already support Ryzen in 11.3, 12 just includes enhanced support (which is mostly centered around NUMA and multithreading afaik)

 

[rdvanbuuren]

You run ryzen, this is a known ryzen flaw thats already reported and not fixed yet:

https://jira.ixsystems.com/projects/NAS/issues/NAS-104597

There is a workaround that MIGHT work in some cases, but it's a ryzen related problem:
you could try setting rc.conf variable microcode_update_enable="NO" via WebUI and power cycling the system

I just trashed my ryzen install and moved to intel, due to it being the actual platform FreeNAS and Truenas run on, so more chance its stable.

Thanks for the info! I just added the setting. Now just fingers crossed and hopefully this fixes the issue on my system.
Else I'll be going back to 11.2-U8 :)

 

[ornias]

Thanks for the info! I just added the setting. Now just fingers crossed and hopefully this fixes the issue on my system.
Else I'll be going back to 11.2-U8 :)

Be aware that downgrading to 11.2U8 might brake your jails (I tried 3 times, all times broke my jails due to 11.3 also containing a huge iocage version bump)

 

[rdvanbuuren]

Thanks for the info! I just added the setting. Now just fingers crossed and hopefully this fixes the issue on my system.
Else I'll be going back to 11.2-U8 :)

Well, that didn't work :( System already crashed again

 

[Luc De Bolle]

Hi everyone,
I am experiencing the same issue, but it is not on a Ryzen processor but on Intel...
Version: FreeNAS-11.3-U2
Physical Memory: 7985 MiB
Model: Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz

Freezes once a day with no indications in the log.
Update: 13/04/201 - 18:05: System hangs (no access possible via web or ssh) but still responds to ping....

I upgraded quite recently to this version. Before the upgrade, the system ran for months without interruption and was only restarted after updates - no crashes.

I am considering downgrading, but before doing that I would like to have your opinions and perhaps there is already some unposted workaround.
Please let me know if I should better report this as a bug - I couldn't find a similar issue in the list except for the Ryzen one...

I look forward to your comments and suggestions,

best regards,

Luc

 

[Luc De Bolle]

Hi,

Just figured out that it could be the boot device giving me this trouble:
Similar errors on the screen I now attached...
(da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
(da1:umass-sim1:1:0:0): Retrying command

I'll try to solve this one....

Best regards,

Luc

 

[Yorick]

If that's a USB stick you are booting from, it is likely dying.

 

[Luc De Bolle]

Thank You Yorick,
I appreciate my post was read and reacted upon

I was afraid you were gong to say this...
I am currently trying to clone it before it goes completely dead... I hope it works.

I'll post back unless it is better not to trash this thread with this...

 

[Luc De Bolle]

Hi Yorick,

Just a quick confirmation that the clone didn't work and after reading post in the forum it is 'not done'.
I made a copy of the configuration prior to upgrading so I had a recent config and after installing freenas on a new boot device an uploading the config I am up and running again.
Best regards,
Luc