SOLVED Frustrated with AMD Ryzen Stability on 11.2-U5

[Dopamin3]

edit- SOLVED: upgraded to 2700X and freezing went away. See my last post for more details.

I've searched a lot on this topic but wanted to make my own thread for my specifics. Every 1 - 5 days my system just locks up with whatever output on the screen (usually the FreeNAS options and some messages about interfaces going up or down depending on what I'm doing). It doesn't respond to any keyboard input, can't ping or reach anything on it, and a hard shutdown and power on gets it running again. My system specs are:

• SuperMicro SuperChassis 933T-R760B- a 3U rack mount server case with 15 hot swappable SATA and triple redundant 760W power supply. Plenty of airflow.
• Asrock X370 Taichi- latest BIOS 5.80 but same issue on all previous
• AMD Ryzen 1700- with 2700X stock cooler aka Wraith Prism. Typically idles around 30C and doesn't go above 60C under heavy load.
• 4 x 16GB Crucial CT16G4WFD8266 ECC RAM
• Visiontek 5450 GPU
• 10x Toshiba X300 5TB Hard Drives in RAIDZ3- no encryption, all 10 running off motherboard (eight from chipset , two from Asmedia controller)
• Samsung PM961 128GB NVME SSD- boot drive (formerly on mirrored flash drives with same behavior)
• FreeNAS 11.2-U5

All the drives are only a few months old. Long test/short test with smartctl show no errors and no parameters out of whack with smartctl -a. I've tested for stability with 200% HCI memtest coverage, and 12 hours Prime95 small FFT/12 hours Prime95 blend test temporarily booting with Windows 10 x64 with no issues. I've updated to every BIOS revision available including the latest 5.80 which has the newest AGESA 1.0.0.3ABB. I've messed with the idle power supply voltage value in the BIOS (C6 state I believe this controls), disabled Cool n Quiet, upped the SOC voltage (basically memory controller voltage since I'm running 4 x 16GB sticks of 2666MHz thought it might help), and disabled C States completely. No matter what I have done I can't get uptime of more than a few days. This is a lot newer batch of CPU (purchased in early 2019) so I'm fairly confident the segfault issue is not on my chip either (but I think that affected Linux moreso than FreeBSD). To rule out the Asmedia controller I even disconnected the two drives from it but it would still freeze like this.

Mostly it freezes at idle or very low load, sometimes it freezes when doing SMB transfers of larger files (5GB+). I can't find any relevant information in logs when this happens. When I search Ryzen in Redmine there are a bunch of done or closed bugs. I know a lot of people had issues on earlier FreeNAS with Ryzen, but how is 11.2-U5 working out for anyone with Ryzen? I think I've done enough testing to rule out my hardware, and it's just another Ryzen specific issue but I'm open to hear other opinions on what additional troubleshooting can be done. Is there any hope for 11.3 or future versions to fix me or is it just a lost cause trying to get this setup stable?

 

[Dopamin3]

Bump for any input

 

[Fogelholk]

This is not the reply you're looking for @Dopamin3, but just another "Me too".
I got the same Motherboard as you:
* Asrock X370 Taichi on BIOS-version 4.80
* AMD Ryzen 1600
* SuperMicro SuperChassis 846E26-R1200B
Instead of using the internal SATA-ports I run my disks via a LSI/Avago/Broadcom/(whatever it is today) SAS9211-4i
* FreeNAS 11.2-U5
* 4 x Samsung 16GB Module DDR4 2400MHz ECC U-DIMM
* SAMSUNG SSD 830 as boot drive
* Got two Zpools
** 1 x Mirror with 2 x 120GB Intel SSDs
** 1 x Mirror ("raid10") with 2 x 10TB WD Red and 2 x 3TB Seagate NAS

My FreeNAS randomly freezes as well, but it seems to work without trouble for more than a month.
Plugging in an external screen and keyboard yields nothing, the computer just does not respond to any input, either via ethernet (ping and such) or inputs with the keyboard.
The screen does not say anything before the freeze happened and the logs are clean as a whistle at the time of the freeze, since the system can't write to the logs if it's frozen :)

I did however notice a text on Asrocks website for BIOS-updates that if you run any of "Pinnacle, Raven or Summit Ridge CPU" (1600 and 1700 is Summit Ridge) you should probably avoid BIOS-version 5.80. But since I also have the freezing-problem with version 4.80 and you had the problem on your previous versions this probably isn't what is causing these very annoying freezes.

Hope someone has some ideas or things to try out which can resolve this!

 

[Dopamin3]

This is not the reply you're looking for @Dopamin3, but just another "Me too".
I got the same Motherboard as you:
* Asrock X370 Taichi on BIOS-version 4.80
* AMD Ryzen 1600
* SuperMicro SuperChassis 846E26-R1200B
Instead of using the internal SATA-ports I run my disks via a LSI/Avago/Broadcom/(whatever it is today) SAS9211-4i
* FreeNAS 11.2-U5
* 4 x Samsung 16GB Module DDR4 2400MHz ECC U-DIMM
* SAMSUNG SSD 830 as boot drive
* Got two Zpools
** 1 x Mirror with 2 x 120GB Intel SSDs
** 1 x Mirror ("raid10") with 2 x 10TB WD Red and 2 x 3TB Seagate NAS

My FreeNAS randomly freezes as well, but it seems to work without trouble for more than a month.
Plugging in an external screen and keyboard yields nothing, the computer just does not respond to any input, either via ethernet (ping and such) or inputs with the keyboard.
The screen does not say anything before the freeze happened and the logs are clean as a whistle at the time of the freeze, since the system can't write to the logs if it's frozen :)

I did however notice a text on Asrocks website for BIOS-updates that if you run any of "Pinnacle, Raven or Summit Ridge CPU" (1600 and 1700 is Summit Ridge) you should probably avoid BIOS-version 5.80. But since I also have the freezing-problem with version 4.80 and you had the problem on your previous versions this probably isn't what is causing these very annoying freezes.

Hope someone has some ideas or things to try out which can resolve this!

Thanks for the confirmation on this. Now, I wouldn't recommend you upgrade to the new BIOS as there is no benefit for Zen/Zen+ users (only Zen 2 is benefited by new AGESA) and it was still freezing on me. BIOS 5.80 works fine but has two conditions:
1)Removes support for Athlon x4 970/950/940, A12-9800/9800E, A10-9700/9700E, A8-9600, A6-9550/9500/9500E (your 1600 would still work)
2)Changed some items (Cool N Quiet is now called PSP Support for example) and removed some items (only thing I personally noticed was Advanced\AMD CBS\FCH Common Options\SATA Configuration Options which had some options for the Asmedia controller like "Aggressive SATA Device Sleep".)

I'm not sure if it will fix it, but my desktop had a 2700X with Asus Crosshair VII Hero. I swapped the CPUs, so now my FreeNAS has the 2700X and my desktop has the 1700. It's too early to tell anything, but I'm on 1 day 18 hours of uptime so far. I'll update again if it crashes or if I can get more uptime than a week or so. My suggestions to you to try to get it stable and you can report back if you haven't already tried it:

• Advanced\CPU Configuration
  • Cool'n'Quiet -> Disabled
• Advanced\AMD CBS\Zen Common Options
  • Power Supply Idle Control -> Typical Current Idle

 

[Fogelholk]

Thanks for the confirmation on this. Now, I wouldn't recommend you upgrade to the new BIOS as there is no benefit for Zen/Zen+ users (only Zen 2 is benefited by new AGESA) and it was still freezing on me. BIOS 5.80 works fine but has two conditions:
1)Removes support for Athlon x4 970/950/940, A12-9800/9800E, A10-9700/9700E, A8-9600, A6-9550/9500/9500E (your 1600 would still work)
2)Changed some items (Cool N Quiet is now called PSP Support for example) and removed some items (only thing I personally noticed was Advanced\AMD CBS\FCH Common Options\SATA Configuration Options which had some options for the Asmedia controller like "Aggressive SATA Device Sleep".)

I'm not sure if it will fix it, but my desktop had a 2700X with Asus Crosshair VII Hero. I swapped the CPUs, so now my FreeNAS has the 2700X and my desktop has the 1700. It's too early to tell anything, but I'm on 1 day 18 hours of uptime so far. I'll update again if it crashes or if I can get more uptime than a week or so. My suggestions to you to try to get it stable and you can report back if you haven't already tried it:

• Advanced\CPU Configuration
  • Cool'n'Quiet -> Disabled
• Advanced\AMD CBS\Zen Common Options
  • Power Supply Idle Control -> Typical Current Idle

Thanks for the BIOS suggestions, I'll take a look at them next time it freezes up or when I have a maintenance-window. I think I set "Cool'n'Quiet" to Enabled before I put the machine into production, but it was a while ago so I'll have to double-check next time I reboot the NAS :)

 

[Dopamin3]

Welp, my desktop had a Ryzen 2700X and the FreeNAS server had the 1700. I swapped them, so now I'm running the Ryzen 1700 in my desktop and the 2700X is running in the server. So far, 12 days and 17 hours of uptime with no freezing or hitches. No BIOS settings were changed, everything basically on auto including Cool N Quiet, the Power Supply Idle Control etc... Only thing I changed was the boot order to make sure it only tries to boot off the NVME.

Conclusions from my experience:
Zen (Ryzen 1k series) is flawed on FreeBSD and you will probably get random freezing with no logs. Even later batches that shouldn't have segfault.
Zen+ / Zen 2 (Ryzen 2k / 3k series) should be good to go.

 

[RelativeAsh]

Just piling on. I too have this specific motherboard (X370 Taichi) with a 1700X. I am getting a lot of freezes. It has been freezing every half hour or so. I just updated to 5.10 and turned off CoolnQuiet. I just upgraded my motherboard and processor (main rig now has a 2700x) so if problems persist I will try putting the 2700x in to see if that fixes it.

 

[RelativeAsh]

Just piling on. I too have this specific motherboard (X370 Taichi) with a 1700X. I am getting a lot of freezes. It has been freezing every half hour or so. I just updated to 5.10 and turned off CoolnQuiet. I just upgraded my motherboard and processor (main rig now has a 2700x) so if problems persist I will try putting the 2700x in to see if that fixes it.

I haven't had a single crash since I updated the BIOS and disabled CoolNQuiet. I think the CoolNQuiet was probably the bigger factor. Happy about almost 2 full days without a crash!

 

[Fogelholk]

I swapped my 1600 to a 2600 for a pretty good price and hoping that it works better for me. So far I've had 21 days of uptime, but I was able to get 20-40 days before crashing with the 1600 so time will tell!
I also took the time to update the BIOS to very latest version (5.80) since I am now able to do it with the newer processor.

 

[averyfreeman]

FreeBSD gets hardware compatibly upgrades slower than Linux and Windows, and ixSystems keeps FreeNAS a version behind the latest FreeBSD release, so one shouldn't expect it to work as well on cutting edge hardware or less common systems like AMD processors.

Like all things, it just takes time.

Kinda surprised the newer Ryzen has less issues than the first gen - I wonder what issues FreeBSD/AMD/ixSystems worked out (?)

Thanks for the info, it's really helpful for those of us looking into building Ryzen systems! (Although I'll probably go the ZoL route - want to put together a b450m KVM+kubernetes cluster).

How's that ECC working out for you @Dopamin3 ?

 

[ykhodo]

I haven't had a single crash since I updated the BIOS and disabled CoolNQuiet. I think the CoolNQuiet was probably the bigger factor. Happy about almost 2 full days without a crash!

@RelativeAsh are you freezing on freenas 11.3? What bios version are you running now? I am on 5.50 with a ryzen 1700x and x370 taichi and locking up every couple of days since updating from 11.2.

 

[skicek]

edit- SOLVED: upgraded to 2700X and freezing went away. See my last post for more details.

@Dopamin3, @ykhodo, Im not sure if Im getting same problem as you guys, could you check my topic topic ? Im using asrock ab350 and ryzen 3 1200, in headless setup so cant see the screen of device, but after random time Im loosing connection to it and CPU report usage is showing like its shutdown but device is power up, on 11.3 had this issue, now Im on 11.3-u1 and waiting till crash

 

[ykhodo]

@Dopamin3, @ykhodo, Im not sure if Im getting same problem as you guys, could you check my topic topic ? Im using asrock ab350 and ryzen 3 1200, in headless setup so can't see the screen of device, but after random time Im loosing connection to it and CPU report usage is showing like its shutdown but device is power up, on 11.3 had this issue, now Im on 11.3-u1 and waiting till crash

It sounds like the issue I was having. Sorry to say, but I had to upgrade to the 2nd gen Ryzen and it solved my problem. Went from 1700x -> 2700x and I have been up without issues.

 

[skicek]

I went again thru BIOS settings and by this reddit post, I change OC to manual, and set Power state idle to typical. Let see if this will help. Im guessing it should as before that I had script that was in cron run every 1 minute and machine was working stable, but then I had to restart it and forgot to add to cron again script, as FreeNAS dont allow thru UI to schedule every minute...

 

[Dopamin3]

I went again thru BIOS settings and by this reddit post, I change OC to manual, and set Power state idle to typical. Let see if this will help. Im guessing it should as before that I had script that was in cron run every 1 minute and machine was working stable, but then I had to restart it and forgot to add to cron again script, as FreeNAS don't allow thru UI to schedule every minute...

Replace the Realtek NIC before trying any additional troubleshooting. FreeBSD and Realtek notoriously don't play nicely together.

Top Picks for FreeNAS NICs (Networking)

We have 1GbE, 10GbE (SFP+ and 10Gbase-T) and 40GbE networking options for the top FreeNAS NICs as storage performance is network dependent.

www.servethehome.com

 

[skicek]

Replace the Realtek NIC before trying any additional troubleshooting. FreeBSD and Realtek notoriously don't play nicely together.

Top Picks for FreeNAS NICs (Networking)

We have 1GbE, 10GbE (SFP+ and 10Gbase-T) and 40GbE networking options for the top FreeNAS NICs as storage performance is network dependent.

www.servethehome.com

if you read my topic I already done that many days ago... Now after changing power state idle it is stable for almost 2 days will see what next.

 

[Dopamin3]

if you read my topic I already done that many days ago... Now after changing power state idle it is stable for almost 2 days will see what next.

Still stable?

 

[skicek]

Still stable?

stable as pyramids in Egypt.

 

[jasperb182]

Hey guys.

I have a Ryzen 1300x with a A320 Mobo and 16gb RAM and have been having the same issue. I am going to try and settings you have recommend and report back incase other have the same issue.

 

[kappclark]

FWIW - running Freenas with Ryzen 2200G and as skicek says - like the Egyptian Pyramids