APEI Corrected Memory Error - Are my RAM sticks bad?

[sdealmeida]

So long story short, I've been notificing that when watching anything on Plex, the audio would glitch randomly for like half a second. After looking at my Truenas server, I saw the following errors:

Code:

Nov  6 06:56:10 freenas APEI Corrected Memory Error:
Nov  6 06:56:10 freenas Node: 1
Nov  6 06:56:10 freenas Device: 0
Nov  6 06:56:10 freenas Memory Error Type: 2
Nov  6 06:56:10 freenas Flags: 0x1
Nov  6 06:56:10 freenas FRU Text: CorrectedErr
Nov  6 07:04:21 freenas APEI Corrected Memory Error:
Nov  6 07:04:21 freenas Node: 1
Nov  6 07:04:21 freenas Device: 0
Nov  6 07:04:21 freenas Memory Error Type: 2
Nov  6 07:04:21 freenas Flags: 0x1
Nov  6 07:04:21 freenas FRU Text: CorrectedErr
Nov  6 07:05:27 freenas APEI Corrected Memory Error:
Nov  6 07:05:27 freenas Node: 1
Nov  6 07:05:27 freenas Device: 0
Nov  6 07:05:27 freenas Memory Error Type: 2
Nov  6 07:05:27 freenas Flags: 0x1
Nov  6 07:05:27 freenas FRU Text: CorrectedErr
Nov  6 07:07:05 freenas APEI Corrected Memory Error:
Nov  6 07:07:05 freenas Node: 1
Nov  6 07:07:05 freenas Device: 0
Nov  6 07:07:05 freenas Memory Error Type: 2
Nov  6 07:07:05 freenas Flags: 0x1
Nov  6 07:07:05 freenas FRU Text: CorrectedErr
Nov  6 07:07:38 freenas APEI Corrected Memory Error:
Nov  6 07:07:38 freenas Node: 1
Nov  6 07:07:38 freenas Device: 0
Nov  6 07:07:38 freenas Memory Error Type: 2
Nov  6 07:07:38 freenas Flags: 0x1
Nov  6 07:07:38 freenas FRU Text: CorrectedErr
Nov  6 07:13:06 freenas APEI Corrected Memory Error:
Nov  6 07:13:06 freenas Node: 1
Nov  6 07:13:06 freenas Device: 0
Nov  6 07:13:06 freenas Memory Error Type: 2
Nov  6 07:13:06 freenas Flags: 0x1
Nov  6 07:13:06 freenas FRU Text: CorrectedErr
Nov  6 07:15:49 freenas APEI Corrected Memory Error:
Nov  6 07:15:49 freenas Node: 1
Nov  6 07:15:49 freenas Device: 0
Nov  6 07:15:49 freenas Memory Error Type: 2
Nov  6 07:15:49 freenas Flags: 0x1
Nov  6 07:15:49 freenas FRU Text: CorrectedErr
Nov  6 07:16:22 freenas APEI Corrected Memory Error:
Nov  6 07:16:22 freenas Node: 1
Nov  6 07:16:22 freenas Device: 0
Nov  6 07:16:22 freenas Memory Error Type: 2
Nov  6 07:16:22 freenas Flags: 0x1
Nov  6 07:16:22 freenas FRU Text: CorrectedErr
Nov  6 07:31:07 freenas APEI Corrected Memory Error:
Nov  6 07:31:07 freenas Node: 1
Nov  6 07:31:07 freenas Device: 0
Nov  6 07:31:07 freenas Memory Error Type: 2
Nov  6 07:31:07 freenas Flags: 0x1
Nov  6 07:31:07 freenas FRU Text: CorrectedErr
Nov  6 07:32:12 freenas APEI Corrected Memory Error:
Nov  6 07:32:12 freenas Node: 1
Nov  6 07:32:12 freenas Device: 0
Nov  6 07:32:12 freenas Memory Error Type: 2
Nov  6 07:32:12 freenas Flags: 0x1
Nov  6 07:32:12 freenas FRU Text: CorrectedErr
Nov  6 07:33:51 freenas APEI Corrected Memory Error:
Nov  6 07:33:51 freenas Node: 1
Nov  6 07:33:51 freenas Device: 0
Nov  6 07:33:51 freenas Memory Error Type: 2
Nov  6 07:33:51 freenas Flags: 0x1
Nov  6 07:33:51 freenas FRU Text: CorrectedErr
Nov  6 07:34:56 freenas APEI Corrected Memory Error:
Nov  6 07:34:56 freenas Node: 1
Nov  6 07:34:56 freenas Device: 0
Nov  6 07:34:56 freenas Memory Error Type: 2
Nov  6 07:34:56 freenas Flags: 0x1
Nov  6 07:34:56 freenas FRU Text: CorrectedErr
Nov  6 07:41:30 freenas APEI Corrected Memory Error:
Nov  6 07:41:30 freenas Node: 1
Nov  6 07:41:30 freenas Device: 0
Nov  6 07:41:30 freenas Memory Error Type: 2
Nov  6 07:41:30 freenas Flags: 0x1
Nov  6 07:41:30 freenas FRU Text: CorrectedErr
Nov  6 07:42:02 freenas APEI Corrected Memory Error:
Nov  6 07:42:02 freenas Node: 1
Nov  6 07:42:02 freenas Device: 0
Nov  6 07:42:02 freenas Memory Error Type: 2
Nov  6 07:42:02 freenas Flags: 0x1
Nov  6 07:42:02 freenas FRU Text: CorrectedErr
Nov  6 07:46:24 freenas APEI Corrected Memory Error:
Nov  6 07:46:24 freenas Node: 1
Nov  6 07:46:24 freenas Device: 0
Nov  6 07:46:24 freenas Memory Error Type: 2
Nov  6 07:46:24 freenas Flags: 0x1
Nov  6 07:46:24 freenas FRU Text: CorrectedErr
Nov  6 09:00:08 freenas APEI Corrected Memory Error:
Nov  6 09:00:08 freenas Node: 1
Nov  6 09:00:08 freenas Device: 0
Nov  6 09:00:08 freenas Memory Error Type: 2
Nov  6 09:00:08 freenas Flags: 0x1
Nov  6 09:00:08 freenas FRU Text: CorrectedErr
Nov  6 09:11:04 freenas APEI Corrected Memory Error:
Nov  6 09:11:04 freenas Node: 1
Nov  6 09:11:04 freenas Device: 0
Nov  6 09:11:04 freenas Memory Error Type: 2
Nov  6 09:11:04 freenas Flags: 0x1
Nov  6 09:11:04 freenas FRU Text: CorrectedErr
Nov  6 09:18:43 freenas APEI Corrected Memory Error:
Nov  6 09:18:43 freenas Node: 1
Nov  6 09:18:43 freenas Device: 0
Nov  6 09:18:43 freenas Memory Error Type: 2
Nov  6 09:18:43 freenas Flags: 0x1
Nov  6 09:18:43 freenas FRU Text: CorrectedErr
Nov  6 09:19:15 freenas APEI Corrected Memory Error:
Nov  6 09:19:15 freenas Node: 1
Nov  6 09:19:15 freenas Device: 0
Nov  6 09:19:15 freenas Memory Error Type: 2
Nov  6 09:19:15 freenas Flags: 0x1
Nov  6 09:19:15 freenas FRU Text: CorrectedErr
Nov  6 09:32:55 freenas APEI Corrected Memory Error:
Nov  6 09:32:55 freenas Node: 1
Nov  6 09:32:55 freenas Device: 0
Nov  6 09:32:55 freenas Memory Error Type: 2
Nov  6 09:32:55 freenas Flags: 0x1
Nov  6 09:32:55 freenas FRU Text: CorrectedErr

Don't worry about the "freenas" tag. This was originally a Freenas instance and I've upgraded it to TrueNAS Core.

Here are my server specs:

OS: TrueNAS-12.0-U8.1
Motherboard: Asus P11C-I
RAM: 2x 32gb Samsung sticks (Samsung 32GB DDR4-2933 ECC UDIMM PC4-23466U-E Dual Rank x8 Module, M391A4G43AB1-CVF)
CPU: Intel Xeon E-2146G @ 3.50 GHz

What could this mean? I'm assuming that my 2nd RAM stick has gone bad?
I've installed these 2 RAM sticks about a year and a half ago, and this is the first time I'm seeing this error message.

Thanks

 

[jgreco]

I'm assuming that my 2nd RAM stick has gone bad?

You can assume what you like, but there's no particular sanity or standardization to these sorts of subsystems especially from vendors like Asus who do not specialize in servers. You can ask Asus support for an interpretation if you like, but it may be easier just to kick the system back to burn-in phase and try each stick of memory one at a time.

Building, Burn-In, and Testing your FreeNAS system

I've been meaning to post some guidance here for a while now. We frequently see people come to the forums with hardware problems that should have washed out in the system build process, but since many of the users here are DIY'ers without...

www.truenas.com

 

[cengjingcaihua]

I had the same error while installing TrueNAS 13.0-U2 on a bad ssd. Tried a few times the error message pops up every time exactly at the time I enter the password for root. Switched to a brand new Samsung 970 Evo and no issues ever since. I had done 12 hours of memtest on each stick and no errors.

 

[sdealmeida]

@jgreco Might've been a blip in the system. For the last 24 hours, I haven't gotten any RAM errors. After snooping a bit more in the logs, it seems like these RAM issues happened during a Pool Scrub. Are pool scrubs that RAM intensive?


@cengjingcaihua thanks for the tip. I monitored all my disks and they seemed ok. I also did a short + long SMART test and there's nothing out of the ordinary. I did a quick memtest and it also came back clean. I wish I had another stick so that I could use it while I test my RAM sticks are bit intensively.

 

[jgreco]

Are pool scrubs that RAM intensive?

Well, sorta, but on the other hand, that shouldn't matter. No amount of memory access should cause RAM errors.