FreeNAS 11 crash during write

[cchr82]

I'm extremely new to FreeNAS and just set up a new NAS. I am trying to back up all of my data to my system, but after a few hours of writing to my storage pool FreeNAS appears to crash, shutdown, and all I can see is a black screen. It requires me to physically restart the computer.

I have tried looking for some answers but most of the references I found for this issue were with users using < 2 GB of RAM on FreeNAS versions << 11.

As I am new, I really have no idea how to debug this issue or solve it and any suggestions would be useful.

Based on my initial tests I do not think it is an issue with bad memory sticks or CPU overheating? The following are the tests I performed:

1) memory test -> ran memtest - no errors
2) CPU stress test -> ran prime95 test for 12 hours on all threads - no issues and computer did not crash or restart
3) reinstalled FreeNAS on new flash drives
4) I don't seem to see any errors from smart tests (only performed short ones right now)

The following summarizes my system:
1. 6x4TB HDD setup in raidz2
2. 16 GB RAM (I assumed I only need 1 GB RAM per effective memory in pool (~14TB)?)
3. boot drive is installed on two mirrored Sandisk 16GB ultra fit flash drives (again, i have tried this with multiple installs to new flash drives, same issue)
4. Data is being copied from an external HDD directly attached to the NAS. Shutdown does not seem to display any error message and I cannot predict when it will fail, usually within 1-2 hours of writing.

 

[wblock]

Please identify the rest of the hardware: motherboard, processor, power supply, exact type of RAM. How is the external drive connected to the NAS? What type of filesystem is on the external drive?

 

[Chris Moore]

Still need more information on the hardware, but

4. Data is being copied from an external HDD directly attached to the NAS. Shutdown does not seem to display any error message and I cannot predict when it will fail, usually within 1-2 hours of writing.

My experience with a similar situation a few years ago was that attaching an external USB drive to the FreeNAS system to move data onto the NAS was not a workable solution. Data needs to flow into the system via the network and the network shares that have been configured.
Also, where were you putting the data you were copying to the NAS?

 

[cchr82]

Please identify the rest of the hardware: motherboard, processor, power supply, exact type of RAM. How is the external drive connected to the NAS? What type of filesystem is on the external drive?

More hardware specs:
motherboard: asrock taichi x370
processor: ryzen 5 1600
power supply: corsair TX-M 550w gold
RAM: cant remember exact type on top of my head but its. 2x8GB crucial sticks, waiting for ECC RAM in mail

External HDD are ntfs

 

[cchr82]

Still need more information on the hardware, but

My experience with a similar situation a few years ago was that attaching an external USB drive to the FreeNAS system to move data onto the NAS was not a workable solution. Data needs to flow into the system via the network and the network shares that have been configured.
Also, where were you putting the data you were copying to the NAS?

OK thanks. I'll try to copy the data over the network instead and see if it crashes.

I was putting the data into a dataset I created in the share. /mnt/data(the pool)/media/..../more folders

 

[Chris Moore]

External HDD are ntfs

But are they USB? I am guessing, yes?
I had big trouble trying to get that to work. I had multiple external drives I dumped into my NAS and I ended up having to do it all from a Windows computer through the network because I had crashing issues also. In my situation, I was guessing that the controller in the external drive just couldn't talk to FreeNAS properly or maybe the NTFS file system was throwing it somehow.

 

[cchr82]

But are they USB? I am guessing, yes?
I had big trouble trying to get that to work. I had multiple external drives I dumped into my NAS and I ended up having to do it all from a Windows computer through the network because I had crashing issues also. In my situation, I was guessing that the controller in the external drive just couldn't talk to FreeNAS properly or maybe the NTFS file system was throwing it somehow.

Sorry, yes I am using USB. That actually reminded me that I think when I was copying data from one of my older internal HDD (via sata) I dont remember it restarting and thought the issue was solved, only to find that it started again when I tried transferring via USB. Tonight I will go back again, test out copying more from the internal drive and then confirm that issues only occur via USB. Then finally transfer external drive data over network. Thanks for advice and will let you know what happens.

 

[cchr82]

Sorry, yes I am using USB. That actually reminded me that I think when I was copying data from one of my older internal HDD (via sata) I dont remember it restarting and thought the issue was solved, only to find that it started again when I tried transferring via USB. Tonight I will go back again, test out copying more from the internal drive and then confirm that issues only occur via USB. Then finally transfer external drive data over network. Thanks for advice and will let you know what happens.

 

[cchr82]

So last night I tried copying data from old internal HDD via SATA connection. Similarly to via USB, copying of the data directly also crashed FreeNAS after copying for about ~ 1 hour.

But copying my NTFS data over the network using a smb share caused no problems. I was able to copy about ~1 TB of data without issues. Thanks so much for the advice!

 

[cchr82]

So it seems as though the issues are starting to occur again. While transferring data over the network did allow me to copy all of my data to the pool without it crashing, it did not stop my problem completely.

Currently I cannot run the computer for more than 24 hours before it will still randomly crash. Anyone have any ideas on how I can begin to debug this problem or monitor it? Also I'm wondering if I need to worry about running the scheduled scrubs now? If I have an unstable system that will randomly crash and it crashes during a scrub, will that destroy all of my data?

 

[rs225]

Has a ryzen cpu problem been ruled out? AMD was doing RMAs on processors that weren't stable with Linux. I think someone else also posted that they turned off cool n quiet, or something like that in the BIOS.

A scrub shouldn't be a risk, and if it crashes in the middle, it will just pick up about where it left off.

 

[Redcoat]

What about the board's Dr Debug utility? Any error codes from it? I know nothing about it, just that it has such a utility according to the manual.

 

[cchr82]

What about the board's Dr Debug utility? Any error codes from it? I know nothing about it, just that it has such a utility according to the manual.

The last time I checked it went through all of the codes without issue. I will double check but not sure its that.

 

[Xelas]

I don't think AMD is a first-class citizen in the FreeBSD world, so driver support has historically been iffy. You are also using desktop hardware - this can cause potential issues down the road as well - for example, many boards don't support anything other than graphics cards in the primary PCIe slots very well. If this will be a dedicated FreeNAS machine, I strongly recommend going with proven hardware builds. Many of the successful (meaning stable for many years running) use SuperMicro server-class boards, which are not necessarily that expensive. FreeNas also doesn't need much of a CPU unless you really get into running many jails or VMs.

 

[cchr82]

Has a ryzen cpu problem been ruled out? AMD was doing RMAs on processors that weren't stable with Linux. I think someone else also posted that they turned off cool n quiet, or something like that in the BIOS.

A scrub shouldn't be a risk, and if it crashes in the middle, it will just pick up about where it left off.

I was able to run all 12threads at 100% using prime95 tool for ~12 hours I think. So I dont think that the CPU was causing the problem but cannot think of another way to check that so maybe it is an issue? I also turned off cool n quiet last night but then it crashed again this afternoon after ~4-5 hours. Is there any way of debugging or capturing a crash report when this happens?

 

[Xelas]

It may not be directly related to the CPU. It very well could be an I/O system somewhere (USB, SATA, etc) that isn't stable with FreeBSD (which FreeNAS is based on).
Incidentally, searching for "Ryzen unstable FreeBSD" brings up results where lots of people are having various issues similar to what you are describing - the systems just aren't stable when stressed with a real workload, even when they pass memtest.
Here's a long thread, for example:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219399

I would ditch that whole setup and move to different hardware. Server hardware is supposed to be able to run stably indefinitely, even when maxed out.

 

[rs225]

I would contact AMD to see if they tell you to RMA it.

 

[cchr82]

I would contact AMD to see if they tell you to RMA it.

Wouldnt it only be a bad CPU if it fails during their bug test? Also it was purchased after the reported bad batch? I could try and see if a new CPU will not crash, but based on everyone's responses above I am skeptical its a faulty CPU?

 

[Redcoat]

Are you by any chance over overclocked? I see this board is said to be a favorite for it.


Sent from my iPhone using Tapatalk