Truenas going offline during large file transfers

[ryanvox]

I have two Truenas systems. A main and a backup. I got my backup going and am transferring data from main to backup. The large file transfer is messing with my main truenas. I first noticed that plex stopped working. then i couldn't get to the web interface to see what is going on. It just says "Connecting to TrueNAS ... Make sure the TrueNAS system is powered on and connected to the network." It is still connected because i can see the file transfer still taking place on my backup device. I can also still ping the main truenas. I thought maybe the output was causing issues which is why plex is not working. I purchased two sfp pcie cards and put those into both truenas boxes and have them connected peer to peer and have RSync set up to use that nic for file transfers leaving the main nic open for everything else. I kicked off Rsync and i've lost plex again and can no longer get into the web interface on the main truenas. I plugged a monitor into the main truenas and am seeing the following error, "freenas.local collected 3066 - - plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics". Any help would be appreciated. I just upgraded from Freenas and didn't have any issues like this. This isn't just an issue with Rsync either. If i file share into each device from a windows machine and transfer files that way i have the same issue. Here are the specs of my main device.

MSI Z370 Gaming Plus
Intel I7-8700
Micron RAM - 32GB - PC4-3200-AA-UA2-11
(8) 6TB WD Red drives in RAID Z2 config
Generic 160GB m.2 drive
Corsair RM750

 

[NugentS]

A few questions for you
1. How are those HDD's connected to the motherboard
2. What model are the HDD's
3. Are you using dedupe?

 

[ryanvox]

A few questions for you
1. How are those HDD's connected to the motherboard
2. What model are the HDD's
3. Are you using dedupe?

1. 5 of the drives are connected via sata. The other 3 are connected via sata as well, but to a PCIE sata expansion card
2. WD60EFAX
3. I'm not sure what dedupe is

More info: I am running Truenas 13.0 and have an intel NIC.

 

[Morris]

Two things catch my attention:
- Are your WD Red drives CMR? If they are new, then they are likely SMR will not perform well and a rebuild will take forever.
- You have 3 drives on a PCIe card. Depending on the chipset this could be an issue. Also, how many PCIe lanes for that STAT card?

For a start, put one more drive on the motherboard's SATA as it can support 6 reducing demand on that add in card.

 

[ryanvox]

Two things catch my attention:
- Are your WD Red drives CMR? If they are new, then they are likely SMR will not perform well and a rebuild will take forever.
- You have 3 drives on a PCIe card. Depending on the chipset this could be an issue. Also, how many PCIe lanes for that STAT card?

For a start, put one more drive on the motherboard's SATA as it can support 6 reducing demand on that add in card.

- They are SMR. Here is what i bought - https://www.amazon.com/dp/B07MYL7KVK?psc=1&ref=ppx_yo2ov_dt_b_product_details

- 16x lanes for that card. I'll get one of the drives moved over to the motherboard. I only have 6 sata ports on the board unfortunately. The OS SSD is an m2 and that lane takes up the first sata port. So i can clone m2 to a regular sata ssd, put that drive on the pcie expansion slot and move one the pool drives back to the motherboard.

It makes sense that the pcie expansion is the culprit. This has been a headless build and i just hooked up a monitor on it and saw the error i've attached. looks like a pool issue. I am currently testing ada 6 and 8 to make sure the drives are ok, but i'd be willing to bet that those two drives are on the pcie expansion.

 

[Morris]

IS your HBA an LSI chipset card? Some others have issues. If there are a lot of writes going on while you are doing that copy, the fact that your drives are SMA drives may be slowing things down. Errors are a different story. If any of those drives fail you should replace them with CMR drives, even if they must be larger. Read this to understand the issue:

WD Red SMR vs CMR Tested Avoid Red SMR

We tested WD Red SMR v CMR drives to see if there was indeed a significant impact with the change. We found SMR can put data at risk 13-16x longer than CMR

www.servethehome.com

I'd expect you can read from the SMR drives and not have issues yet I can't say for certain.

Good luck and let us know how it goes.

 

[Davvo]

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Also, HBAs need to be properly cooled.

 

[NugentS]

- They are SMR. Here is what i bought - https://www.amazon.com/dp/B07MYL7KVK?psc=1&ref=ppx_yo2ov_dt_b_product_details

- 16x lanes for that card. I'll get one of the drives moved over to the motherboard. I only have 6 sata ports on the board unfortunately. The OS SSD is an m2 and that lane takes up the first sata port. So i can clone m2 to a regular sata ssd, put that drive on the pcie expansion slot and move one the pool drives back to the motherboard.

It makes sense that the pcie expansion is the culprit. This has been a headless build and i just hooked up a monitor on it and saw the error i've attached. looks like a pool issue. I am currently testing ada 6 and 8 to make sure the drives are ok, but i'd be willing to bet that those two drives are on the pcie expansion.

So you have two - probably fatal issues.
1. SMR Drives. You are trying to dump a load of data to an pool based on SMR Drives - this will not work well. SMR drives are not fit for purpose. In particular WD RED SMR Drives are what I would basically describe as Landfill. If they are a recent purchase then send them back and get WD Pro, or some other drive type. Make sure they are not SMR

2. Your SATA adapter card almost certainly isn't an LSI HBA Card. This is also potentially fatal - what exactly is it? Your idea of using it for the boot drive and putting all the data drives on the motherboard is possibly a good one, but a better long term solution is to replace it with a proper card. See "Art of Server" on ebay for something that WILL work and won't cost too much. For HDD's you don't need anything more than a card in the 9200 range.

Others have posted links to stuff you NEED to read about disks and HBA's - as you clearly haven't done your research before spending money. TrueNAS is an excellent NAS but ZFS has certain basic requirements that aren't onerous, but are fatal if not adhered to.