Sequential reads from FreeNAS server intermittently dipping to 0MB/s?

[Strohminator]

Apologies - I posted this in the Performance section by mistake - would a mod mind deleting that thread? Thanks!
https://forums.freenas.org/index.ph...server-intermittently-dipping-to-0mb-s.42447/


Hi folks

Long time... Hope everyone is well.

Please help - I've had an intermittent issue with my server that's been driving me around the bend, and I simply can't figure out the cause.

Build is as follows:

Norco RPC-4224
SuperMicro X9SRi-F board
Xeon E5-2620 Hex core chip
128GB LR-DIMM Samsung RAM (4 x 32GB modules)
2 x Dell Perc H310's flashed to 9211 IT mode
8 x 8TB Seagate Archive series drives in RAID-Z2 for a total of 40TB formatted, running a CIFS share
Cisco X3500 ADSL / Gigabit router and a 24-port managed switch, all hard-wired.
Latest 9.3.1 stable build updated


When writing to the server, it saturates a Gigabit connection - 105-113 MB/s, on a stable transfer.

When reading from the server, speeds seem good while reading from ARC, but after a few seconds, will intermittently dip to 0MB/s, then slowly climb up again, only to flatline for a second or so again a few minutes later... my effective average transfer speeds will be anything from 10-30 MB/s, dependent on the server's mood.

I've already disabled HyperThreading on the CPU, and while that seemed to cure it, it was only temporary.

The server seems to do it less often after a fresh restart, and then the issue gets more pronounced after more uptime.

I've tried auto-tune, to no avail.

Scrubs and long SMART checks have returned no alerts as to the ill drive health

I've bypassed both the ADSL router and the 24-port switch, trying to isolate a network issue, to no avail.

The only Plug-In I'm running on FreeNAS is Plex, and it still does this when the Plug-in is switched off.

On a previous server build, (E3-1230 V2 using 32GB of unbuffered RAM) using the same drives and RAID-Z2 volume, the system ran flawlessly... easily getting 90-100MB/s reads.

Please could someone shed some light or point me in the right direction? Is there a memory hole or something poorly configured in the ARC? Faulty memory? A hard drive fault I'm not picking up? I honestly don't know where to fiddle and what to try anymore.

Any help would be appreciated. Thanks in advance.

Regards

Philip

 

[m0nkey_]

Going to need some more information. Can you state your pool utilization, want to make sure you're not above 80%.

When reading from the server, speeds seem good while reading from ARC

Do you have a L2ARC device by any chance?

Also, be interested in the specs of the client machine, specifically the NIC.

 

[Strohminator]

Going to need some more information. Can you state your pool utilization, want to make sure you're not above 80%.

Do you have a L2ARC device by any chance?

Also, be interested in the specs of the client machine, specifically the NIC.

Thanks for the reply

Pool utilization is sitting on about 75-76% at the moment - it's been doing this since about 68% usage when I originally imported the pool on the new system

I was originally running the 8 x 8TB drives off the motherboard's onboard SATA controller - and after tweaking / fiddling, I finally concluded that it must be something to do with the SATA controller - switching to the Perc H310 worked fine for a while (which happened to coincide with a new stable FreeNAS update) - but shortly afterwards, it started acting up again - unsure if that once again coincided with an update...

No L2ARC device

Client machines all experience the same issue:

Acer Veriton all-in-one (Realtek NIC, 8GB RAM, 1TB SSHD, 4570T CPU)
Dell E7440 Ultrabook (Intel NIC, 16GB RAM, 256GB SSD, i5-4310U CPU)
Gaming PC (Intel NIC, 16GB RAM, 512GB SSD, i7-4770k CPU)
Gaming PC2 (Realtek Standalone NIC, 16GB RAM, 512GB SSD, i7-3930k)

So I'm reasonably confident that the issue lay somewhere on the server side.

 

[Strohminator]

Hold the phone...

I just enabled Autotune again, restarted the machine, and the flatlining issue is gone... read speed isn't wonderful - averaging about 65MB/s and fluctuating as far as 44 & 98 MB/s (which could be partially thanks to the Realtek NIC I'm using on the PC I'm on at the moment), but none of the 0MB/s flatline followed by 10-20MB/s, then back to 0 business...

I'll try to copy about 200GB worth of stuff to make sure I fill up the ARC allocation, and see if it speeds still remains consistent.

Assuming there's a prefetch setting or something that I was missing? (Sorry, I know zilch about tuneables - I've only managed to screw them up in the past)

 

[Strohminator]

Nope, a couple of days later, and we're back to square one... slow reads and intermittently dropping to 0MB/s every few seconds.

Please help

 

[BigDave]

Please could someone shed some light or point me in the right direction? Is there a memory hole or something poorly configured in the ARC? Faulty memory? A hard drive fault I'm not picking up? I honestly don't know where to fiddle and what to try anymore.

I think the RAM testing would be logical at this point, since you have not mentioned doing that...

 

[Strohminator]

Hi there gents

Thanks again for the replies and the PM's with advice. I figured an update was due here.

I tried upgrading to 9.10, and got a funny database locked error, then had my system hobble around for 2 weeks before figuring out that my 8GB pen drive used for FreeNAS boot, was sitting over 90% capacity...

I wiped a few of the previous upgrade images from last year, and proceeded to install 9.10.

Lo and behold, the issue seems to be gone, and a whole host of console warning messages and bugs along with it - the system is significantly more stable, and read speeds are back to where they were previously - 90-100MB/s consistently on large sequential transfers, even when using a Realtek NIC on the client machine.

So, no hardware issues - just a seemingly broken FreeNAS build and a muddled upgrade attempt... now SORTED!

Regards,

Philip

PS - could anyone advise as to whether it would be recommended to now enable HyperThreading again, or keep it turned off?

 

[Mirfster]

PS - could anyone advise as to whether it would be recommended to now enable HyperThreading again, or keep it turned off?

I never turn mine off and it doesn't seem to be an issue. If you (or someone else) has reason to then I am all ears and willing to give it a whirl.

P.S. -1 for the RealTek NIC ;)

 

[Strohminator]

I never turn mine off and it doesn't seem to be an issue. If you (or someone else) has reason to then I am all ears and will to give it a whirl.

P.S. -1 for the RealTek NIC ;)

I experimented with turning mine off when transfer rate problems initially surfaced straight after the migration to the new system, and it did seem to help at the time... FreeNAS is also documented as having performance issues with HyperThreading enabled in some instances. I just figured someone might know whether the underlying cause had been discovered and addressed --- But I suppose I could just give it a bash and see for myself.

Lightning sent my gaming rig's onboard LAN port to the big silicon scrap pile in the sky.... And there's no way in hell I'm dropping $60 on an Intel NIC (our local pricing in South Africa is insane) - so the Realtek will have to do for now.

 

[Mirfster]

No worries, if things are good for you then that is all that matters.