Cannot sustain GB performance over long/large file transfers ?

[cheezystix]

Completed my FreeNAS build last weekend.
All seemed fine, getting expected 110 MB/s transfers on Gigabit Network (all CAT6 wires and all Gigabit Netgear prosafe switches)
I also have a QNAP NAS setup that also gets 110 MB/s transfers on same network and same switch

But my issue is that during long/large file transfers (backing up my PLEX library of videos) the FreeNAS cannot sustain 110 MB/s ?
In fact it will every 20 secs or so drop down to 3 MB/s before again going back to 110 MB/s ?
doing this over and over for hours

BTW, There is no other network traffic during these transfers (just me).

My QNAP NAS DOES NOT do this on the same network and same switch (it can sustain 110 MB/s for hours)

Any ideas on what could be causing the issue ?

HARDWARE
Mobo ASUS P8Z77-V PRO
CPU Intel i5-3570K
RAM 16GB DDR3 1600

DRIVES;
Boot Samsung SSD EVO 250GB
6x Seagate 5TB HDD
RAID Z

 

[melloa]

Did you test the pool write speed? If might be that it's pausing to write, before resuming. You probably can find posts here on how to test or can try:

Test write and read pool's speed

 

[Chris Moore]

DRIVES;
6x Seagate 5TB HDD
RAID Z

Is that RAIDz2 or an actual RAIDz1 pool? Not that it matters much.

In fact it will every 20 secs or so drop down to 3 MB/s before again going back to 110 MB/s ?

I have seen this type of behavior before. This speed fluctuation is the cache flushing to disk. Transfer runs fast while the cache fills, then slows down significantly when the cache flushes, then speeds back up. The problem, your disk pool can't keep up with the transfer.

The problem might even be just one of the disks because a single vdev pool is subject to the speed of the slowest disk in the vdev.
We could determine more about the cause (regarding the disks) if you give the model number of the disks you are using.

@jgreco created a tool for testing the speed of your pool. You might want to take a look at it:

solnet-array-test (for drive / array speed) non destructive test
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

 

[cheezystix]

Did not test pool write speed (yet)

I am using actual RAIDz1 (not critical data, just backups)

Disks I am using are 5x Seagate 5TB desktop drives (not NAS) that I had laying around
and actually 1x Western Digital Black 5TB desktop drive

Sorry for slow response .. I was researching on this forum and there were recommendations to turn on jumbo frames.
I tried this and ended up bringing down my entire NAS (web interface was un-accessible)
Took awhile to dig out the server, and hook up another monitor to undo what I had done.
But it's back online now

 

[cheezystix]

I believe most (if not all) the Seagate drives are model ST5000DM000
The Western Digital Drive is model WD5001FZWX

 

[cheezystix]

I have seen this type of behavior before. This speed fluctuation is the cache flushing to disk. Transfer runs fast while the cache fills, then slows down significantly when the cache flushes, then speeds back up. The problem, your disk pool can't keep up with the transfer.

This could be the issue

Perhaps using these older non-nas drives are an issue, but was trying to repurpose them and build a useful NAS

My QNAP has all Seagate - IronWolf NAS drives .. and doesn't have these issues

 

[jgreco]

I believe most (if not all) the Seagate drives are model ST5000DM000
The Western Digital Drive is model WD5001FZWX

The ST5000DM000 is thought by many to be a SMR drive, so this is quite possibly all about the drives.

https://forums.unraid.net/topic/324...mystery-solved/?do=findComment&comment=489442

If it's in fact an SMR drive, there won't be much FreeNAS can do about it.

 

[cheezystix]

Thx .. yep I believe it may be time to replace with some better drives.
Best buy recently was selling 10TB external drives which when shucked were white label, but actually WD RED 10TB NAS drives for $169/each
so may replace with similar

Suddenly FreeNAS isn't so FREE ;)

 

[jgreco]

Well, if this is for Plex, that's a read-mostly type of activity. If you are creating a Plex library that you are adding to, and not deleting from, this is actually a use model that the SMR drives aren't too bad at. You just have to get over the poor write performance.

It's not optimal, but it might be an option to just keep using the drives for that purpose.

 

[cheezystix]

I planned on using this NAS just for backups (writing nightly), including backing up my PLEX library.
I prefer PLEX reading library from local disk.
So intent was to write more than read.

I think you guys were spot on about the SMR drives, and I have a plan now to replace.

Thanks all for your help.

 

[Chris Moore]

I had four of the 5TB Seagate Desktop drives in a backup pool for a while. They were very slow. Slower than expected. I replaced them with 6TB drives from another company and got significantly better transfer speed. If you have a plan to replace them, that will probably be the solution.

 

[jgreco]

I had four of the 5TB Seagate Desktop drives in a backup pool for a while. They were very slow. Slower than expected. I replaced them with 6TB drives from another company and got significantly better transfer speed. If you have a plan to replace them, that will probably be the solution.

And it's worth being aware that many people have complained that Seagate has been crappy about identifying SMR drives.

 

[cheezystix]

All my NAS builds using Seagate NAS Drives saturate my 1GB network .. so I really can't complain about performance here.
But I will take a look at other options when replacing.

This NAS build was to be a cheap archival freenas build .. nothing fancy. So I was hoping to use a bunch of 5TB desktop SMR drives I had laying around. But the cache flushing Chris pointed out was killing my transfer rates.

Hence the reason I am migrating to better drives. Still working on this.

 

[cheezystix]

BTW, fairly recent NAS drive performance write up
https://www.tomshardware.com/reviews/wd-red-10tb-8tb-nas-hdd,5277-3.html

 

[Chris Moore]

This NAS build was to be a cheap archival freenas build .. nothing fancy. So I was hoping to use a bunch of 5TB desktop SMR drives I had laying around. But the cache flushing Chris pointed out was killing my transfer rates.

Hence the reason I am migrating to better drives. Still working on this.

As far as I have been able to discover, Seagate is not using SMR in their NAS or Exos (Constellation) models, just in the Desktop (Barracuda) and Archive drives.

 

[cheezystix]

Just a quick follow-up
Last night I rebuilt my data pool using only NAS hard-drives and I am NO LONGER seeing the poor performance during long/large file copies.
So the issue did in fact appear to be from SMR drives I had before that were experiencing cache flushing.

I am now seeing a steady 113 MB/s for hours on end (normal for gigabit network)

Thanks again

 

[jgreco]

Just a quick follow-up
Last night I rebuilt my data pool using only NAS hard-drives and I am NO LONGER seeing the poor performance during long/large file copies.
So the issue did in fact appear to be from SMR drives I had before that were experiencing cache flushing.

I am now seeing a steady 113 MB/s for hours on end (normal for gigabit network)

Thanks again

Appreciate the follow-up. Good to hear it wasn't really a FreeNAS issue. Sorry about the SMR drives. :-(

 

[djb]

I have same issues with strange behaviour on transfers. sometimes it stacks on 0mb/s for 1-2 minutes and then restarting. Is affecting also the SMB Service which it seems that is restarting.
If it is a cache problem, wouldn't be better to use SSD drives ? for example crucial MX,BX series 2,5" Sata SSD.
I have 2 truenas boxes doing the same strange behaviour, 16 and 32GB Ram, with healthy storage mirror pools with SSD's. Intel network cards also.

Any suggestions ?

 

[ChrisRJ]

@djb please describe your system in detail, as per the forum rules. Without such information we cannot help you.

 

[djb]

@djb please describe your system in detail, as per the forum rules. Without such information we cannot help you.

Hello Sir,
the system is i5-4690 3,3Ghz 4 core 4 threads. Ram 32GB.
the storage pool is healthy 2Xmirror SSD's Crucial 480GB BX500
onboard NIC is Intel. Truenas Core 12 U8.1