Large-File Transfers -- Peaks to 600MB/s (5-10s) ... drops to 0KB/s & peaks again after pausing.

[TrumanHW]

HARDWARE USED:

• FreeNAS 11.3 U4.1 -- Dell T620 PowerEdge	Networking Equipment between them...	Client: Windows 10 Pro
• Xeon E5 1.8GHz QC v2	• L3 Routing + Wireless AP: Airport Extreme	• Intel i7-8700k
• 48GB ECC 1333	• L2 10Gb Switch: D-Link SFP+	• 32GB DDR4 2600MHz RAM
• 8x 10TB IBM-HGST 7.2k	• SFP+ (Cisco) DAC Cable	• WD Black: NVMe (easily sustains 2+ GBps)
• NIC: Chelsio: T520-SO-CR (SFP+)		• NIC: Chelsio: T520-SO-CR (SFP+) also ...

Transferring data TO the FN server initially starts normally ... ramping up from 150MB/s and quickly reaching 500-600MB/s...

After perhaps 10 seconds however it will start falling until reaching 20KB/s where it'll stay, or as the cycles iterate, will drop to 0KB/s.

I experimented with pausing the transfer via Win10 to see what occurred upon resuming:
Pausing the transfer never occurs quickly, and sometimes takes 2+ minutes to completely respond to the pause command.

During this time, especially if the transfer rate slowed to under 1MB or down to 0KB/s even ...
-- I CAN ping the IP and get normal-latency for replies ...
-- after noticing the CPU utilization wasn't updating I attempted to reload FreeNAS (reload web page IP address) ...
Often it wouldn't load and would eventually reply:
Connecting to FreeNAS ... Make sure the FreeNAS system is powered on and connected to the network.

Once the transfer completes pausing on Windows, the FreeNAS server again, becomes responsive.
I noticed getting even higher peak transfer speeds upon resuming if I waited a little longer after it paused the transfer.

After the transfer was resumed...
The slowest 20% of transfers peaked out at 225MB/s ... though, only for a few seconds before falling again.
The slowest 50% of transfers peaked out at 300MB/s ... though, "" "" same as above.... "" ""
Peak avg. ±10% of transfers peaked out at 375MB/s ... though, "" "" same as above.... "" ""
The highest 50% of transfers peaked out at 450MB/s ... though, "" "" same as above.... "" ""
The highest 15% of transfers peaked out at 550MB/s ... though, "" "" same as above.... "" ""
The highest 2% of transfers peaked out at 1.2 GB/s ... though, "" "" same as above.... "" ""

That's just to provide an overview of how massive the range is, despite the file sizes all being large ...

Though like everyone, I'd love the fastest of the bunch to be the default -- anything over 300MB/s sustained would've been great, and totally acceptable.

But over the course no more than 10 seconds after resuming, it'd follow this trend ... before resuming to just a few kb or even 0.
100MB/s, 200MB/s, 300MB/s, 200MB/s 300MB/s 500MB/s, 200MB/s 50MB/s ... 50KB/s ... 0 ....

While the transfer rates dropped to their lowest,

Local Read // Write test on (source) NVMe still yielded: 2,600MB/s ... for 16GB Tests...
ping tests yielded the same results as when it was working ... BUT:

Because the CPU's utilization wasn't refreshing quickly I reloaded the web-page ... on to get:
Connecting to FreeNAS ... Make sure the FreeNAS system is powered on and connected to the network.


The IP // web page would resume responding once Windows Pause-command was successfully implemented ...


Time where the the webpage was accessible while waiting for the transfer to complete pausing,
the dropping CPU utilization (below 5-10%) would coincide with the transfer pausing...


And as mentioned above ... the FreeNAS server pings just fine, even when it can't be reached, even before the transfer rate hits zero!
(though, it's difficult to orchestrate some of these tests to determine causation.)


Last night the transfer behavior was a little different.
It'd start quick, slow down (as mentioned) ... but it'd power through 3-15GB files (even if only at 1-2MB/s) ...
But would start each NEW file ... by jumping up to 500-600MB/s for a few seconds before running through the remainder of the file slowly.
I was awake long enough to catch two issues which required my skipping the file for some reason, but when I woke up, it'd completed.

The files I did in the top description were slightly smaller (like 300MB - 1.5GB) ...


in those brief periods of peaks between ~500MB/s --> 1.2GB/s the CPU utilization would range from 45-60%...
However, the histograph for temperature and IO or disk temperatures all looked like 'pretty low' utilization and low temperatures shown to be FLAT in the histograph.



I feel badly having such an absurdly RIDICULOUS set of symptoms and questions. I cannot IMAGINE practically searching for someone who's had similar results ... and I'm skeptical that anyone's ever had this same exact issue. :) I'll be AMAZED if anyone has any idea WTH this is.


Thanks to any and everyone who's taken any time to read any of this. I really have done my best to describe it ... but so WIERD to say the least.


I'll continue checking to see if it's another layer at fault
(the switch, a cable, a CPU, cooling, something... I have no idea)...
...and will update this thread accordingly. Thanks!

 

[c77dk]

Local Read // Write test on (source) NVMe still yielded: 2,600MB/s ... for 16GB Tests...
ping tests yielded the same results as when it was working ... BUT:

Have you tested on the FreeNAS server as well? My initial thought is to check the disk subsystem on FreeNAS

 

[Constantin]

One of the more infuriating aspects of OSX is how system preferences re: network adapters are routinely ignored. I know you have a Windows system but I'd try and disable its Wifi, followed by re-running the tests. I have to turn off Wifi every time I connect to the 10GbE network adapter because sometimes WiFi still takes precedence over 10GbE. That's the only way I get consistent 10GbE connections.

 

[TrumanHW]

...infuriating aspects of OSX ....

None of the issues involved using OS X, at all actually... (if that's the premise you were responding to..?)
The only 'use' of OS X in this case was in using Team Viewer from my mac to control the PC sometimes...

I keep WiFi disconnected except if there's a problem with the adapter or something.
But didn't have WiFi enabled at any point, either.

When you say "re-run the tests" you mean ... run transfers again..?
I did actually attempt another transfer and it was actually slower (over all).


I'm using LRO and LSO ... is there any chance that's causing a problem..?

 

[TrumanHW]

Have you tested on the FreeNAS server as well? ...check the disk subsystem on FreeNAS..?

You mean drill down to the IO report ...? Sure. (Good Idea) ...
Which all stats would would be most helpful for me to post..?

Example of peak transfer speeds... in reviewing the video I made ... it took about 3 minutes to pause each time (via Win10).
When it resumed, many times it'd exceed 600MB/s ... and perhaps there was a proportion of how long it took to slow back down, and how long it was paused for ..?

 

[c77dk]

You mean drill down to the IO report ...? Sure. (Good Idea) ...
Which all stats would would be most helpful for me to post..?

I was thinking of doing some "iozone" or similar tests directly on the FreeNAS server, just to see if it's network or disks. The complete stall of transfer and not being able to get into the web-UI had me thinking of a disk/controller issue. If the local transfers doesn't suffer, then it's on to the network stack. just like eating an elephant - one small bite at a time and you hopefully find the problem.

 

[TrumanHW]

...iozone (or similar) tests (directly on FreeNAS) to distinguish whether it's the network or the HDs.
The transfer-stall being unable to access web-UI seemed like either the HDs or the SAS controller.

Thanks, good ideas ... what do you think of the below

I have to run out real quick, but once I'm planning to:
• Swap the SAS controller for another I have already.
• Put an SSD in a PCI slot (just an AHCI drive) to test real throughput for local transfers.
Obviously that'll let me rule out and test a few things ...

Drawbacks of the tests:
-- Local transfer will omit any affects of an SMB transfer on the CPU.



Order of likeliest plausibility:
Would you mind thinking about what's most likely before I start swapping components, etc..?

HD Issue:
-- The HD temps don't fluctuate & this would be literally unprecedented behavior if it's actually the HD...
-- It'd also require multiple HDs to have the same problem, which makes me think the HDs are the least-likely culprit.
--Agree or Disagree-- ...?

Network issue:
-- The NICs are brand new (though that doesn't make them unlikely).
-- The Switch must be working, as I'm controlling Win10 via TeamViewer which I can only do if the Switch is working.
--Agree or Disagree-- ...?

SAS-2 Controller:
-- I have another HBA to swap this SAS-2 controller with in case it's failing...which could cause it to run hotter.
-- Speaking of how hot we all know they get -- maybe the poor ventilation within this case is inadequate for this HBA ...
(I'm sure these LSI 9300-8i get even hotter at speeds between 300MB - 1.2GB/s)
--Agree or Disagree-- ...?


Perhaps the HBA has farted the bed until the temp drops down.
Benefit of this hypothesis: Intermittent SAS function would still allow FreeNAS to be pinged.
Problem w the hypothesis: FreeNAS is installed on an SSD -- connected via SATA. If the OS [is] isolated from the HBAs performance, why won't it load??.
(It's annoying that the SSD's connection likely negates what was otherwise an elegant solution... explaining why it can be pinged but slowed down after heating up or something)

Opinion: However unlikely the SAS hypothesis is, the ultimate culprit seems like it's a likely dependency to both the HBA & SSD.


Set Network MTU to 9000:
(though I've heard that it's unreliable and more trouble than it's worth, even if only for diagnostic purposes, would it make sense to attempt 9000 MTU?)
Which device' settings must be:
• Dell FreeNAS (via Chelsio T520 over SFP+ DAC)
• D-Link SFP+ Switch
• Win10 Pro i7 (via Chelsio T520 over SFP+ DAC)
--Agree or Disagree-- ...?




Again: Once back I'll:
• Swap the SAS controller for another I have already.
• Put an SSD in a PCI slot (just an AHCI drive) to test real throughput for local transfers.
Obviously that'll let me rule out and test a few things ...

Drawbacks of the tests:
-- Local transfer will omit any affects of an SMB transfer on the CPU.


I have to ...
Repair a mac
USPS a laptop
Drop a MBP & replace client's router...


Again, thanks!

 

[Constantin]

None of the issues involved using OS X, at all actually... (if that's the premise you were responding to..?)

Consider re-reading my message. Is there a way to disable any and all potential parallel paths to and from the server? In my case, I have a WiFi connection that, on occasion, will take precedence over a 10GbE and a 1GbE wired connection, even though system preferences are set to give WiFi the last spot (think of this as a failover-LAGG with the 10GbE in top spot, 1GbE second place, Wifi in 3rd place).

• It doesn't look like your are using SMR drives, which could explain the inconsistent performance.
• However, do you have some sort of SLOG? A SLOG SSD with a fast cache up front but a slow back end could explain these issues.
• The LSI units are famous for getting hot, I struggle to keep my chip below 45*C.
• I would not fiddle with MTU sizes until you have the transfer issue sorted.
• I doubt the HBA is the issue (though it is a good idea to check the firmware and ensure it's the latest non-RAID, IT version) as the FreeNAS should not stop being responsive re: the UI just because the HBA I/O is locked up.
• I would check the speeds of the local disks / pools just to confirm what the pool can handle. Use random data and test locally via the shell.

 

[Herr_Merlin]

Why the heck change hardware prior to local testing?
Please type the exact name of those Heads in here

 

[TrumanHW]

Excellent question re: SLOG // L2arc, unfortunately, nope -- none installed.
I'm providing you pictures ... I did mention that WiFi was disabled but maybe others have lied and delayed solutions by their omissions..?

WiFi is disabled on the PC
No WiFi installed on FreeNAS
No WIFI ON THE PC, EITHER. :)
See pics of NICs so you know they're all either unplugged or disabled.

8x HGST are UltraStar HDs (all RAIDZ2 HDs)
All are therefore, PMR (Perpendicular Magnetic Recording)
See pic of HD model if you'd like to confirm...


8x = HGST UltraStar -- HUH721010AL4200
1x small boot SSD drive.
No L2arc nor SLOG -- didn't want to add em 'til after getting the basics worked out. :)


The ONLY WAY link on FreeNAS is the 10GbE SFP+ (only one port of the 2x in the card is even plugged in).
The 1GbE ports on the FreeNAS aren't plugged in and they're not configured.
No WiFi card installed in FreeNAS


Windows 10 Pro:
SFP+ has ONE CABLE connected ...
The other port is empty.
The 1GbE is not connected
The WiFi is disabled.


And ... the above settings has been that way for a couple of days, since I received the Chelsio T520 cards.
Basically, I agreed with you that making it absolutely impossible to confuse how it's connected would be preferable.


Honestly though, I hope there IS something I boneheadedly overlooked...and that this doesn't actually require "inventing a wheel" ...


Thank you both for all your time and help trying to figure this out.



FreeNAS RAIDz2 ARRAY DRIVES:

FreeNAS Hard Drives 'devs' Installed:

FreeNAS Device Connectivity:

WINDOWS 10 PRO CONNECTIVITY SETTINGS:

 

[Herr_Merlin]

Could you please test the performance locally in the FreeNAS as mentioned above by the others?
This way we know if the issue network or local system?

 

[TrumanHW]

This way we know if the issue network or local system?

Thank you Merlin:
I'd be screwed trying to use a system like this,
...without help from people like you.


Put about 400GB of data inside the FreeNAS machine and used the IMPORT DRIVE feature ... which has taken over 10 hours to copy ~350GB.


The drives all show PEAK writes of a whopping 4MB/s according to IO ...

(I can consolidate all the pictures showing the histograph of IO performance over the period of time tomorrow for you to see) ...

Are there any other pictures you'd like of activity..? The CPU, or temps, or anything ..?

I'm thinking the RAM is what I was writing to when it hit 1GB/s ... and once the RAM filled up, it took about 3min because 8x Drives writing at 4MB/s each for 180 seconds can write out about 5GB of the contents that was in RAM... once all the data sent to the NAS was caught up, it was able to pause ... at which point the cycle could be repeated.

I DO NOT think the drives are limited to all of 4MB/s. Could it be the SAS-2 card or something..?

I have other nearly identical systems here as well as additional SAS controllers.
I have 7x 6TB HGST UltraStar drives (maybe 8, 1 of them I need to test) ...
I have 8x 4TB HGST UltraStar drives in another Dell T320 that's virtually identical

(except I don't have a new-model SFP+ card for these two)

What testing can I do ...?

FYI, transferring this 350GB or so has taken 10 HOURS...

whereas I have cloned (verbatim) 4TB to the 4TB ultrastar drives that are here in under 14 hours for a RAID array I recovered using DD Rescue from which I was able to clone 4 HDs in each of the two T320 machines I have here ... though, I wasn't using the 10TB drives.

Still, the data copied yesterday took about an HOUR to copy 500GB ... but required I pause the transfers ... it took about 3min to pause, but after it did it'd write data at over 1GB/s at times (again, perhaps writing to the RAM or something, but no matter what, it still copied the data at least 10x faster).

Any ideas..?

 

[Herr_Merlin]

Please check this for a test as an example https://calomel.org/zfs_raid_speed_capacity.html
What you need to test the different IO and write/read speeds for different file sizes and queue deeps.

ZFS is really complicated to tune for each deployment but once you figure it out it's an amazement. I was skeptical at first myself but now I know how I can handle it and how I can get great performance out of it. This performance is tuned for each use case and might be horrible for use cases.

Can you share your detailed configuration of your dataset and the underlying pool?
Are you using synced writes? If so there is no way to not use a SLOG/ZIL.
Do you use atime?
SMB is a single core task is a single core maxed out?
What level of compression are you using?
Are you using dedub?

BR

 

[TrumanHW]

Ha you're good! I had just disabled sync on my dataset about 25 min before you replied.
It had been set to the middle ... basically it would sync at the request of the client OS ...

but remember, the results above are with a LOCAL SSD INSIDE FreeNAS -- using the IMPORT DISK feature ...
Meaning whatever the effects of sync or SMB, etc., shouldn't be an issue ... (SHOULDN'T be, though, who knows, re: sync right..? And of course were it set to always it would have mattered ... but that's more re: the SMB and CPU issues.)

I LOVE that website (Calomel) and have read the shit out of thehir posts...

What would you like me to provide re: underlying pool..?
I can send screenshots of all of it if that's easier ...

I will be adding a SLOG, ZIL and L2arc ... but I want to get the core working well before adding more parts ...

Remember, the files I've been transferring are all very large files (minimum 300MB)
... yet, the writes are at 4.3MB/s ... per HD!!



atime..? What's that..?

Yes, when I was transferring things via SMB I think the 1x CPU core was about 100% ... which on a QC machine it'll show as 25% ...
While individual cores were reaching 100% ... I think 2 cores were actually reaching 100% ...

SMB is the only sharing service I have enabled as people emphasized that SMB is faster than NFS and I should use SMB ... and until I COULD use SMB (which I had to wait a damned YEAR for a FreeNAS update in which the service would actually start!) ...

It's almost like FreeNAS isn't particularly polished or something. lol. :)


I'm using the LZ4 at 1.01% ... I could disable compression, as the files are all video lately which are likely incompressible ... and would increase CPU usage for a task it can't really benefit from ... if that's what you're thinking..?

 

[anodos]

Ha you're good! I had just disabled sync on my dataset about 25 min before you replied.
It had been set to the middle ... basically it would sync at the request of the client OS ...

but remember, the results above are with a LOCAL SSD INSIDE FreeNAS -- using the IMPORT DISK feature ...
Meaning whatever the effects of sync or SMB, etc., shouldn't be an issue ... (SHOULDN'T be, though, who knows, re: sync right..? And of course were it set to always it would have mattered ... but that's more re: the SMB and CPU issues.)

I LOVE that website (Calomel) and have read the **** out of thehir posts...

What would you like me to provide re: underlying pool..?
I can send screenshots of all of it if that's easier ...

I will be adding a SLOG, ZIL and L2arc ... but I want to get the core working well before adding more parts ...

Remember, the files I've been transferring are all very large files (minimum 300MB)
... yet, the writes are at 4.3MB/s ... per HD!!



atime..? What's that..?

Yes, when I was transferring things via SMB I think the 1x CPU core was about 100% ... which on a QC machine it'll show as 25% ...
While individual cores were reaching 100% ... I think 2 cores were actually reaching 100% ...

SMB is the only sharing service I have enabled as people emphasized that SMB is faster than NFS and I should use SMB ... and until I COULD use SMB (which I had to wait a damned YEAR for a FreeNAS update in which the service would actually start!) ...

It's almost like FreeNAS isn't particularly polished or something. lol. :)

Did you file a bug ticket about SMB not starting?

 

[TrumanHW]

Did you file a bug ticket about SMB not starting?

I think you and I spoke about it at the time actually ... and you said to wait until the next version. I can find the thread ... maybe I'm mistaken but I recall your avatar. Thanks for checking in, also -- and of course, it is resolved (which I mention above in this thread also I think) ... as of 11.3. But it burnt about a year of waiting for SMB to start working again as I couldn't rule it out and people assured me that was the "key" to fixing the performance issues.

Again though, thanks for following up.

It was a thread in which several people (again, if memory serves) were all having SMB issues ... I had to search for a module v4 or something ... but long since resolved and on to a series of newer frustrations. :)

 

[no_connection]

I'm no expert but it sounds like you see fast build up to txg and everything stalls when the next one fills up and first one take too long to flush to disk?

Not 100% sure how you have the pool configured but if it's 8 drive z2 than a single drive could bring everything down right?

How do the graphs look like for all the drives in reporting during a transfer?

Have you booted the server with any tool and done a similar test to HDTune reading the entire drive and graph all of them to make sure they behave as expected?

Any IRQ storm throttle the HBA?

 

[TrumanHW]

1. sounds like you see fast build up to txg & everything stalls when the next one fills up & first one take too long to flush to disk?

The only info I found for TXG is: Tandem Switchless Gateway ...?

Remember, this transfer wasn't dependent on the NIC at all; The content was in a local hard drive in which I used
Import Disk ... feature.

It's a RAIDz2 array, right..? If so, any problem-drives will slow the group...

Yes: RAIDz2 comprised of 8x HGST 7.2k PMR drives.
Another thing I was thinking ... I have condensed all the drives IO performance on to one image. :)

Do the graphs look uniform during the transfers?

Yes, VERY VERY much so. In fact, they all have the same average and max ... ultra impressive numbers I might add ...
A whole 1.5MB/s !! Which, when all 8 drives provide this blistering performance provides 12MB/s !! (amazing, right..?)

Have you booted the server with any tool like HDTune to read the entire drive & graphed all to ensure they behave as expected?

No, there's data on the drives ... but, I have other arrays which do not have data on them and are VERY similar drives...

7x HGST UltraStar 6TB 7200rpm SAS2 drives
8x HGST UltraStar 4TB 7200rpm SAS2 drives. <-- these ARE tested & took ~12 hours to clone 4TB via DD rescue.

Any IRQ storm throttle the HBA?

How do I check this ...? Would it be a result of additional devices..? I have the minimum of devices installed:
SAS-2 controller
10Gb SFP+
...that's it.

 

[no_connection]

The only info I found for TXG is: Tandem Switchless Gateway ...?

Tansaction group. It's how ZFS can do the batch writing instead of sending every little thing directly to disk.

No, there's data on the drives

It's only read but just checking if you noticed something odd with one of the drives.

How do I check this ...?

I think it would show up in console if it throttle but not sure.


Do a scrub give expected speed/performance?

 

[TrumanHW]

Tansaction group. It's how ZFS can do the batch writing instead of sending every little thing directly to disk.

Ahh...thanks. Yes, I think it was just loading the RAM and as it flushed it I was pausing ... but still, for some reason a local transfer was even slower than over 10GbE ..??

It's only read but just checking if you noticed something odd with one of the drives.

Unfortunately, no. That would've been nice though. :'(

I could superimpose every drive's graph from the history over each other and it'd look like one result.

I think it would show up in console if it throttle but not sure.

Like, the CPU ..? I was looking for any indicators but really saw none. I'm going to try swapping parts and testing on other machines today.

Do a scrub give expected speed/performance?

I'm not sure I'd know how to tell the difference.
Are scrubs based solely off the amount of data used on each drive..?

Does a scrub show you individual drive performance...?
It's not writing data during a scrub, correct..? I wouldn't even have a MB/s I'd think ..?

Thanks again for the reply....