Transfer speeds slow to 0 Mbps for several minute, then returns to normal

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30
I'm having an issue that seems odd to me. I'm running TrueNas 12 and have two disks configured in a mirror. This pool is configured with 3 datasets, two which are being shared as samba shares, and one that's being shared as an NFS share.

Here's the issue: When I transfer data using Windows from Samba Share #1 to Samba Share #2, the transfer starts nicely (~60MBps), then slows to a stop (0 MBps). It will remain at a stop for several minutes. During this time, there is no read/write capability from any of the datasets, including the NFS share. Then, it'll return to transferring again, and will repeat in a few minutes.

My server is a dedicated hardware with 32GB of mem, and 16 cores.

What could be happening? I would think that perhaps the data would buffer in memory, and then at least if the memory got all used up, it would slow to the write-through speed on the disks.

Any advice?

1626100339312.png
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I'm running TrueNas 12 and have two disks configured in a mirror.
Before we dig too deeply, I want to rule out those drives as a potential pain point - please post exact model numbers eg: ST2000DM008

Compare against the list of known SMR drives maintained by @Yorick here:


and/or use the script created by @Basil Hendroff here:

 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
The symptoms also align with a network driver dumping after an amount of data is pushed through them... perhaps not an Intel card?
 

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30
Before we dig too deeply, I want to rule out those drives as a potential pain point - please post exact model numbers eg: ST2000DM008

Compare against the list of known SMR drives maintained by @Yorick here:


and/or use the script created by @Basil Hendroff here:


Device Model: HITACHI H7220AA30SUN2.0T 1002ME9NJT
Device Model: HITACHI H7220AA30SUN2.0T 1002MEHEPT
 

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30
The symptoms also align with a network driver dumping after an amount of data is pushed through them... perhaps not an Intel card?

I'm using a Broadcom NetXtreme II BCM57711 10-Gigabit PCIe card for this connection without any link aggregation.
 

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30
Before we dig too deeply, I want to rule out those drives as a potential pain point - please post exact model numbers eg: ST2000DM008

Compare against the list of known SMR drives maintained by @Yorick here:


and/or use the script created by @Basil Hendroff here:

1626104907889.png
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I rebooted into BIOS and found SR-IOV was disabled already. So, I enabled it. Since booting back up, the transfers still come to a halt after about 10 minutes of transferring - same exact issue :(
That wasn't what I suggested.

Under Network | Interfaces | That interface | Edit

Tick the box "Disable Hardware Offload"
 

bmoreitdan

Dabbler
Joined
Oct 16, 2018
Messages
30
That wasn't what I suggested.

Under Network | Interfaces | That interface | Edit

Tick the box "Disable Hardware Offload"
Thanks. I didn't realize that's where you meant. I disabled it but I'm still experiencing the issue. I started to tail /var/log/messages and saw a few instances of this....
1626175209001.png


I'd like to look more into the smb logs, but there's just so much there I'm not sure what I'm looking for. If I tail /var/log/samb4/log.smbd, I see some permissions issues

1626175278014.png
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Well, crashing SMBD would certainly produce at least some of the results you're seeing. (but doesn't explain NFS shares also unresponsive)

you might also want to look under /var/db/system/cores to see if you have any dumps in there... then sharing those with @anodos might be useful together with raising a support ticket.
 
Top