I'm typically pretty good at locating the source of performance issues, but these slow network writes have me stumped (and has for awhile).
-> Local Writes are Very Fast (so it's unlikely the disks, controller, server I/O)
-> Network WRITES are Very Slow (consistently exactly 11MB/s), Regardless of Network Protocol (scp, NFS, smb, ftp).
-> Network READS are just fine (>90MB/s).
These three data points are very difficult to reconcile IMO. I should add that Network WRITES via the localhost loopback are just fine.
So, it has to be something to do with the network, but what network issue manifests only in writes versus reads irrespective of the network application used?
I'm not expecting a solution, just a critical evaluation of the above details and some direction of where to go or look next. The usual suspects all seem to be precluded by one of the above data points, and syslog is bingo (at least at default verbosity).
Thanks in Advance for anyones time and consideration,
-Michael
Scenario Details Hardware Details:
FreeNAS-11.2-U6 (Issue also was manifest with same hardware on 9.3)
Supermicro X8S Ti-F w/Xeon E5504 & 12GB ECC RAM
Western Digital 2TB "Blue" Drives qty.4 on Integrated Intel ICH10R SATA Controllers in RAIDZ2 with lz4
Again, Very Fast Local Writes seem to preclude the above ^^from having issues or being the culprit.
Network Details:
Using a single Integrated Intel 82574L Gigabit Ethernet. FreeNAS is electing to use the Intel Pro/1000 driver
Seems like this ^^ has to be the issue, but why would it care, know, or manifest in WRITES and not READS? I would expect a faulty NIC or network issue to manifest in both types of operations (unless it was related to the network application layer, which I've ruled out).
-> Local Writes are Very Fast (so it's unlikely the disks, controller, server I/O)
-> Network WRITES are Very Slow (consistently exactly 11MB/s), Regardless of Network Protocol (scp, NFS, smb, ftp).
-> Network READS are just fine (>90MB/s).
These three data points are very difficult to reconcile IMO. I should add that Network WRITES via the localhost loopback are just fine.
So, it has to be something to do with the network, but what network issue manifests only in writes versus reads irrespective of the network application used?
I'm not expecting a solution, just a critical evaluation of the above details and some direction of where to go or look next. The usual suspects all seem to be precluded by one of the above data points, and syslog is bingo (at least at default verbosity).
Thanks in Advance for anyones time and consideration,
-Michael
Scenario Details Hardware Details:
FreeNAS-11.2-U6 (Issue also was manifest with same hardware on 9.3)
Supermicro X8S Ti-F w/Xeon E5504 & 12GB ECC RAM
Western Digital 2TB "Blue" Drives qty.4 on Integrated Intel ICH10R SATA Controllers in RAIDZ2 with lz4
Again, Very Fast Local Writes seem to preclude the above ^^from having issues or being the culprit.
Network Details:
Using a single Integrated Intel 82574L Gigabit Ethernet. FreeNAS is electing to use the Intel Pro/1000 driver
Seems like this ^^ has to be the issue, but why would it care, know, or manifest in WRITES and not READS? I would expect a faulty NIC or network issue to manifest in both types of operations (unless it was related to the network application layer, which I've ruled out).