I upgraded 4 FreeNAS servers recently, to 11.3 newest release, and it appears all are having issues. Issue affects all flash server where I have raidz2 of NvME’s, raidz2 of 2.5” ssd’s, striped mirrors of 2.5” ssd’s, and it also happens on mech hdd pools some of which are striped mirrors and some of which are raidZ3.
SMB transfer will start, halt for at least 30 seconds, sometimes up to 1+ minute. This happens on every server. In years of using FreeNAS, I’ve never seen this happen.
All servers are well capable, spec-wise. Supermicro x10 and x11 servers, all with at least dual 16 core xeon’s, all servers have over 256gb RAM. 40 or 100gbe Mellanox NIC’s.
server tuning was set initially, 2 years ago or so. All have functioned without issue over SMB, NFS, ISCSI. I’m also having generally slow performance overall.
when transfers don’t play this halting game, I’m seeing widely variable transfer speeds. Sometimes, I’m seeing a 50 megabyte per second cap, transferring large sequential files which are compressed .rar or .7z. Occasionally these same file transfers are crawling at like 5 megabytes per seconds, even in a brand new pool with new Seagate Exos SAS drives.
Also, for instance on an all flash array, I’ll see writes for through at normal speed, about 550 megabytes per second for 2.5” ssd z pools, 2 gigabytes per second + for nvme or striped mirror, and about 1.2-1.5 gigabytes per second for striped mirrors on an hdd array, which is normal. I’ll see a couple gigabytes written, then a long pause/halt, then, it typically resumes at pace.
so, here we’ve got the halt problem, and the wildly variable SMB performance.
i can’t test NFS yet, because I’m getting weird permission errors on all my NFS shares.
ISCSI seems unimpacted.
my assumption is software related, because the problem exists across 4 servers, all which were working perfectly prior to the updates.
any suggestions on how to fix?
SMB transfer will start, halt for at least 30 seconds, sometimes up to 1+ minute. This happens on every server. In years of using FreeNAS, I’ve never seen this happen.
All servers are well capable, spec-wise. Supermicro x10 and x11 servers, all with at least dual 16 core xeon’s, all servers have over 256gb RAM. 40 or 100gbe Mellanox NIC’s.
server tuning was set initially, 2 years ago or so. All have functioned without issue over SMB, NFS, ISCSI. I’m also having generally slow performance overall.
when transfers don’t play this halting game, I’m seeing widely variable transfer speeds. Sometimes, I’m seeing a 50 megabyte per second cap, transferring large sequential files which are compressed .rar or .7z. Occasionally these same file transfers are crawling at like 5 megabytes per seconds, even in a brand new pool with new Seagate Exos SAS drives.
Also, for instance on an all flash array, I’ll see writes for through at normal speed, about 550 megabytes per second for 2.5” ssd z pools, 2 gigabytes per second + for nvme or striped mirror, and about 1.2-1.5 gigabytes per second for striped mirrors on an hdd array, which is normal. I’ll see a couple gigabytes written, then a long pause/halt, then, it typically resumes at pace.
so, here we’ve got the halt problem, and the wildly variable SMB performance.
i can’t test NFS yet, because I’m getting weird permission errors on all my NFS shares.
ISCSI seems unimpacted.
my assumption is software related, because the problem exists across 4 servers, all which were working perfectly prior to the updates.
any suggestions on how to fix?
