10G Network Upgrade | Next Steps?

[schoolpost]

Hello,

I'm in the process of upgrading my setup to use 10G networking, added 2 X520-DA1's to both my TrueNAS Server and my Workstation.

Noticeable improvement in speeds over 1G; into 200-300MB/s territory quite often with transfers using SMB, however I wasn't aware until now that my ZFS POOL seems to be slower than I would anticipate, but I hadn't noticed it because it was still fast enough to saturate a 1G link.

The Primary pool is a RAIDZ3 with 10x4TB drives. ( it is a mix of various drives, 4x WD RED / 2x WD GOLD / 3x IRONWOLF / 1x BARRACUDA )
( rest of the system specs can be found in the footer. )

I realize Z3 is not going to be the most performant configuration, given the parity process for 3 drives. However I do feel like the I "should" be closer to a 400-600MB/s ballpark? I could be very be wrong about this, so that is where I'm looking for some opinions and thoughts about the setup.

The Pool is 53% full and the fragmentation is at 9%, wondering if there is anything I can configure to improve the speed. I've seen mentioned that it can helpful to recreate the backup/delete/recreate the pool from scratch.

 

[Arwen]

If I am reading your specifications correctly, you have 4 x WD Red SMR drives. SMR drives are both slower for write and because of the fragmentation possible, reading can be slower than expected. Further, drive replacement with SMR drives is much slower. The upshot is that SMR drives are not recommended for use with ZFS.

Suggested reading about SMR drives:

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

Update: WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we uncovered and reported on a ZFS compatibility issue with some capacities (6TB and under) of WD Red drives that use SMR (Shingled Magnetic Recording) technology. Most HDDs use CMR (Conventional Magnetic Recording) technology which works well with ZFS. Below is...

www.truenas.com

WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we have uncovered a potential ZFS compatibility issue with some capacities of newer WD Red drives that use SMR (Shingled Magnetic Recording) technology. Update to this post...

www.truenas.com

Further, you don't list the Seagate Barracuda model. But, it appears all 4TB models, both 2.5" & 3.5", are SMR as well.

That said, I don't know if that is your "whole" problem.

 

[schoolpost]

If I am reading your specifications correctly, you have 4 x WD Red SMR drives. SMR drives are both slower for write and because of the fragmentation possible, reading can be slower than expected. Further, drive replacement with SMR drives is much slower. The upshot is that SMR drives are not recommended for use with ZFS.

Suggested reading about SMR drives:

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

Update: WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we uncovered and reported on a ZFS compatibility issue with some capacities (6TB and under) of WD Red drives that use SMR (Shingled Magnetic Recording) technology. Most HDDs use CMR (Conventional Magnetic Recording) technology which works well with ZFS. Below is...

www.truenas.com

WD Red SMR Drive Compatibility with ZFS

Thanks to the FreeNAS community, we have uncovered a potential ZFS compatibility issue with some capacities of newer WD Red drives that use SMR (Shingled Magnetic Recording) technology. Update to this post...

www.truenas.com

Further, you don't list the Seagate Barracuda model. But, it appears all 4TB models, both 2.5" & 3.5", are SMR as well.

That said, I don't know if that is your "whole" problem.

Thanks for the suggestions, I've been extra careful about the whole CMR/SMR controversy with the WD RED's from a while back; my system only uses the CMR models ( WD40EFRX )

As for the Seagate Barracuda, I've been meaning to swap that one out for a WD RED that I have had in my components drawer as a "cold spare" ; figured I wouldn't force a resilver prematurely until a drive would actually fail. Should I swap that drive anyway?

 

[Arwen]

Glad your WD Reds are CMR.

As for swapping out the Seagate Barracuda, your call. If you do performance tests before and after, that should tell you if that drive being a SMR one, caused performance problems.

 

[Elliot Dierksen]

Performance tuning is always a whack-a-mole kind of thing. When you speed up one component, it can reveal weakness in another component. I would encourage you to use iperf to test the network speed without using any I/O. If you aren't getting 90-95% of the available link speed with iperf then there may be some other components slowing you down. fio is the tool of choice for testing storage. I am not up enough on it to suggest what options to use with fio, but I am sure you could find that in some other forum posts.