TrueNAS Server performance not quite what is expected, any tips?

[maci]

Dear TN community, thanks to your help I was able to put up a decent server for my storage & media needs. Once everything was ready I ran some tests but the transfer speeds are not as high as I expected. I did various different transfers to and from the server PC to my main machine. Copying files onto the server usually hovers around 220-300MB/s. Copying from the server to my main machine is similar, although some odd ones were faster around 600-700MB/s, one even reaching 2GB/s. However, what I'm curious about is the average transfer (220-300MB/s). I ran tests through TrueNAS' shell and iperf3 on my pc. Several tests showed stable Read and Write speeds both as sender or receiver at 600-700MB/s (5-6Gbit/s). Now, when copying files onto my main PC it stays fairly consistent whether thats an internal NVMe, SSD, HDD or external SSD. Some differences of course, but fairly within the range of 220-300MB/s. Here are my specs:
1. Server PC:
-Intel i5 3750k
-Gigabyte GA-z77-ds3h
-16GB RAM @ 1333MHz
-LSI 9211-8i SAS HBA - Dell H310 - IT Mode --> Connected to 4x 8TB Seagate Barracuda 5400rpm drives via forward sata breakout cable
-Intel X520-DA2 Dual port 10GB SFP+ --> x2 of the same card connected with 10Gbit SFP+ cable
-Remaining 2x 8TB Seagate Barracuda 5400rpm drives connected via SATA to MOBO (Total of 6x 8TB drives)
-Arranged in RAID Z1 6 Wide in 1 VDEV

During transfers all parameters seem fairly low, nothing seems to be getting overloaded, but I can run more specific tests and look for data if this would help you, just let me know which ones.

2. Main PC
-Intel 7700K
-32GB DDR4 Ram @ 3200MHz
-Intel X520-DA2 Dual port 10GB SFP+ --> same card as in server as mentioned above
-512GB NVMe M.2 Drive with OS
-1TB NVMe M.2 Drive currently empty

For any further details please ask, but I'd be intersted to know if there's any better way to set this up, or improve performance, specially considering tests show it's possible in it's current configuration. Ah! Just to mention, the files I work with are usually video, ranging from H.264 to ProRes 4444, RAW, EXRs, etc, however there are photo stills and other files as well that I want to keep in the server. Appreciate any imput! Thanks.

 

[LarsR]

If your HDDs really are Baracuda and not Baracuda Pro, they're SMR drives, which are known to have performance problems with zfs...

 

[maci]

If your HDDs really are Baracuda and not Baracuda Pro, they're SMR drives, which are known to have performance problems with zfs...

Could you elaborate on what SMR is? And to be honest is the only option I could afford, but if you've got any recommendations I still have the chance to return them. Do you know how might they perform in a different environment such as unraid?

 

[LarsR]

SMR and CMR are recording technologies for HDDs. What you want for zfs are CMR drives, like the seagate baracuda pro, ironwolf, ironwolf pro, or exos line. For WD drives you want the red pro series.
Never used unraid, but i've heard that it works better with smr drives (don't quote me on that).

For further information see https://www.truenas.com/community/threads/WD-SMR-iX-Statement/

 

[chuck32]

For WD drives you want the red pro series.

Starting at Red Plus WD features CMR drives. If you don't need the extra feature of the pro series, they are the cheaper alternative.

 

[maci]

Thank you both, yeah I did some more digging and I see the difference between CMR and SMR. I'm debating exchanging these seagate barracuda drives for the ironwolf NAS drives, although I don't see them listed on Seagate's page but I suppose they're part of the ironwolf series (https://www.seagate.com/gb/en/products/cmr-smr-list/). Something like this is my only option as all other choices are too expensive:

Seagate IronWolf 8TB, Internal, 7200RPM, 3.5" (ST8000VN004) Hard Drive for sale online | eBay

Find many great new & used options and get the best deals for Seagate IronWolf 8TB, Internal, 7200RPM, 3.5" (ST8000VN004) Hard Drive at the best online prices at eBay! Free delivery for many products!

www.ebay.co.uk

This would mean I'd have 5 drives on 7200rpm and one ironwolf I already have at 5400 but I saw it's not a huge issue. It really comes down to how much of a performance boost I could get with CMR drives oppoed to SMR, if the speed is small I might not bother. Any insight would be helpful! Thank you.

 

[probain]

You could go down to 5400RPM drives for the new ones as well.. As RPMs are mostly for seek times. And this where the ARC picks up the slack. You'll have a system that takes less power, and will therefore run cooler.

 

[maci]

You could go down to 5400RPM drives for the new ones as well.. As RPMs are mostly for seek times. And this where the ARC picks up the slack. You'll have a system that takes less power, and will therefore run cooler.

That's the problem, I can't find affordable 8tb CMR drives at 5400rpm

 

[ChrisRJ]

It is not easy

It really comes down to how much of a performance boost I could get with CMR drives oppoed to SMR, if the speed is small I might not bother.

No this is not the issue. SMR drives are completely unsuitable for ZFS. You run a considerable risk of data loss using them.

 

[ChrisRJ]

Just to mention, the files I work with are usually video, ranging from H.264 to ProRes 4444, RAW, EXRs, etc

If you want to use your NAS for just storing these files there that is fine. But if you want to edit videos directly off the NAS you will have a very unpleasant user experience. In other words the system is way too slow. You will effectively need an all-flash system with a fast CPU and loads of RAM. For more details there is a number of such threads on the forum.

 

[maci]

If you want to use your NAS for just storing these files there that is fine. But if you want to edit videos directly off the NAS you will have a very unpleasant user experience. In other words the system is way too slow. You will effectively need an all-flash system with a fast CPU and loads of RAM. For more details there is a number of such threads on the forum.

80% of its use will be just storing files, at the moment switching to CMR drives is too much of a gamble. I haven't done a full test on a proper project but trying some media out in my editing software the speeds were really good, pretty much as if I was editing from an external ssd as I have been doing until now. I can re-do my pool to have 2 redundant drives instead of one, but I'd lose 8TB which I'm not that excited about haha.

My server won't be in use all the time anyways, pretty much only when I use my computer and need to access the files.

 

[probain]

A really good writeup by Patrick at STH
Both a good article and companying video.
here

For me the main takeaway is how much longer it takes to resilver.

 

[ChrisRJ]

For me the main takeaway is how much longer it takes to resilver.

While that is certainly at least highly inconvenient and also increases the overall risk, should something happen during the now considerably longer resilvering, for me that is the smaller of two problems.

The real issue is that even during normal operation there is a risk with drive-managed SMR disks to loose data.

We have had at least case here in the forum where someone lost data because the SMR drive would need to long to remap and caused data corruption. We can certainly argue the probability of that. But on the other hand: If I am not really paranoid about my data, using TrueNAS makes no sense anyway. In that case something like unRAID is a better choice from where I stand.

Good luck anyway and I hope that my skepticism is not necessary here