jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
(idly wonders wtf this would be)
-
Important Announcement for The TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Poor performance on 10g network
- Thread starter egghog
- Start date
(idly wonders wtf this would be)
Sorry, striped z1. Still sorting out all the terminology
"1TB Samsung 980 Evo Pro"
Do you mean 890 EVO Plus? As soon as the SLC cache fills, the drive writes will slow down dramatically as it becomes a TLC drive.
For video editing, I'm wondering how fast a drive you need. 4K video only uses about 25-Mb/sec on the wire or 3.2-MB/sec. Have you tried editing using that SMB share?
Do you mean 890 EVO Plus? As soon as the SLC cache fills, the drive writes will slow down dramatically as it becomes a TLC drive.
For video editing, I'm wondering how fast a drive you need. 4K video only uses about 25-Mb/sec on the wire or 3.2-MB/sec. Have you tried editing using that SMB share?
No I mean 980 pro, but I imagine the principle is the same
That figure might be for compressed video or proxies. Except I'm actually not an editor but a VFX artist and don't work with proxies. The footage I work with easily surpasses 300MB/s, and this also doesn't include stock elements and other assets that need to be loaded.
I'm obviously not too concerned with realtime playback, but optimizing bandwidth and removing as many bottlenecks as possible is still important in my workflow.
You will have the same storage efficiency with two 4 drive RAID Z1 Striped. Either way it's 8 drives 6 data drives and two parity drives. I have not tried with two 4 drive RAID Z1, only 3 yet it should be faster as long as you don't run out of CPU. Your i7-4930k is a bit slower than the i7-7820X I'm using.
Also, are you using TrueNAS Core or Scale? If you are using Scale, take a look at the latest benchmarks for sequential access. The last I looked Core was faster.
But with a strip of two raidz1 vdevs, I can only lose one drive in each vdev vs. any two drives in the single vdev, wouldn't this make it less safe in terms of drive failure?
Another thought just occurred to me: Since my array is made up 4 of these old drives in addition to the 4 new iron wolves, is it possible that they're the cause of the bottleneck? Could they have degraded over time somehow?
edit: Also to answer your last question I'm using TrueNAS Core
Last edited:
Yes the risk is higher yet do the math. The drives you are using have 5 year warrantees which indicates the manufacture expects them to last longer than this and even as they age, they will not hit a day that they all fail at the same time. In my 30 year IT career I witnessed 2 duel drive failures in an array of 8 or more drives. In both cases, the second drive failure was caused by a human pulling a good drive out of an array when trying to replace the bad drive. If you are worried, add a hot spar to allow for a faster rebuild and to avoid a human error during the time the array is at risk.
Important: If you are storing important data on your array, then it must be backed up as no form of RAID is backup. You must have at least two copies of your data at any time. To meet data protection standards, it should be 3 copies, original and two on sight backups and one off site backup.
While they may not spin at the same speed as the new drives, the memory buffers will keep them all busy whiteout delays. Any drive errors will report and you will get an alarm as a warning.
Good
Take a look at the CPU usage when you copying to the array via SMB. If you have very high CPU, then you have the issue I'm concerned about. If not, then it's something else.
The striped raidZ1 solution is very interesting. Does this essentially mean you have the combined performance of two raidZ1s? So 4x read and 2x write compared to a single disk
I don't currently have anywhere to move the data, so at the moment this is not an option for me to try. But of course I am planning on backing it up. I'll probably start with a cloud backup before I actually build another server.
My CPU usage on TrueNAS doesn't really go above 5% when reading or writing.
I don't currently have anywhere to move the data, so at the moment this is not an option for me to try. But of course I am planning on backing it up. I'll probably start with a cloud backup before I actually build another server.
My CPU usage on TrueNAS doesn't really go above 5% when reading or writing.
Update: I noticed something interesting! When copying certain folders from one SMB share to another (through windows), the write speed is in excess of 1GB/s. It varies wildly from file to file, with folders containing many small files naturally taking longer than fewer large files.
I assume since the copy is going through windows, that confirms that the network is not the issue at hand? Or is it smart enough to bypass windows and just copy directly to itself? Somehow I doubt it.
Meanwhile, copying these same folders to my other drives results in the same sluggish speeds I've been getting so far.
I assume since the copy is going through windows, that confirms that the network is not the issue at hand? Or is it smart enough to bypass windows and just copy directly to itself? Somehow I doubt it.
Meanwhile, copying these same folders to my other drives results in the same sluggish speeds I've been getting so far.
The low CPU you are experiencing is different than what I experienced. With 6 drives in RAIDZ2 I was at 98% CPU when copying from windows to TrueNAS across the 10-Gb link. G drives in RAIDZ1 also very high CPU. Lots of googling and I found the striped RAIDZ1 solution and now my CPU is in the 25% range and I can write as fast as I can hit about 6-Gb/s on the 10-Gb link. I was barley able to go above 1-Gb with RAIDZ2. The post I found long ago showed the same as I'm seeing. I don't know what your bottle neck is yet it is holding back the CPU and throughput.
As for copies from shares on the NAS, I don't think the data is going over the LAN. You can verify this in Task Manager. I was surprised that TrueNAS emulates Windows for this.
What kind of throughput do you get when you copy large files from one disk to another on your Windows computer?
As for copies from shares on the NAS, I don't think the data is going over the LAN. You can verify this in Task Manager. I was surprised that TrueNAS emulates Windows for this.
What kind of throughput do you get when you copy large files from one disk to another on your Windows computer?
Looks like you're correct about the LAN. The ethernet barely changes above 32Mbps when copying from one SMB to another despite showing 1.6GB/s transfer. Pretty cool.
I get about 1.6 GB/s when copying a 10GB file between my two NVME disks. Compared to 170MB/s when writing to NAS and 230MB/s when reading from NAS.
Could it be a lack of memory? I thought 32GB would be sufficient, but I know they recommend 1GB per TB so maybe new memory is in order.
I get about 1.6 GB/s when copying a 10GB file between my two NVME disks. Compared to 170MB/s when writing to NAS and 230MB/s when reading from NAS.
Could it be a lack of memory? I thought 32GB would be sufficient, but I know they recommend 1GB per TB so maybe new memory is in order.
I only have 32 GB of RAM and and 11 GB is taken up by services, mostly VMs so I've only got 18.5 GB for ZFS Cache. I have not done a throughput test in a while so I just copied to the 2 x 3-disk RAIDZ1 at 500 MB/Sec and then read it at 600 MB/Sec though the read varied between 500 and 600. CPU during the write varied 20% to 30%. The V13 uses much less CPU than previous versions.
Do you have any small disks that you could create a test pool with? If you don't have the ports, you could disconnect the production pool.
Do you have any small disks that you could create a test pool with? If you don't have the ports, you could disconnect the production pool.
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
The ZFS write cache is your main system memory, and what happens here is that as writes occur, a transaction group (txg) is built up and staged to be flushed out to the pool. While not completely accurate, it is easiest to think of this process as there being two active transaction groups at any time, one building up and one actively being flushed out to the pool.
A lot of work has been done on the txg mechanism in the last decade, so it's harder to make generalizations today. Back then, it was easy to figure out the rules of the game, such as that the write cache was limited to 1/8th of system memory.
It was easy to point to that and to explain that failure to have sufficient memory resulted in a smaller write cache and smaller txg's. However, write performance isn't just a function of the size of the write cache. It also has a lot to do with fragmentation on the pool. As your pool fills, ZFS has to do a lot more searching through metadata to find free space for you. If it has to pull metadata off the pool in order to process it, then that's super slow. You want that to already be easily available in ARC or L2ARC. So these are two separate reasons why you want to be liberal with the amount of memory you give your filer. ZFS suffers when there's too little, and ZFS excels when there's excess, so giving it too much memory rarely hurts.
Thanks for the info, understood, I'll get some more RAM, and possibly another drive for L2ARC.
Separate thought: I've been looking at my motherboard, and it seems like it uses a number of different storage controllers for its SATA connectors. Two of the controllers are 6Gb/s, one is 3Gb/s. So in total there are 4x SATA2 connectors and 5x SATA3 connectors. Could this be the source of the bottleneck? I initially purchased a 10 port SATA card but then returned it as I didn't think I'd need it. But maybe I should give it a try.
Also, I ran iperf3 and finally got a speed of 9.34Gb/s both ways. So that officially rules out the network. I feel like I'm closing in on this!
Separate thought: I've been looking at my motherboard, and it seems like it uses a number of different storage controllers for its SATA connectors. Two of the controllers are 6Gb/s, one is 3Gb/s. So in total there are 4x SATA2 connectors and 5x SATA3 connectors. Could this be the source of the bottleneck? I initially purchased a 10 port SATA card but then returned it as I didn't think I'd need it. But maybe I should give it a try.
Also, I ran iperf3 and finally got a speed of 9.34Gb/s both ways. So that officially rules out the network. I feel like I'm closing in on this!
- Joined
- Apr 16, 2020
- Messages
- 2,947
Given an HDD is 150MB/s (on a good day) = 1.2 Gbps then a HDD cannot flood a 3Gb/s SATA port. Not for any serious amount of time anyway.
What 10 Port SATA Card - these are usually (almost certainly) cheap nasty expander cards that will destroy your data and are highly not recomended . I guess is PCIe x1 (they usually are). Do not use it.
If you need more ports then get a proper LSI card.
There are some specific chipsets on SATA expansion cards that do work properly. I don't remember what they are but they tend to be at most 4 ports on a card, not 10. Even then most 4 port cards are crap.
@jgreco has mentioned some chipsets that can work in other threads I think
What 10 Port SATA Card - these are usually (almost certainly) cheap nasty expander cards that will destroy your data and are highly not recomended . I guess is PCIe x1 (they usually are). Do not use it.
If you need more ports then get a proper LSI card.
There are some specific chipsets on SATA expansion cards that do work properly. I don't remember what they are but they tend to be at most 4 ports on a card, not 10. Even then most 4 port cards are crap.
@jgreco has mentioned some chipsets that can work in other threads I think
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Gaaah, don't do that. See
Multiply your problems with SATA Port Multipliers and cheap SATA controllers
In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...
www.truenas.com
So I'll just remind you that even 3Gbps works out to about 375MBytes/sec, which is far faster than any HDD out there. However, if your mainboard ports happen to be supplied by a port multiplier, then you have problems.
If you're doing this on Core, I suggest trying out the solnet-array-test-v2 script to see if it can identify any performance issues.
While I'm waiting on more RAM, I ran an IOzone test. However I'm not sure how to interpret the results. Would anyone be willing to explain it?
Also I was informed that the 4 older drives in the NAS are PMR. Could mixing drive types cause this dramatic of a slowdown?
Also I was informed that the 4 older drives in the NAS are PMR. Could mixing drive types cause this dramatic of a slowdown?
