Slow copy/transfer speed in TrueNAS

[pawel30w]

Hi,

I am looking to understand what should be expected speeds of transfer inside the dataset and between two datasets (on two difeerent pools). It looks like my speeds are quite low and can't find the reason why.

Specs:
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Supermicro x10SRi-F
3x 4TB WD Red (...EFAX ones, SMR's)
2x 4TB Segate Ironwolf
Seasonic GC-650

I've just build the TrueNAS box and I am playing with it to understand it's behaviour before I will put there any data for good. I have created the RAIDZ2 Pool with all 5 drives.

I copied 200GB file from my PC to TrueNAS through cable (stable 110 MB/s). Then I took this file and created copy of it inside the same folder (the same dataset and pool). I had stable copy speed around 130-140 MB/s. This was using Windows to copy. I repeated this excercise with 'cp' command inside the Shell and copy speed were the same. Those are pretty crappy speeds and I thought it may be these SMR 4TB WD Reds that are messing with the whole pool.

To understand if that's the case, I destroyed the 5 disc pool and created three smaller pools. 2x WD Reds pool (mirror), 2x Segate pool (mirror), 1x WD Red pool.

I did simmilar excercise. I copied 200GB file inside each pool and also from one mirror to second mirror. I connected TrueNAS to monitor and used Shell and 'cp' command. Speeds are as below:
- Inside WD Mirror - 87 MB/s
- Inside Segate Mirror - 71 MB/s
- Inside WD Single - 81 MB/s
- WD mirror to Segate Mirror - 153 MB/s
- Segate Mirror to WD Mirror - 138 MB/s
- Segate Mirror to WD Single - 138 MB/s

I'd like to think that this excercise excludes any issues with Windows because I used Shell and exclude any issues with heterogeneous pool as I tested mirror pools as well.

Please see below output from 'dd' command. I dont really understand how this command works so let me know if better to use different chunks or amounts.

root@truenas[~]# dd if=/dev/zero of=/mnt/Segate/Segate/ddfile1 bs=2048k count=10000 10000+0 records in 10000+0 records out 20971520000 bytes transferred in 4.239338 secs (4946886038 bytes/sec) root@truenas[~]# dd if=/dev/zero of=/mnt/WDmirror/WDmirror/ddfile1 bs=2048k count=10000 10000+0 records in 10000+0 records out 20971520000 bytes transferred in 4.149101 secs (5054473098 bytes/sec) root@truenas[~]# dd if=/dev/zero of=/mnt/WDsingle/WDsingle/ddfile1 bs=2048k count=10000 10000+0 records in 10000+0 records out 20971520000 bytes transferred in 4.051045 secs (5176817903 bytes/sec)

What else I can do to check where is the issue. I would expect that 'inside' the pool I would get sth around 800-1500MB/s. Am I being naive?

Thanks,
Pawel

 

[Alecmascot]

3x 4TB WD Red (...EFAX ones, SMR's)

This is not a good idea.....

 

[pawel30w]

This is not a good idea.....

Well, unfortunately I can't return them. I am going to do stress them and resilver each one of them with 90% full pool.

It will be RAIDZ2 and additional backup for the most important data so should be fine

 

[HoneyBadger]

I am going to do stress them and resilver each one of them with 90% full pool.

This will end badly.

I don't have a gentle way to say this, but SMR drives, especially the WD Reds, are a bad choice for your pool to the point where there is a risk of losing the whole thing. Selling them and purchasing other drives should be on your radar if returning is not possible.

 

[pawel30w]

This will end badly.

I don't have a gentle way to say this, but SMR drives, especially the WD Reds, are a bad choice for your pool to the point where there is a risk of losing the whole thing. Selling them and purchasing other drives should be on your radar if returning is not possible.

I think I've read half of the internet about the issue (after I bought them unfortunately).

As far as I understand the SMR's may have long latency/read time. If this latency will be to long zfs may mark disk as a unreadable. If I test/resilver them I can see if there is issue or not. Why would the resilver be such a bad idea? It's not destructive process

I mean, I can return them if something would be wrong with them but can't return with just saying "well, I bought wrong ones". If resilver would create bad sectors then I'm fine.

 

[HoneyBadger]

Simple process.

Let's say you have a Z2 vdev.
If a drive takes too long to return an I/O, ZFS will mark it as failed. You're then down to 1-drive failure tolerance.
Resilver starts.
Another drive "fails."
Now you're down to 0-drive tolerance.
Resilver continues.
Another drive fails.
Zpool is now unavailable.

It's not guaranteed, but filling SMR drives and then rewriting them is exactly what provokes them to have long I/O times.

 

[pawel30w]

I get your point, and understand you are right, but I am trying to weigh the risk of those failures vs cost and time of getting new drives Well, I'll need to do some more research and then decide.

In the meantime, going back on the topic of this thread, any thoughts on the described issue?

 

[HoneyBadger]

In the meantime, going back on the topic of this thread, any thoughts on the described issue?

"Performance as expected" generally speaking. A five-drive Z2 will trend towards a single drive under a mixed workload like that, so 130-140MB/s seems correct. The drive-to-drive copies also align with expected values.

The dd tests are skewed because you're using /dev/zero as a source and compression is enabled, so ZFS will compress the zeroes down to nothing. You'll have to use actual data or disable compression on the dataset to get an accurate result.

 

[Chris Moore]

In the meantime, going back on the topic of this thread, any thoughts on the described issue?

Many of these pool performance testing and results questions have been asked and answered in the forum many times over the years. As @HoneyBadger said, you are seeing results as you should see. The reason for having multiple vdevs in a pool is to improve performance because the performance of a single vdev is roughly equivalent to a single constituent disk in the vdev. The capabilities of the pool is additive of the constituent vdevs. You do loose some performance to overhead, but more vdevs in the pool give more performance and that has been reiterated many times over the years. You can find a lot of data on this by searching the forum.

Circling back to the SMR issue, they will slow the performance of your pool. That is how I initially found that there was a problem to begin with. I had integrated two SMR drives into a pool that had ten other 'standard' drives in it (two vdevs of six drives) and the performance was so much less than an identical system that had all 'standard' drives that I started searching for what the problem was. SMR drives don't just make your system more likely to completely fail, they also make it slower, so if you have a need for speed, you want to get those drives out of your system.