Poor copy/rsync performance from CMR to SMR drives

[TheWalkingEggplant]

Hello everyone!

I am new to TrueNAS user (I just moved from OMV6) so I kinda need help to understand how it works.

I know that there are some threads talking about this topic, that SMR drives are not completely suitable for TrueNAS and rsync/cp performs poorly with them. However, most of these threads only talk about transfer data over the network and not really about local transfer (with rsync/cp) and I didn't have a clear answer to my problem.

Here is what I did to keep data from OMV6 to TrueNAS Scale:

1. I created one pool with my 18TB to ZFS using zpool command in Ubuntu
2. I rsync-ed all data (14TB) from my 6x 2.5" 5TB drives to the pool newly created (still using Ubuntu OS, because Ubuntu is capable of reading ext4 and zfs)
3. Once done, I installed TrueNAS Scale
4. I created another pool with my 6x drives with Raid Z2 configuration
5. Finally, I copy/transfer (with rsync/cp command) data back from my 18TB pool to my Raid-Z2 pool

What I don't really understand is that the data transfer in step 2 was pretty fast, it took like 3 days to finish copying all the data to the 18TB pool but the data transfer in step 5 took much longer time like after 3 days, only ~400GB data were transfered.

Could anyone help me to understand why there is a big difference between these two performances?
Again, I know there are threads talking about this, but I don't really understand why rsync between pools (step 5) start with around 200MB/s for the first files but then decrease drastically, the speed stays around 7MB/s in average.
What could I do to improve the transfer in step 5?


Thank you very much for your helpü

---

TrueNAS Scale System
TrueNAS-SCALE-22.12.0
Intel Xeon E-2224G 3.50GHz
Asus WSC246 Pro S1151 ATX
3 x 16GB DDR4 - 2666MHz Corsair Vengeance LPX
Bequiet! PurePower 11 300W

6x 5TB 2.5" SMR drives (Seagate, WD) (Raid Z2 Configuration)
1x Toshiba 18TB CMR - MG09ACA18TE (Stripe Configuration)

 

[Ericloewe]

If you're not being naïve, writes are easier to optimize. This applies both to ZFS and to SMR drives - and even SSDs.

If you're just dumping data, the disk doesn't have to worry about seeking around to put everything in a specific place (that's ZFS's work). The disk is also not incurring SMR penalties because it doesn't need to overwrite anything.

When you read your data back, you're stuck going through the metadata and chasing down data in whatever order the application is asking for it. Since rsync has no visibility into how to optimize this, it doesn't. So you get very slow reads and no help from ARC because you're always moving on different data.

 

[TheWalkingEggplant]

If you're not being naïve, writes are easier to optimize. This applies both to ZFS and to SMR drives - and even SSDs.

Could you guide me on how to do that? I am good at linux in general either ^^'

 

[sretalla]

OK, I think there's a disconnect somewhere here...

Step 2 is writing data from some kind of RAID (presumably set up in OMV) to a single disk ZFS pool... importantly, a CMR one.

The CMR disk has no problem writing the data as it doesn't have to do the crazy shuffling required of DM-SMR disks, where writes first go to a cache area (which isn't SMR) and then get written at a later point to the SMR part (managed my the drive itself, hence Drive Managed (DM) SMR).

Because the writes you're doing in step 5 are causing large numbers of writes to DM-SMR disks, you'll quickly be reaching the limits of the CMR part of the drive, forcing the drive to manage that out to the SMR part in addition to accepting the continuing stream of more data coming in... exactly why DM-SMR disks are often kicked out of ZFS pools since they kind of stop responding when the writes can't be put somewhere quickly enough.

I don't think there's:

a. A way of making it go faster with the hardware you have.
b. Any getting around the limitations of SMR disks with ZFS other than doing your copy in small increments and waiting for the drives to catch up.

You're doing something specifically recommended against by this forum in using those SMR disks with ZFS, so don't expect a lot of other help here, but I do wish you good luck in keeping your data (make sure you have backups).

 

[TheWalkingEggplant]

OK, I think there's a disconnect somewhere here...

Step 2 is writing data from some kind of RAID (presumably set up in OMV) to a single disk ZFS pool... importantly, a CMR one.

The CMR disk has no problem writing the data as it doesn't have to do the crazy shuffling required of DM-SMR disks, where writes first go to a cache area (which isn't SMR) and then get written at a later point to the SMR part (managed my the drive itself, hence Drive Managed (DM) SMR).

Because the writes you're doing in step 5 are causing large numbers of writes to DM-SMR disks, you'll quickly be reaching the limits of the CMR part of the drive, forcing the drive to manage that out to the SMR part in addition to accepting the continuing stream of more data coming in... exactly why DM-SMR disks are often kicked out of ZFS pools since they kind of stop responding when the writes can't be put somewhere quickly enough.

I don't think there's:

a. A way of making it go faster with the hardware you have.
b. Any getting around the limitations of SMR disks with ZFS other than doing your copy in small increments and waiting for the drives to catch up.

Thank you for your reply! That makes more sense to me now.

You're doing something specifically recommended against by this forum in using those SMR disks with ZFS, so don't expect a lot of other help here, but I do wish you good luck in keeping your data (make sure you have backups).

I know I should use CMR drives. I just want to reuse the old drives and it would cost me alot to buy another CMR and throw the rest away.

 

[Ericloewe]

Could you guide me on how to do that? I am good at linux in general either ^^'

I didn't mean you my present interlocutor, I meant abstract developer "you".

What you could have done is try ZFS replication. It may have been faster, but I'm not going to guarantee it. Don't expect monstrous improvements, though, SMR has broadly been worse than many expected in real scenarios.

 

[TheWalkingEggplant]

Oh, thank you for your suggestion :)

 

[ChrisRJ]

I know that there are some threads talking about this topic, that SMR drives are not completely suitable for TrueNAS and rsync/cp performs poorly with them. However, most of these threads only talk about transfer data over the network and not really about local transfer (with rsync/cp) and I didn't have a clear answer to my problem.

I would rather characterize SMR drives as utterly unsuitable for ZFS. The resilvering (test done by Serve The Home/STH?) took more than a week with an SMR drive, compared to less than 24 hourse with CMR.

I don't see how local vs. network is relevant here.

 

[TheWalkingEggplant]

I would rather characterize SMR drives as utterly unsuitable for ZFS. The resilvering (test done by Serve The Home/STH?) took more than a week with an SMR drive, compared to less than 24 hourse with CMR.

Thank you for the information!

I don't see how local vs. network is relevant here.

I mentioned it to avoid people supposing that it was because of my network congestion (in some posts that I saw). Other than that, you are right, it doesn't matter at all.

 

[olddog9]

Related to this topic ...of hard disks and CMR versus SMR ... there is a new post on Slashdot.org about 'How Long Do Hard Drives Last ?".
The post refers to this blog by Secure Data Recovery. Their blog tells of 15 years experience recovering hard disk data and opinions formed.
I don't know if their tests are statistically valid, however the conclusions presented appear valid (to me).

How Long Do Hard Drives Last

Want to learn more about hard drive lifespans and failure rates? Discover how long hard drives last with unique data from Secure Data Recovery.

www.securedatarecovery.com

Some of the more interesting statements are ....
"Thus, disks with CMR appear more durable and resilient than those with SMR.
And, in general, old drives seem more durable and resilient than new drives."
...and ...
"But developing a comprehensive plan and maintaining regular backups is essential regardless of manufacturer or model.
Our experts recommend following the 3-2-1 Rule: Keep three copies of important data on two devices, with one of those copies saved offsite."

...and the SCARIEST statement of all is ....

"Despite that, almost half of our recent survey’s respondents admitted they did not have backups."

 

[Ericloewe]

Counterpoint: people tasked with recovering data from dead disks lack the dataset to draw meaningful conclusions about non-dead disks.

It's like the guy who worked at a depot for knocked-out Shermans and claimed they were a death trap. Of course, the data completely contradicts him.

 

[winnielinnie]

Counterpoint: people tasked with recovering data from dead disks lack the dataset to draw meaningful conclusions about non-dead disks.

Anti "survivorship bias"?

(I'm sure there's a formal term for it, but I'm too lazy to search right now.)

 

[WI_Hedgehog]

Counterpoint: people tasked with recovering data from dead disks lack the dataset to draw meaningful conclusions about non-dead disks.

It's like the guy who worked at a depot for knocked-out Sherman's and claimed they were a death trap. Of course, the data completely contradicts him.

My liver has far outlasted medical estimates, however I rarely go to the doctor due to my excellent state of health. Plus I find drinking solves all the problems drinking creates, so I have virtually no stress, thereby coming out ahead of non-drinking people, medically speaking.

Similar with CMR vs. SMR. SMR stores more information but stresses itself out doing so which leads to an early grave. As-is, CMR densities are so dense the S.M.A.R.T. system often has to do a sector re-write on geriatric data. I can only imagine how much worse this is with the high densities of SMR.

 

[Ericloewe]

I don't expect (opinion alert!) SMR to make a difference, with the possible exception of mechanical health of the actuators, all else being equal. However, not all else is ever equal.

The ones that come closest to reporting useful info are Backblaze, but their data is bizarre (who reports number of disk failures without accounting for age of the disks!?) and difficult to work with. Makes for lots of low-effort headlines though, which is why it's so popular.

 

[Whattteva]

I don't expect (opinion alert!) SMR to make a difference, with the possible exception of mechanical health of the actuators, all else being equal. However, not all else is ever equal.

The ones that come closest to reporting useful info are Backblaze, but their data is bizarre (who reports number of disk failures without accounting for age of the disks!?) and difficult to work with. Makes for lots of low-effort headlines though, which is why it's so popular.

I think their most recent report does account for age though it still only has 5 years of data and a specific use case (boot drives). Also, I think their business model also means they're using cheap consumer drives for both types.

New Backblaze Data Confirms SSDs Are More Reliable Than Hard Drives

No moving parts really is more reliable than spinning disks

www.tomshardware.com