WD Red SMR Drive Compatibility with ZFS

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Thanks but no thanks. Zoned writes make a lot of sense for SSDs and for specific applications in HDDs (Ie cold storage of large data sets with custom drivers and all that).

however, in ZFS Land with its general purpose application and still heavy reliance on HDDs, I am very skeptical. TRIM, garbage collection, etc. all have inherent costs in terms of performance and the impact is greater with HDDs than SSDs. This is even more acute with SMR due to the long time it takes to write out a zone.
 

Forza

Explorer
Joined
Apr 28, 2021
Messages
81
It could be used like that, but it has a few problems:
  • I think it doesn't support any standardized reporting beyond "this is a separate chunk", which leaves the problem of making the OS aware of the properties of said chunk.
  • It doesn't apply to HDDs.

That seems to be WD-exclusive. Seagate will have their own incompatible stuff, most likely. Until widely adopted, it's irrelevant for wide use.


Btrfs is hardly a ringing endorsement for anything.
You can choose to look backwards or forwards. SMR is on the market, even if you do not like it. Zoned support is coming and there are standards developed for it. I think that there are valid use cases for it, not just enterprise, but for for the millions of home users using NAS boxes too. Just because you don't want to use a particular feature set, it doesn't mean that it is a bad feature.

I did not mean to start a subjective fan war on Btrfs/ZFS or ZFS vs zoned. Just meant to look constructively on new technology.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
No one says it’s a bad feature. Eric pointed out that the oligopoly of HDD manufacturers has yet to zero in on a industry standard for that feature…. and until that day comes, do not expect wide-spread adoption by the likes of ixSystems. They have plenty of other new features and improvement opportunities to work on in the meantime.

As for SMR, it has its use case also, especially pools that will be written to infrequently (Ie cold storage). It’s a neat way to tickle an extra 20% of capacity out of an extant hardware platform, mostly via software. Have you seen the infamous OpenZFS presentation by Manfred Berger?

For me, the performance impacts of SMR in a TrueNAS do not even begin to justify the small discount that HDD firms sell SMR drives for. The OEM gets to drop a platter, some heads, etc. then sells the drive for 5% less, and I get to wait 4x longer for a pool to heal itself, random disk dropouts, and crummy, unexplained performance in general. No thanks.

For some types of home use or travel use, the extra 20% space via SMR can be interesting however - somewhat lower cost, responsiveness / performance doesn’t matter as much, etc. Leave the machine on overnight, and hopefully by morning the task is complete.
 
Joined
Jan 4, 2014
Messages
1,644
The majority of FreeNAS/TrueNAS drives I use are a mix of WD Red+ and WD Red CMR drives, though I have a small number of Seagate IronWolf drives thrown in the mix as well. The only reason I mainly use WD Reds is because of iXSystems recommendation. Given all the issues and grief around WD and SMR drives, I'm curious to understand why iXSystems still recommend WD as the preferred NAS drive supplier? Personally, I have no allegiance to any drive supplier, but if I look at Seagate, for instance, none of their IronWolf range of drives designed for NAS has ever been caught out as using SMR technology https://www.seagate.com/au/en/internal-hard-drives/cmr-smr-list/. The perceived blinkered and seemingly still endorsed view of iXSystems to WD is surprising in the light of what's happened. This attitude seems almost discriminatory to other drive manufacturers. Why is iXSystems still favouring WD over other NAS drive manufacturers?

EDIT: I had three WD Red SMR disks in the mix, the last which I replaced yesterday. It's been a painful and protracted process with the loss of a pool along the way. WD is 'happily' replacing these drives. I think they must be worried about multiple class-action suits. The catch is that I pay the postage to send the disks back. It's cost me AUD$57 to return disks with a total original purchase cost of between AUD$750 and AUD$900. I'm somewhat aggrieved that I'm still paying for WD's mistake. I wonder if I should join a WD class action suit down under?
 
Last edited:

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,110
iXSystems has a contract with WD. So, even if the message to users gets somewhat complicated—promote "WD Red Plus" (or "Pro") while avoiding unqualified "WD Red" at all costs—, iX Systems does not want to break ties and lose the sponsor's money.

As for joining a class action, the choice is yours…
Unfortunately, the mere asking for a qualified legal advice is likely to cost you much more than what you may recover.
 
Joined
Jan 4, 2014
Messages
1,644
As for joining a class action,
That was meant to come across as a rhetorical question. I didn't quite hit the mark on that one :wink: Thanks for your reply. Now that you've mentioned sponsorship, it all makes sense. Unfortunately, sponsorship can have undesirable side effects. Case in point. It forces an organisation to bend its principles, handle, with kid gloves, issues that can have a detrimental impact on the sponsor, and generally behave in a way it might ordinarily adopt a very different stance on. Maybe iXSystems should, on the quiet, explore future sponsorship opportunities with other NAS drive manufacturers like Seagate. There's nothing wrong with having a Plan B in place.
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Interesting, this is apparently just the first of several class action lawsuits. I am heartened that the courts saw through the dissembling baloney that WD served up in its defense. Especially that SMR drives should not be used in NAS applications. Now WD is court-ordered to disclose the recording technology used in its hard drives.

Tom’s hardware also mentioned Seagate and Toshiba sneaking SMR drives into their lineup without disclosing it. Presumably not in their NAS lines.

Dumb question: what happens with a DM-SMR drive whose power is disconnected mid-CMR to SMR cache transfer? (Think USB + power getting yanked on a portable). Presumably the next time the drive is plugged in, the drive re-attempts to transfer the cache until it succeeds?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I would like to imagine such transfers are done by atomically moving from one state to the other. Since we're talking about firmware, I fear that may be somewhat optimistic.
 

no_connection

Patron
Joined
Dec 15, 2013
Messages
480
Might be time to start question if WD is really a safe partner to work with.
If you have anything WD you better yank it from the wall if you still have time.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Might be time to start question if WD is really a safe partner to work with.
If you have anything WD you better yank it from the wall if you still have time.
WD is a large corporation.
The MyBook NAS devices are from a small team with a consumer focus. I assume they had an Internet connection and were attacked.
The WD Enterprise drives are the main Enterprise product... and have had very good quality for many years
The WD Reds had the SMR issue, which they acknowledge was a mistake. Otherwise, we are still seeing very good reliability issues and TrueNAS protects them from being wiped. We don't recommend an internet connection, except via VPN.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I'm getting flashbacks to ASUSGATE with the MyBooks getting wiped.


1624571370969.png
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Regarding zoned writes I wrote a blurb on this a while ago (two years now, I'm feeling old) and not much has changed since:

It's a bit of an odd situation certainly. SMR zones are currently 256MB, so trying to operate in smaller chunks means you'd need to do a lot of housekeeping and garbage collection. It's almost similar to how an SSD might program its NAND in 4K/8K blocks, but erase it in 1M - it needs spare area and time to tidy up the dirty pages before it can cleanly write again. ZFS might be a good candidate since it already has the "transaction group" concept, but it would require a fair bit of work and some pretty tight hardware requirements to say "okay, we have to write a minimium of 256MB to each member disk at a time here."

If ZFS gained a new object that was created per physical HA-SMR drive (or logical drive within, in the case of MAMR drives) and could then track the write pointer of each zone, it would in theory be able to dump data into the zones and keeps its own internal records of when a physical zone gets sufficiently dirty to be considered for cleanup (eg: you have two 256MB zones that have 128MB of dirty data each - read both, consolidate to single, upon atomic completion, reset write pointer for the two dirty zones and mark as available again) - this is definitely a far-from-trivial thing to implement and will doubtless come with a lot of performance baggage/overhead. But for certain workloads (media storage, cumulative backups, archival use) it will likely be able to restore performance to an "acceptable" level.

SMR may be here, but it's not a universally applicable technology. Most situations that demand performance are indeed moving to SSD, but the endurance decrease with higher cell density (if you thought QLC was bad, just wait for PLC) merits consideration for all but the lightest write workloads.

As I said before - vendors just need to be honest and up-front. If it's SMR, make it obvious.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,904
If ZFS gained a new object that was created per physical HA-SMR drive (or logical drive within, in the case of MAMR drives) and could then track the write pointer of each zone, it would in theory be able to dump data into the zones and keeps its own internal records of when a physical zone gets sufficiently dirty to be considered for cleanup [..]
Basically that is host-managed SMR and I am certain (didn't look things up) that cloud providers are working on this. As you wrote, once there is enough context (i.e. understanding of the workload) to use SMR in a clever and not brute-force way, it can be a suitable technology for many non-consumer scenarios.

From a commercial point of view this would be an interesting differentiator. Imagine as a company I could automatically migrate content that changes rarely onto drives that are considerably cheaper. And if we are talking about a couple of hundred or thousand drives, that and the non-proprietary nature (no vendor lock-in) could make a pretty compelling story for larger companies.

On an architectural level, this seems to be a variant of the tiered storage that is currently discussed in another thread. At the end of the day both are about transparently moving data based on their access characteristics (if I am not mistaken).
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Two things likely have to happen before wide-spread adoption.
  1. The oligopoly of HDD OEMs have to standardize on how HM-SMR will be incorporated universally (SMART, et al)
  2. Drivers from the software side (operating systems, appliances, etc) have to interface gracefully with said HM-SMR
DM-SMR is the worst of all worlds because the computer trying to write to it has no insight into why the HDD is dropping out periodically to flush the CMR cache to SMR. It just does, the HDD becomes unresponsive and then the pool has to wait for the HDD to become available again before it can complete a write. To the computer, this behavior is eerily similar as a failing hard drive.

The only workaround at the moment is either to deduce DM-SMR on the basis of behavior (repeatable observation of drop-outs under write) or using lists of model / serial numbers, neither of which is efficient. However, perhaps the former approach could be used by the likes of iXSystems as part of the disk setup / format process? At the very least, TrueNAS should warn users re: SMR, with DM-SMR being classified as risky, followed by HM- or HA-SMR classified as performance-reducing (once appropriate drivers, diagnostics, and so on have been developed)

HM-SMR has potential since it actually announces itself as such to the host and the issues associated with SMR can be accounted for. However, in their haste to make money, WD chose to Trojan-horse DM-SMR into markets like NAS that had no business using DM-SMR, ever.

That, combined with the subsequent stone-walling, dissembling, etc. is what is so disappointing. Never mind subsequent shots to the foot like “5900 RPM Class” hard drives. The only winners here are Seagate and Toshiba who may profit re: market share from WDs arrogance and incompetence.
 

Forza

Explorer
Joined
Apr 28, 2021
Messages
81
  1. The oligopoly of HDD OEMs have to standardize on how HM-SMR will be incorporated universally (SMART, et al)
  2. Drivers from the software side (operating systems, appliances, etc) have to interface gracefully with said HM-SMR
  • NVME Zoned Storage is an approved standard in the recent NVME 2.0 specification: https://nvmexpress.org/everything-y...0-specifications-and-new-technical-proposals/
  • SCSI Zoned Block Commands (ZBC) is an approved standard since 2017 under the name "ANSI INCITS 536-2017" - Seagate and others are members of the body that developed this standard.
  • Zoned ATA Commands is also an approved standard in "ANSI INCITS 537-2016".
ZFS could be made HM-Zoned aware, fixing most of the problems. We don't need operating system support other than an up-to-date SCSI/SATA/NVME driver. TrueNAS appliance would have to have to build some support into its tooling to control and manage Zones, once ZFS is Zoned aware.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Standards are nice and all, but they're meaningless unless widely adopted. That is yet to come.
 
Joined
Jan 4, 2014
Messages
1,644
There are lots of initialisms (that all end in R!) around HDD recording technology. For recording methods, there's GMR, TMR (Seagate use the term TGMR in their earlier datasheets) and TDMR. For recording technology, the CMR and SMR initialisms will be familiar to many users now given all the hoopla around some WD RED NAS drives not being suitable for ZFS. There are still plenty of other recording technology initialisms though, including LMR, PMR, MAMR, HAMR, HDMR, ePMR and BPMR. Some of these technologies are still in the lab, while others currently have (or have had) commercial applications.

It seems to me it's possible to avoid having to deal with the science around recording technology when choosing a drive simply by answering the question 'Will my tracks be shingled?' If they will be, then it doesn't matter what recording technology was used; the drive just isn't suitable for ZFS (in its present form). As newer recording technologies find commercial applications, it's going to be interesting to see how drive manufacturers distinguish between drives that are ZFS compatible and drives that are not as it will no longer be just CMR vs SMR.

A more interesting question for me is 'Is ZFS in danger of becoming obsolete, or becoming too expensive to use?' I'm not advocating this btw, but it is a possible future. Look at it from this perspective. Shingling is all the rage atm. It's all about 'packing more in the same amount of space'. If ZFS doesn't embrace shingling, then more hardware is required to provide the same capacity for a ZFS compatible drive. Drive manufacturers then have to create two separate NAS lines - a more expensive non-shingled NAS line and a cheaper shingled NAS line. We're already seeing this happen with cheaper WD RED shingled (SMR) drives and more expensive WD RED+ non-shingled (CMR) drives. It's possible the dollar gap will widen as newer recording technologies are commercialised and higher capacities reached. There's also the danger that ZFS is perceived as a minority file system, which would trigger a further cost spiral upwards for non-shingled drives.

Anyway, enough rambling from me. I stumbled across a site that does a good job of using visual representations to explain the different recording technologies. It also has excellent databases of CMR/SMR drives from the major players. Check it out. GutenData24: How to сhoose hard drive?! Introduction in magnetic recording technology.

Final word: I think pirates would love recording technology as there's plenty of R's to go around :wink:.
 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I think ZFS can adapt to SMR, but it will require work and testing. The SMR will have to be rock-solid even if it is a little slower.

The primary issue with the SMR drives comes from their ability to "collapse" under load.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
A more interesting question for me is 'Is ZFS in danger of becoming obsolete, or becoming too expensive to use?' I'm not advocating this btw, but is a possible future. Look at it from this perspective. Shingling is all the rage atm. It's all about 'packing more in the same amount of space'. If ZFS doesn't embrace shingling, then more hardware is required to provide the same capacity for a ZFS compatible drive. Drive manufacturers then have to create two separate NAS lines - a more expensive non-shingled NAS line and a cheaper shingled NAS line. We're already seeing this happen with cheaper WD RED shingled (SMR) drives and more expensive WD RED+ non-shingled (CMR) drives. It's possible the dollar gap will widen as newer recording technologies are commercialised and higher capacities reached. There's also the danger that ZFS is perceived as a minority file system, which would trigger a further cost spiral upwards for non-shingled drives.

I suspect the answer is in the expected remaining remaining useful lifespan of magnetic recording media. It's 2021... Last time I checked, spinning rust stayed competitive until roughly 2030, or 9 years away... I have disks sitting around in boxes for various reasons that are twice this age. (Anyone need a 110Mhz TurboSPARC SPARCStation 5 pizza box? I can throw in a 13w3 to VGA adapter! :smile: ) Shingled recording is not going to stop this onslaught. At some point SSD's finish the takeover and you're going to need PCIe switch HBA's and a bunch of NVMe drives. There are 16Tb 2.5" NVMe devices in development or on the market now, but they cost $7k+/ea. But the end is in sight! ZFS may or may not continue into this new world. It needs to start adapting now.

You would not believe the failure modes some of these 16Tb SSD's present. :smile:
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I suspect the answer is in the expected remaining remaining useful lifespan of magnetic recording media. Shingled recording is not going to stop this onslaught. At some point SSD's finish the takeover and you're going to need PCIe switch HBA's and a bunch of NVMe drives. There are 16Tb 2.5" NVMe devices in development or on the market now, but they cost $7k+/ea. But the end is in sight! ZFS may or may not continue into this new world. It needs to start adapting now.

I've been in the flash business for 14 years and disagree on three levels:

1) 16TB SSDs cost >$5K, while equivalent HDDs cost $500. 10X is a huge cost difference. The flash vendors do not want to drop their prices and can't afford to ship the volume of bits that are currently shipping on HDDs. The high capacity HDD market keeps growing. (the sub 2TB HDD is shrinking and dying). Sports car companies aren't really interested in the school bus market... it's unprofitable for them.

2) ZFS works very well on flash SSDs. It provides all the snapshots, clones and replication. It manages the data integrity and compression well. Its also pretty good with QLC; it aggregates writes and increases the endurance.

3) The hybridization of ZFS storage keeps on improving. Mixed SSD and HDD (fusion) pools are now viable in addition to standard L2ARC and SLOG functions. ZFS enables efficient replication of active flash pools to low cost HDD pools.

Managing large amounts of data is complex..... it need reliability, application-specific performance and economics. ZFS needs to keep evolving, but there are many willing participants that like the tools that it provides.
 
Top