Helping SMR drives with a SSD

[AdrianB1]

Hello again. I have my small NAS setup to keep some pictures and home video captures (VHS era camera conversions) and it was always very slow. In the past it was weak hardware and slow disks, with the latest configuration I tracked the mystery to Seagate 8TB SMR drives that I have (3 pcs). These disks can write at about 45-50 MB/sec, read speed is decent (enough to be limited by the 1 Gbps NIC).

I have an Intel Datacenter SSD drive in 240 GB capacity; it is relatively small for a SSD, but solid, it is MLC (good endurance) and I already have it, so I am thinking how to best use it for caching. The IO should not be a problem, I don't write a lot, most of the time I navigate reading some pictures (tens of thousands) and sometimes even navigating the folder structure is quite slow. I tried in the past to index the pictures with software like digiKam, but anything I tried was either extremely slow (5% per day) or crashing after a while.

The questions are: will the SSD help in any way for this kind of usage? If yes, how to configure it best - I have only 1, so I can use it as L2ARC or as a SLOG but I guess a SLOG is not needed in this context, will the L2ARC help if I do random reads from all over? There is a very small chance something to be cached on the SSD as the pictures can almost fit entirely on the SSD (about 300 GB of pics) but they will not be cached until they are first opened and then how long will they stay in the L2ARC?

 

[ChrisRJ]

If this is in the context of FreeNAS/TrueNAS (there is not enough information to determine this with certainty) the issue of SMR drives is primarily when it comes to resilvering. During normal operation, many people will not see big issues. Your favorite search engine will provide you with more details.

As to your question: I cannot figure out what your problem is. Please rephrase.

 

[AdrianB1]

It is TrueNAS, of course. I am currently resilvering the array for the past 3 days and it is still work in progress, but that is a different story. As I said, other than sequential reads everything else is slow, from browsing folders to opening relatively small files like regular pictures. For example opening a folder to display a list of 100 files with no preview can take 5 seconds vs. less than 1 second on any other drive, opening a single 500kb picture takes several seconds, displaying the next one takes a few seconds, all these operations are almost instantaneous on a local disk so the perceived performance difference is huge.

 

[jgreco]

It's difficult without a description of your NAS.

ZFS does its trick by using lots of resources, and usually when someone says "slow", it means one or more of

1) Bad disks
2) Bad pool design
3) Slow CPU
4) Limited memory
5) Bad network

 

[AdrianB1]

Here is the configuration:
- 4 cores from a Ryzen 5600x (it is a VM)
- 32 GB of RAM
- 4 disks, 8TB Seagate ST8000DM004, 3 in RAIDZ1 the and 1 hot spare (added later)
- 2 Gbps NIC (2 x 1 Gbps with link aggregation)

The CPU is definitely not slow, network and memory are fine. The disks are fine (SMART reports no problems), the pools design is whatever can be built using the GUI with no tweaks. The same disks were tested in a software mirror (RAID1) in a Windows server, the performance was about the same. I don't see anything related to this specific configuration other than the SMR disks with very low write performance, this is why I did not list the full specs from the beginning. I just checked the local (no network) performance copying from the TrueNAS VM to the HyperV host some large files (a dozen GB per file) and the Windows file copy on the HyperV reports ~ 300 MB/sec, so the read speed from the TrueNAS VM is good for large files, writing is still ~ 50 MB/s for large files, but the access time seems terrible even for reads.

 

[ChrisRJ]

So you are saying that TrueNAS is running inside a VM on HyperV?

 

[AdrianB1]

Yes, for some time - since mid 2019. I tried with a test bench with an Intel i8100 and 32 GB of RAM as a physical computer, there is no difference. I have a second TrueNAS VM on a different computer (i7100, 32 GB RAM) with smaller and older disks (CMR, not SMR) as a backup, it is faster - read and write close to network speed, so the virtualization part is not the cause.

 

[Constantin]

SMR drives are trouble and should be replaced if possible. However, their problems are related to writing, not reading data. Basically, think of them as a conventional (CMR) and a shingled (SMR) drive in one. A relatively small CMR sector is used as "fast" write buffer and once it fills, the data is flushed to the SMR parts of the drive in one big write.

There are all sorts of reasons why such hard drives will not play nice with ZFS, especially so-called Device-Managed SMR drives (which yours are). See here as a start. There is a lot more data over at ServeTheHome, Blocks and Files, among other places. Basically, as far as ZFS can tell, the drives are dropping out and dying as SMR sectors are written. Think of them as sort-of functional despite being fundamentally flawed for use in ZFS as explained by Manfred Berger at HGST/WD.

As for your second question, I have found having a so-called metadata-only L2ARC to be very useful for applications where the name of the game is lots of directory access, traversals, and so on. In my case, it's related to rsync, for you, it's browsing. See this testing I did here. However, keep in mind that you want your system to have adequate RAM to allow the ARC to do its work. 32GB is likely too low. Also, if you use FreeNAS or a default configuration TrueNAS system, the metadata saved in the L2ARC will not be persistent. Each time the system is rebooted, the cache has to get "hot" again. TrueNAS 12.x does allow the L2ARC to become persistent.

Hot spares are also trouble. You really want to use badblocks and like approaches to burn in a drive, qualify it, and then set it aside as a cold spare. Otherwise, you'd be just as well off configuring your pool from the start as a Z2.

There are also a couple of guides on how to improve browsing experience by tweaking SMB settings to drop DOS attributes and the like. @Cyberjock posted a nice list a few years ago.

I doubt link aggregation will do much. Both AFP and SMB are a single-thread process, so neither will take advantage of the two pipes being available - both protocols can only fill one or the other. NFS may behave better but has other issues. There is no free lunch.

 

[AdrianB1]

Thank you very much for all the info provided. I will see what I can do about the drives, but replacing them will not be easy. I see why they are not a good choice for a new NAS, but at the time I bought it there was no information available about the recording technology, so it is what it is.

I read the discussion about the L2ARC performance improvements, it suggests that I would do it too. Is now a tested and supported way to make it persistent after reboots? As I have a 240 GB SSD and just about 300 GB of pictures, should it be metadata only or a regular L2ARC?

I read about hot spares too late, 18 months after I have created the pool, so there was no way to make it a Z2 from the start especially because the disk hot spare was added only a week ago. Even now I have no means to backup everything and rebuild the pool as Z2, the old disks in the backup TrueNAS are really small and I have some less important stuff that I want to keep and that takes a few TB.

The link aggregation is done in the HyperV host, so TrueNAS is not aware of it. It is using a LAG with the Cisco router/switch and in theory is doing a port aggregation that should deliver 2Gbps, but I did not test that yet because the rest of the network is not fast enough at this time. I would like to go to a SPF+ NIC and a DAC, but I cannot find an used NIC in this part of the world and a new one is too expensive.

 

[Constantin]

See here re persistent L2ARC, which was but a short search away.

How to best configure the L2ARC is debatable but the good news is that it’s optional. So you can configure it one way or the other and then see how it suits you.

i still fail to see the reason to keep a hot spare. Pull the drive if you’re confident it’s good and set it aside. Hot spares give no benefit in a residential or small office setting. They make more sense in a remote server sitting in a colo site but even then I’d configure for better Z redundancy rather than have a drive spinning and wearing for no benefit.

but more than anything, if better write performance and overall responsiveness is desired, I’d plan for a transition to a CMR based pool.

 

[AdrianB1]

I had some errors on a disk and this is when I added the hot spare: if that disk was to fail, the hot spare was there to replace it. This is how I used to do 20 years ago when I was managing Novell file servers with Adaptec SCSI RAID arrays, maybe it is not the best idea today but it should not be just bad. In the meanwhile that disk was faulted and the hot spare replaced it, I removed the bad disk and I will send for replacement under warranty, so there will be no hot spare in the future.

Write performance is not a problem, I am using the TrueNAS storage mostly as WORM. I would just like to improve the read performance if possible, otherwise I can live with it.

 

[Constantin]

The issues with SMR are most apparent during a resilvering event. I would go and educate myself re: to what extent IXsystems has been able to work around the known issues with DM-SMR drives (see the links above).

Even during regular operation, an occasional CMR-SMR sector transfer will happen as the on-drive cache fills up and the HDD firmware does the equivalent of what is common in SSDs with fast flashes in the front and slower flash in the back. The main difference being that HDDs are much slower at this.

As I understand it, the more drives you have in a pool, the more this behavior will become an issue as *any* drive going offline for a CMR-SMR transfer will hold the entire pool hostage until it completes. So the more drives you have in a pool, the more apparent the impact.

Because the device manages these CMR-SMR cache transfers, it's not as if ZFS can coordinate a global pool cache transfer and hence minimize the impact of CMR-SMR transfers. Nope, DM-SMR drives all do this individually, which is a huge problem in RAID, especially if the drive does not communicate this operation properly to the host.

Some high-volume B2B customers are getting Host-Managed SMR drives for cold storage to help mitigate this issue. However, HM-SMR drives are not sold into the usual consumer distribution channels (you may find it on the used market in the future, I suppose) and the relevant software to play nice with HM-SMR has to be developed and deployed. To my knowledge, IXsystems has not done so.