Uncorrectable I/O failure and disaster recovery

[ghost_za]

Hi,

I am satill diagnosing the problem. But, wanted to get feedback and shared my frustrations on the forums.

I bought a WD 3TiB for extra storage on my personal TrueNAS system.
Recently upgraded to 13

Created a single ZFS pool for the data as I don't have the cash to buy 3 drives and put them all ina pool. There is lots of drives and 1 of them is a 3 drive zpool.

Initially I didn;t have any problems creating the zpool or share or copying the data to the drive.
Once in a while (like every 3 months or so) I will do a complete drive health check to ensure no unexpected drive failures. I found a few pending sectors on 1 of the drives and repaired them but the new WD failed the EXTENDED smart test. It stuck on 90% every time I ran it. I contacted the supplier and informed them of the issue. Both SHORT and CONVEYANCE test completed fine. The SMART did not report any failures, just stuck at 90% evcery time. I assumed the drive was faulty but as SMART was not reporting any immanent failure I waited on a M.2 - 6SATA expansion card to expand the Truenas system (to have more sata ports) so I can backup the data and send it back to the supplier. The m.2 card has a ASM1166 chip. So the SMART issue presented before the m.2 card. I received the card Friday(6-10-23) and continued with the installation. Everything worked right off the bat with some performance issues - I expected the expansion card would have reduced throughput to no bif surprise. Left the new WD on the on-board SATA ports and added 2 drives as tests. Shortly there after I received some minor I/O failures on this WD. I assumed I was unrelated and it was just the drives' natural failre creeping up. Saturday it started giving me and "Uncorrectable I/O failure" pool is suspended warning and the poolo was unaccessable.

TrueNAS restart fixed the issue (for a while - like 5 hours or so) - which leads me to my first FRUSTRATION with TrueNAS.

I could still access the other drives but the CPU load went ape-shit. Running at 75-80 degrees with 100% load. I doidn't investigate exactly what service was causing the issue. just restart. Now irrelevant of what caused the issue a server OS like TrueNAS can't be causing a runaway CPU load like that. If it was an unattended server and the load went unnoticed for weeks it could have severly damaged the CPU (or even other components). Then OK it could be argued that no preventative measure could have helped and that shit-happens right...OK fine.
But when I tried to reboot the system remotely it backfired. While I left the system as-is the remaining pools were still accessable and functioning - exactly what I'd expect of a high availalbility NAS system like truenas. But when I rebooted it stopped some services including all the managing services like WebUI and SSH services but froze on another part ( irrelavant of on what it froze ). So essentially rebooting the system killed it. If this was an unattended system in a remote location then I would have entirely killed the server because 1 drive failed. No way to reboot the server short of cutting the power physically.
Now this is unacceptable - this means I can (or rather won't) ever install this in a production environment like a business.

Now I'm diagnosing the issue now / trying to disaster recovery / data recovery this drive as the data isn't essential but I would prefer to recovery it if possible.
I disconnected/exported the pool to remove the pool, becasue by Sunday (yesterday) the truenas wouldn't even boot anymore if the WD is connected. same 'Uncorrectable I/O" error and then it freezes on creating the cache for a different pool. Once again the system is essentailly killed due to 1 drive failure. If it was remote I would have been screwed. To diagnose I disconnected the pool.

If I do that then I can boot the system. Even with the drive connected at bootup. In the system I can see the drive displayed and here is the interesting part. All smart tests succeed - even the LONG ( that previosuly froze at 90%) the slugishness I noticed on the system is gone and everything seems fine again.

Then I tried to import the pool - I did this as the last step before removing the drive and performing data recovery on it. When I do this it kills the system again the entire middlewared is completely unresponsive and once again can't reboot remotely. it freezes on the reboot process. So system killed by 1 drive failure again.

I left the first import try for 12 hours nad tried 2 times after that. When rebooting the pool is back and tries to import it and then freezes on the cache part of another pool like previsouly stated. So can't boot again.

I have now as a last step on the server removed the M.2 expansion card and remove the drive to diagnose independantly and perform data recovery if necessary.

So the point of this post is:
1. To vent my frustrations with TrueNAS and to inform the devs of IX system to please have a look at how ZFS errors are handled to avoid a complete system failure in a scenario like this.

2. To get some feedback on what could be causing this behaviour. Because where I am now in testing is I'm not finding any errors on the drive inself. Smart is succedding and recovery is not reporting any errors (so far). So is't looking more and more like the drive is fine and this was caused by some oither issue.

Thoughts? Ideas?

PS: If someone needs debug logs (or something) please just ask for what you need I will try to provide it.

 

[Redcoat]

Hi,

I am satill diagnosing the problem. But, wanted to get feedback and shared my frustrations on the forums.

I bought a WD 3TiB for extra storage on my personal TrueNAS system.
Recently upgraded to 13

Created a single ZFS pool for the data as I don't have the cash to buy 3 drives and put them all ina pool. There is lots of drives and 1 of them is a 3 drive zpool.

Initially I didn;t have any problems creating the zpool or share or copying the data to the drive.
Once in a while (like every 3 months or so) I will do a complete drive health check to ensure no unexpected drive failures. I found a few pending sectors on 1 of the drives and repaired them but the new WD failed the EXTENDED smart test. It stuck on 90% every time I ran it. I contacted the supplier and informed them of the issue. Both SHORT and CONVEYANCE test completed fine. The SMART did not report any failures, just stuck at 90% evcery time. I assumed the drive was faulty but as SMART was not reporting any immanent failure I waited on a M.2 - 6SATA expansion card to expand the Truenas system (to have more sata ports) so I can backup the data and send it back to the supplier. The m.2 card has a ASM1166 chip. So the SMART issue presented before the m.2 card. I received the card Friday(6-10-23) and continued with the installation. Everything worked right off the bat with some performance issues - I expected the expansion card would have reduced throughput to no bif surprise. Left the new WD on the on-board SATA ports and added 2 drives as tests. Shortly there after I received some minor I/O failures on this WD. I assumed I was unrelated and it was just the drives' natural failre creeping up. Saturday it started giving me and "Uncorrectable I/O failure" pool is suspended warning and the poolo was unaccessable.

TrueNAS restart fixed the issue (for a while - like 5 hours or so) - which leads me to my first FRUSTRATION with TrueNAS.

I could still access the other drives but the CPU load went ape-shit. Running at 75-80 degrees with 100% load. I doidn't investigate exactly what service was causing the issue. just restart. Now irrelevant of what caused the issue a server OS like TrueNAS can't be causing a runaway CPU load like that. If it was an unattended server and the load went unnoticed for weeks it could have severly damaged the CPU (or even other components). Then OK it could be argued that no preventative measure could have helped and that shit-happens right...OK fine.
But when I tried to reboot the system remotely it backfired. While I left the system as-is the remaining pools were still accessable and functioning - exactly what I'd expect of a high availalbility NAS system like truenas. But when I rebooted it stopped some services including all the managing services like WebUI and SSH services but froze on another part ( irrelavant of on what it froze ). So essentially rebooting the system killed it. If this was an unattended system in a remote location then I would have entirely killed the server because 1 drive failed. No way to reboot the server short of cutting the power physically.
Now this is unacceptable - this means I can (or rather won't) ever install this in a production environment like a business.

Now I'm diagnosing the issue now / trying to disaster recovery / data recovery this drive as the data isn't essential but I would prefer to recovery it if possible.
I disconnected/exported the pool to remove the pool, becasue by Sunday (yesterday) the truenas wouldn't even boot anymore if the WD is connected. same 'Uncorrectable I/O" error and then it freezes on creating the cache for a different pool. Once again the system is essentailly killed due to 1 drive failure. If it was remote I would have been screwed. To diagnose I disconnected the pool.

If I do that then I can boot the system. Even with the drive connected at bootup. In the system I can see the drive displayed and here is the interesting part. All smart tests succeed - even the LONG ( that previosuly froze at 90%) the slugishness I noticed on the system is gone and everything seems fine again.

Then I tried to import the pool - I did this as the last step before removing the drive and performing data recovery on it. When I do this it kills the system again the entire middlewared is completely unresponsive and once again can't reboot remotely. it freezes on the reboot process. So system killed by 1 drive failure again.

I left the first import try for 12 hours nad tried 2 times after that. When rebooting the pool is back and tries to import it and then freezes on the cache part of another pool like previsouly stated. So can't boot again.

I have now as a last step on the server removed the M.2 expansion card and remove the drive to diagnose independantly and perform data recovery if necessary.

So the point of this post is:
1. To vent my frustrations with TrueNAS and to inform the devs of IX system to please have a look at how ZFS errors are handled to avoid a complete system failure in a scenario like this.

2. To get some feedback on what could be causing this behaviour. Because where I am now in testing is I'm not finding any errors on the drive inself. Smart is succedding and recovery is not reporting any errors (so far). So is't looking more and more like the drive is fine and this was caused by some oither issue.

Thoughts? Ideas?

PS: If someone needs debug logs (or something) please just ask for what you need I will try to provide it

Your 3TB WD HDD isn’t an SMR drive, is it?

What is its model number?

 

[Arwen]

Please list all your hardware, make & model, plus how it is wired. Without further information, it is hard to assist further.


Their are specific hardware configurations that don't work well, or at all with ZFS. Especially in the consumer environment, (aka home & small business). For example, hardware RAID controllers, SATA Port Multipliers and SMR disks.


PS - The iX staff that read and post to these forums are generally quite nice. But, unless you have a paid support contract, don't expect them to fix your problem. It's alright to expect the them to help, if possible. Most of the time it is other users of TrueNAS that assist here.

 

[ghost_za]

I have to add that EVERYTHING worked absolutely PERFECT up until this drive. I have to compliment IX Systems on a well designed OS. I'm just venting my frustrations however I have to add that the unresponsive reboot IS unacceptable.

I'm doing this from memory...
MSI X470 GAMING PLUS MAX
AMD Ryzen 5950G
16GB 3200MHz Klevv Cras DDR4
Intel 8245EM Pro dual 100/1000 PCI-e Expansion card (pci-e x16_0)

sata0+:
Samsung 750 Pro 120GB bootpool drive
Seagate ST2000DM008-2FR10 on single stripe volume
WD WD-WMAYU6409973 in RAIDz volume with 3 drives [0]
Seagate ST500LT012-1DG142 in RAIDz volume with 3 drives [1]
Seagate ST3500418AS in RAIDz volume with 3 drives [2]
WD WDC WD20EARX-00PASB0 on single stripe volume
WD Green WDC WDS240G2G0C-00AJM0 m.2 on m.2_0 (for jails and VMs)
Unbranded m.2 expansion card ASM1166 chip on m.2_1

Like I said everything works perfectly fine now that I removed the drive to do tests on it. BTW: still no errors so far

smartctl output on seperate pc doing the tests.

=== START OF INFORMATION SECTION ===
Device Model: WDC WD30EZAZ-00SF3B0
Serial Number: WD-WX42D61KC81J
LU WWN Device Id: 5 0014ee 2befa76c6
Firmware Version: 80.00A80
User Capacity: 3 000 592 982 016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Oct 09 11:39:45 2023 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: (25320) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 333) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 210 200 021 Pre-fail Always - 2483
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 330
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 587
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 288
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 280
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 123
194 Temperature_Celsius 0x0022 115 099 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1

SMART Error Log Version: 1
ATA Error Count: 20 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 20 occurred at disk power-on lifetime: 557 hours (23 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 61 01 00 00 00 40 Device Fault; Error: ABRT 1 sectors at LBA = 0x00000000 = 0

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA

Error 19 occurred at disk power-on lifetime: 557 hours (23 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 61 01 00 00 00 40 Device Fault; Error: ABRT 1 sectors at LBA = 0x00000000 = 0

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
ef 02 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable write cache]

Error 18 occurred at disk power-on lifetime: 557 hours (23 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 61 01 00 00 00 40 Device Fault; Error: ABRT 1 sectors at LBA = 0x00000000 = 0

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
ef 02 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable read look-ahead]

Error 17 occurred at disk power-on lifetime: 557 hours (23 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 61 01 00 00 00 40 Device Fault; Error: ABRT 1 sectors at LBA = 0x00000000 = 0

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
ef 02 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 08 00:17:58.300 SET MULTIPLE MODE

Error 16 occurred at disk power-on lifetime: 557 hours (23 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 61 01 00 00 00 40 Device Fault; Error: ABRT 1 sectors at LBA = 0x00000000 = 0

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 01 00 00 00 40 08 00:17:58.301 READ DMA
ef 02 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable read look-ahead]
c6 00 10 00 00 00 40 08 00:17:58.300 SET MULTIPLE MODE
ef 10 02 00 00 00 40 08 00:17:58.300 SET FEATURES [Enable SATA feature]

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 587 -
# 2 Extended offline Completed without error 00% 576 -
# 3 Short offline Completed without error 00% 438 -
# 4 Conveyance offline Completed without error 00% 437 -
# 5 Extended offline Aborted by host 10% 339 -
# 6 Extended offline Aborted by host 10% 288 -
# 7 Conveyance offline Completed without error 00% 166 -
# 8 Short offline Completed without error 00% 165 -
# 9 Conveyance offline Completed without error 00% 26 -
#10 Short offline Completed without error 00% 25 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The last successful extended was done on truenas while the pool was disconnected.

 

[ghost_za]

Oh there is one other thing I can complain about. Also regarding the error handling and a simple error killing the system.

I had the dual Intel 1g interfaces on LACP port aggregation with a DGS-1210-28P Gigabit Ethernet Switch

Worked perfect for a few months (like 4 or 5 maybe) then it started killing the server with "interface stopped distributing - possible flapping"
Never had this issue before and I diagnosed it to absolutely no avail. Best I can figure the card is dying. still once that happend it killed the system completely. even the on-board interface I use for management and essential services (like DNS) and VM's.

I have recently disabled the LACP and it seems to run fine (for the moment at least ). Waiting on a dual 10g X540 Intel card and switch to use LACP on dual 10g interfaces.

And nothing restores the system short of a reboot - and again it only reboots on a hard reboot local cli access (reboot command) also doesn't reboot the system.

So it seems to me truenas core has a serious issue with error handling and disaster recovery

 

[Arwen]

This drive, WDC WD30EZAZ, is SMR, not compatible with ZFS. SMR drives can do odd things, especially ones from Western Digital. While other manufacturer's SMR drives may work with ZFS for a long time, whence the drive gets fragmented, it may slow down to a crawl.

It's possible that a scheduled ZFS Scrub is running on that drive, causing bad delays on import. SMR drives torture ZFS' enhanced scrubs because ZFS is attempted to do the data scrub in block order. But, if the SMR drive is fragmented, this may cause ZFS to take much longer than normal to complete a scrub. (If it can even complete without time outs.)

Your disk WDC WD20EARX is ECO drive, WDC Green series. Desktop, laptop and eco drives are generally not recommended for a NAS. ZFS was written for Enterprise servers, and to run 24x7x365 for years. Handing ZFS misc. components does not mean ZFS will be reliable on those components.

The 3 disk RAID-Z1 with WDC disks, seems to have listed the serial number, and not the model number;

WD WD-WMAYU6409973 in RAIDz volume with 3 drives [0]

 

[ghost_za]

Yeah there are a bunch of old drive I don't want to add to my gaming PC but I want the storage space so I started using Truenas to take advantage of this. I'm slowly upgrading the system to be a proper NAS but currently it has all my reject HDD in it so yeah

 

[Arwen]

Oh, I forgot that this might use a SATA Port Multiplier as you mention that it has 6 x SATA ports;

Unbranded m.2 expansion card ASM1166 chip on m.2_1

See;

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

This resource was originally created by user: jgreco on the TrueNAS Community Forums Archive. Please DM this account or comment in this thread to claim it. In the last year or two, we’ve had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use...

www.truenas.com

 

[ghost_za]

Excuse my ignorance but what is SMR , I mean I can google it (and will) but just give me a brief overview of 'SMR' please

 

[ghost_za]

no those are the reported model numbers... if those are incorrect I will have to remove the drives

 

[ghost_za]

I did do research (if it was extensive thats up for debate ;) ). The card used is an ASM1166, thats the better of the chipsets I could find. And yeah I knew from the start that it will prob be slow but ironically it's not giving me ANY issues other then very slow speeds.

I'm assuming it's using 1 PCIe channel and dividing it between all 6 ports so that's not great but if I see that it is causing trouble I will remove it. but thus far it is working fine

 

[ghost_za]

I also have to add I've been using truenas for 2.5 years now with (mostly) this config. It's been operating smoothly (for the most part) with very little problems. Some of the drives are almost at fail point but I'm using them till they eventually die.

 

[Arwen]

Yes, the ASM1166 does appear to be a better chip with 6 native SATA ports, (meaning without a SATA Port Multiplier). Using it with only 1 or 2 PCIe lanes will slow the various disk transfers, but HDDs are slow enough it probably won't be a problem.


CMR is Conventional Magnetic Recording and SMR is Shingled Magnetic Recording. Hard drives write twice as wide as needed to read. SMR disks use that knowledge to overlay the tracks closer together. But, when a track has to be written, it must be read first, and any other related tracks too. Somewhat like the shingles on a roof. Need to remove a shingle part way down? Well, remove all the ones above that are affected. Then you can re-install the shingle / data block.

Because of the need to re-write tracks in SMR disks, they use indirect blocking. So block 0 could be at the end of the disk, just as easily as the beginning of the disk).

Next, SMR disks tend to keep some shingled groups already clear. Thus, if you need to write a lot of data, it can be written in sequence. Meaning track 10 is written, then track 11 is written, partially over writing track 10, (but NOT track 10's data!!!). Track 12 is damaged in the process, but if already clear, not a problem.

Some Western Digital SMR drives, (WD Reds, but not Red Plus or Red Pro), have a bug in their firmware making them less compatible with ZFS.


Sorry if my explanation of SMR disks is not very good. Their are ALOT of other descriptions of SMR disks, some with diagrams and charts, with test results, including with ZFS showing SMR disks are much less suitable for ZFS.

 

[ghost_za]

Sorry if my explanation of SMR disks is not very good. Their are ALOT of other descriptions of SMR disks, some with diagrams and charts, with test results, including with ZFS showing SMR disks are much less suitable for ZFS.

No that was exactly what I wanted to hear. So SMR is another type of magnetic recording. That explains (in my mind) to me why it would be ill-advised for ZFS.

Ok well shit...i didn't know that.

Ok so it's very likely that this drive won't ever work on truenas. Ok then I'll move this to a windows PC. That kinda bumms me out but it's understandable.

 

[ghost_za]

Still my issue with error handling still stands. How can an I/O error on 1 pool cause the system to be unable to reboot remotely.

Or a('possible') port flapping cause a complete outage... that's just not good error handling. Mind you it might be baked into freebsd not directly IX system's fault...might be... i haven't done the research

 

[Arwen]

SMR drives may work, and work for years. But, they can get so slow as to be problematic, (because during next track reads, drive is busy, even for 10s of seconds).

I use a Seagate Archive 8TB SMR drive as a single disk ZFS pool for one of my backups. When I bought it, it was one of the few 8TB disks available. It works, but is getting slower over time. So far it has succeeded in performing my backups fine. But, I am not backup time window constrained. And I have a 12TB disk in my backup rotation.


Next, the single 3TB disk pool may be corrupt due to a memory bit flip. Rare, but it happens with non-ECC memory.

Or it could be that you are using a Gaming system board, which may have over-clocking enabled by default in the BIOS. In general, servers don't over clock, (neither CPU or memory), because they want reliability over speed. TrueNAS complicates that because all pools should be scrubbed, at least monthly, perhaps every 2 weeks. This can create heat in non-server enclosure due to the constant disk reads that last for hours, even days.


As for the other issues, I don't know. Perhaps someone else will have further information.

 

[ghost_za]

Ok so it's very likely that this drive won't ever work on truenas. Ok then I'll move this to a windows PC. That kinda bumms me out but it's understandable.

If I understand the WD SMR issue correctly it's that it tries to cut costs by reducing the magnetic lane width thus creating a smaller platter in the HDD.

This is auto handled by the HDD by reading the lanes it tries to write first so it can restore the data that is overwritten later. Now I assume this is where ZFS doesn't play well with SMR...

Is there a work around or something? something that could potentially reduce speeds but give some degree of reliability - for instance?

 

[ghost_za]

Or it could be that you are using a Gaming system board, which may have over-clocking enabled by default in the BIOS. In general, servers don't over clock, (neither CPU or memory), because they want reliability over speed. TrueNAS complicates that because all pools should be scrubbed, at least monthly, perhaps every 2 weeks. This can create heat in non-server enclosure due to the constant disk reads that last for hours, even day

Nope, I have actually underclocked it for reliability and reduced temps.
I will say that the board doesn't particularly play well with the RAM - plan is to upgrade MB, CPU and RAM later this year.
...My scrubs are scheduled weekly - resilvers monthly. And I keep a close eye on those tasks. They aren't giving me any issue. the scrub times is even within expectations

Next, the single 3TB disk pool may be corrupt due to a memory bit flip. Rare, but it happens with non-ECC memory.

If that's the case, and it certainly sounds like it might be how would I fix it? I thought that exact same thing and thought a disconnect of the pool and an import might fix it. but as you know it didn;t

 

[ghost_za]

I can view the contents of the disk with ZFS data recovery software. Haven;t started the recovery yet. Waiting for the LONG test to complete but I assume the data is intact. So if it is corruption is has to reside in the ZFS metadata, I think

 

[ghost_za]

Oh and it's also relevant to mention that when I initially added the WD 3TB HDD I didn't immediately do a SMART test (mostly due to forgetting). I created the share and started my data transfers - which went fine and only after it was 72% full did I do a HDD health check on all the HDD's and that's when I first saw the frozen smart long test on the drive. So it would make sense that after all the transfers it caused some corruption due to the SMR issue.