I/O error occurred while writing; fd='23', error='Unknown error: 122 (122)'

[Castertr0y357]

Hello,

I've been seeing the following messages for at least 2 weeks now, but I've been unable to find any specific information about what's going on. I've scoured the forums as well as Google, but I can't seem to find anyone else who's hit this message.

Messages:

Code:

Jul 14 08:53:46 freenas syslog-ng[1642]: Error suspend timeout has elapsed, attempting to write again; fd='23'
Jul 14 08:53:46 freenas syslog-ng[1642]: I/O error occurred while writing; fd='23', error='Unknown error: 122 (122)'
Jul 14 08:53:46 freenas syslog-ng[1642]: Suspending write operation because of an I/O error; fd='23', time_reopen='60'

These messages keep appearing over and over in the log file.

Zpool status output for zpool in question:

Code:

root@freenas:~ # zpool status -v Vol1
  pool: Vol1
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 3.44M in 0 days 08:50:42 with 0 errors on Sun Jun 10 08:50:43 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		Vol1											DEGRADED	 0	 0 8.04K
		  raidz1-0									  DEGRADED	 0	 0 16.1K
			gptid/f485c881-655f-11e5-a92e-7427ea01dce4  DEGRADED	 0	 0	 0  too many errors
			gptid/f5aa005d-c333-11e5-9df4-7427ea01dce4  DEGRADED	 0	 0	 0  too many errors
			gptid/a1dc9873-4575-11e6-ba5f-7427ea01dce4  DEGRADED	 0	 0	 0  too many errors
			gptid/cb7dc330-66a9-11e4-82ff-7427ea01dce4  ONLINE	   0	 0	 0
			gptid/77e26c0e-2ac4-11e7-8976-708bcd7c4011  ONLINE	   0	 0	 0

errors: Permanent errors have been detected in the following files:

		/var/db/system/syslog-7d0973d92e38411da7ce5f40724800c8/log/daemon.log

Nothing has really been affected. All my jails and plugins are still functioning as normal, but I'd like to not have a crash.

This is only being used for a personal media server, so I'm not doing any snapshots since nothing is really irreplaceable. I'd rather not have to re-acquire everything since it takes time.

What are my options for here? I don't think I'm a risk for losing disks, but I'd rather not wait and find out.

 

[kdragon75]

Do you have a backup? If not, make one on separate hardware now. Your pool is degraded and has corrupted data in /var/db/system/syslog-7d0973d92e38411da7ce5f40724800c8/log/daemon.log. Thankfully this is just a log file but there's no saying it will end there.
Can you post the output of smartctl -a /dev/DRIVE-NAME-HERE for each drive?

 

[kdragon75]

Also per forum rules, please post full details of your system including model numbers where able.

 

[Castertr0y357]

System info:
Build FreeNAS-11.1-U5
Platform AMD FX(tm)-8320 Eight-Core Processor
Memory 32456MB

Drives are 3TB SATA

Sadly, the raid card I have installed doesn't support SMART on it. I'm just using it to give me more ports. I do have enough ports on the motherboard and enough cables to connect them directly to, but since I had the card, I wouldn't use up the ports on the motherboard. Does smartctl have the capability to fix errors like this? If not, I don't have enough space anywhere else to backup my data to. Would I be able to put the drive in a different machine, run the test and potentially fix any errors, then replace it in my freenas box?

Raid card being utilized: https://www.amazon.com/gp/product/B0050SLTPC/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

I don't need hardware raid support since I'm just using JBOD to freenas. Is there a better card I could be using that has SMART support that isn't $300+?

smartctl -a output... Not very useful though:

Code:

root@freenas:~ # smartctl -a /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HPT
Product:			  DISK 0_0
Revision:			 4.00
User Capacity:		3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Device type:		  disk
Local Time is:		Sat Jul 14 15:02:19 2018 EDT
SMART support is:	 Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Request Sense failed, [Input/output error]
Error Counter logging not supported

Device does not support Self Test logging
root@freenas:~ # smartctl -a /dev/da1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HPT
Product:			  DISK 0_1
Revision:			 4.00
User Capacity:		3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Device type:		  disk
Local Time is:		Sat Jul 14 15:02:38 2018 EDT
SMART support is:	 Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Request Sense failed, [Input/output error]
Error Counter logging not supported

Device does not support Self Test logging
root@freenas:~ # smartctl -a /dev/da2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HPT
Product:			  DISK 0_2
Revision:			 4.00
User Capacity:		3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Device type:		  disk
Local Time is:		Sat Jul 14 15:02:39 2018 EDT
SMART support is:	 Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Request Sense failed, [Input/output error]
Error Counter logging not supported

Device does not support Self Test logging
root@freenas:~ # smartctl -a /dev/da3
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HPT
Product:			  DISK 0_3
Revision:			 4.00
User Capacity:		3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Device type:		  disk
Local Time is:		Sat Jul 14 15:02:40 2018 EDT
SMART support is:	 Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Request Sense failed, [Input/output error]
Error Counter logging not supported

Device does not support Self Test logging
root@freenas:~ # smartctl -a /dev/da4
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:			   HPT
Product:			  DISK 0_4
Revision:			 4.00
User Capacity:		3,000,592,982,016 bytes [3.00 TB]
Logical block size:   512 bytes
Device type:		  disk
Local Time is:		Sat Jul 14 15:02:42 2018 EDT
SMART support is:	 Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Request Sense failed, [Input/output error]
Error Counter logging not supported

Device does not support Self Test logging

 

[Chris Moore]

Sadly, the raid card I have installed doesn't support SMART on it.

You built your system on incompatible hardware and now your drives are dying. Get your data out of that system as soon as possible.

It is about to die.

 

[kdragon75]

I don't need hardware raid support since I'm just using JBOD to freenas. Is there a better card I could be using that has SMART support that isn't $300+?

Most of the cards that are recommended fall under this description. I'm using a $30 reflashed Dell card that provides 8 direct ports with full smart and SAS2 support.

 

[Chris Moore]

System info:
Build FreeNAS-11.1-U5
Platform AMD FX(tm)-8320 Eight-Core Processor
Memory 32456MB

Drives are 3TB SATA

This is not what was asked for. Review the rules: https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/
We need the exact model of the computer / system board, model of the drives, etc. Details matter.

Does smartctl have the capability to fix errors like this? If not, I don't have enough space anywhere else to backup my data to. Would I be able to put the drive in a different machine, run the test and potentially fix any errors, then replace it in my freenas box?

No, smartctl does not fix. SMART is for monitoring the drive health and would tell you when your drive is going bad, that way you can replace it before it goes bad. Your unsupported, incompatible, should never have used it, hardware RAID card masked that information from you and now you have three drives that are unhealthy enough that ZFS has marked them as degraded.

I don't have enough space anywhere else to backup my data to.

Better find some place or you will loose all your data. This system is just lucky it hasn't totally died already. You are living on borrowed time.

Would I be able to put the drive in a different machine, run the test and potentially fix any errors, then replace it in my freenas box?

Most likely no, because you mapped those drives through a hardware RAID controller and the way they are showing up is like you set each drive up as a separate RAID-0 in the RAID controller. Most likely, if you put one of those drives on a different controller, it would show up as blank because the data is being arranged on the disk by the RAID controller and that arrangement is not necessarily compatible with being accessed directly the way it should have been from the start.

Raid card being utilized: https://www.amazon.com/gp/product/B0050SLTPC/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

I don't need hardware raid support since I'm just using JBOD to freenas. Is there a better card I could be using that has SMART support that isn't $300+?

The time to ask that question was before you built the system and put data on it. If that card was doing true hardware pass through, you would be able to get SMART data from the drives. It isn't even showing what kind of drive you have. The header of the output should look like this:

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate Desktop HDD.15
Device Model:	 ST5000DM000-1FK178
Serial Number:	xxxxxxxx
LU WWN Device Id: 5 xxxxxx xxxxxxxxx
Firmware Version: CC49
User Capacity:	5,000,981,078,016 bytes [5.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5980 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Jul 14 14:56:26 2018 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

The kind of controller you should have used:
https://www.ebay.com/itm/Genuine-LS...i-P20-IT-Mode-ZFS-FreeNAS-unRAID/162958581156

PS. Those 3 drives probably each have a massive number of bad sectors that FreeNAS would have warned you about if it had been able to see the drives.
If they are in warranty, you may be able to send them in for replacement, but if they are not in warranty, you might want to start planning to buy some new drives. Once you see bad sectors, it is time for the drive to go away.

 

[Chris Moore]

I've been seeing the following messages for at least 2 weeks now, but I've been unable to find any specific information about what's going on.

Why did it take you two weeks to ask a question? Asking questions is how you get answers.

 

[Castertr0y357]

You built your system on incompatible hardware and now your drives are dying. Get your data out of that system as soon as possible.

It is about to die.

I don't see how I used incompatible hardware. According to the freenas documentation, my raid card is supported:

[i386,amd64] The hpt27xx(4) driver supports the following SAS controllers:

• HighPoint's RocketRAID 272x series

I have the rocketraid 2720 card installed.

While it may not have been the best choice for a card, I went with it because it was relatively inexpensive, was highly rated, and thought it covered me for everything I needed/wanted to do.

I swapped out the chassis about 2 years ago, and found out that the cable that was connecting the SAS card to the HD enclosure would not work. I had to connect my drives directly to the motherboard via individual SATA cables, and the system booted, and recognized the zfs info. ALL of my data/configuration was still there. My card, while being a hardware raid card, was doing JBOD pass-through and allowing the OS to see the individual disks instead of masked raid-0 pools. I can see the serial numbers for each disk I have connected in the GUI.

I still don't see why I wouldn't be able to pull out the drive, plug it into a different system and have a program like HDD regenerator scan through and repair any bad sectors. Worst case, I should be able to wipe the disk and clean the bad sectors while I'm at it, then plug the disk back in and have it rebuild from parity.

 

[Castertr0y357]

Why did it take you two weeks to ask a question? Asking questions is how you get answers.

Because everything was still up, and I don't have an abundance of free time. Plus, I don't have anything on this server that is critical.

 

[Chris Moore]

I don't see how I used incompatible hardware. According to the freenas documentation, my raid card is supported:

If it is able to work with FreeNAS, then you misconfigured the card and the way the drives are showing up is not correct.
If a card can't pass SMART status, it isn't working.

Because everything was still up, and I don't have an abundance of free time. Plus, I don't have anything on this server that is critical.

Then, you are fine with loosing it all and having to build it back from scratch or do without. Great. No problem.

I swapped out the chassis about 2 years ago, and found out that the cable that was connecting the SAS card to the HD enclosure would not work. I had to connect my drives directly to the motherboard via individual SATA cables, and the system booted, and recognized the zfs info. ALL of my data/configuration was still there.

That is great. For testing, you should be able to plug the three problem disks into the system board and run the SMART test on them to find out how bad off they are.

I still don't see why I wouldn't be able to pull out the drive, plug it into a different system and have a program like HDD regenerator scan through and repair any bad sectors.

Try that at your own risk. My view, a drive with bad sectors isn't worth wasting the time on.

 

[Castertr0y357]

If it is able to work with FreeNAS, then you reconfigured the card and the way the drives are showing up is not correct.
If a card can't pass SMART status, it isn't working.

I've looked through the options for this card, and it is not there. Even the site itself does not list SMART capability. Freenas has the drivers to work with them because the project was designed to allow all sorts of hardware configurations to work together to run the OS.

http://highpoint-tech.com/USA_new/series_rr272x_specification.htm

 

[Chris Moore]

I've looked through the options for this card, and it is not there. Even the site itself does not list SMART capability. Freenas has the drivers to work with them because the project was designed to allow all sorts of hardware configurations to work together to run the OS.

http://highpoint-tech.com/USA_new/series_rr272x_specification.htm

Just because there is a driver for BSD (the underlying OS) does not mean that it works properly. If it can't pass the SMART data so that your FreeNAS is able to run SMART tests and tell you when you have developed bad sectors, it is not doing the job it needs to do.
There is no argument here. If it can't do SMART data, it isn't good enough. Sell it on eBay to someone that will use it as a hardware RAID card and get a proper SAS HBA like the one I linked to.

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

 

[kdragon75]

program like HDD regenerator scan through and repair any bad sectors.

You don't fix bad sectors. You remap them to a reserved part of the disk. This is all tracked and logged at negative sector numbers that are more or less inaccessible. Or at least that used to be the case... Maybe its all in flash now? Anyway... The issue with increasing bad sector counts is that it not always bad sectors. It's can be a bad head or armature and if that's the case, all the data could go poof in an instant. You mention you don't have a lot of free time, perhaps this will be a painful and time consuming lesson that more time upfront on research and planning can save a LOT of time and trouble later.

As @Chris Moore said, Just because is works doesn't mean you should. FreeNAS is based on FreeBSD but it is not FreeBSD and has different recommendations for reliability.

 

[Castertr0y357]

So, continuing to look at this, I did a test. I pulled one of the drives out and plugged into another computer and did a scan of all the sectors to see if there were any actual bad sectors on the disk. The scan came up with 0. I've had drives fail on me, but I've never seen 3 drives have a simultaneous I/O error before. The probability of that happening is incredibly slim.

Looking at the zpool status -v command after plugging the drive back in shows all of the disks online, and none of the degraded:

Code:

root@freenas:~ # zpool status -v Vol1
  pool: Vol1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 111M in 0 days 00:00:07 with 0 errors on Sun Jul 15 00:19:52 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		Vol1											ONLINE	   0	 0	 9
		  raidz1-0									  ONLINE	   0	 0	18
			gptid/f485c881-655f-11e5-a92e-7427ea01dce4  ONLINE	   0	 0	 0
			gptid/f5aa005d-c333-11e5-9df4-7427ea01dce4  ONLINE	   0	 0	 3
			gptid/a1dc9873-4575-11e6-ba5f-7427ea01dce4  ONLINE	   0	 0	 0
			gptid/cb7dc330-66a9-11e4-82ff-7427ea01dce4  ONLINE	   0	 0	 0
			gptid/77e26c0e-2ac4-11e7-8976-708bcd7c4011  ONLINE	   0	 0	 0

errors: Permanent errors have been detected in the following files:

		/var/db/system/syslog-7d0973d92e38411da7ce5f40724800c8/log/daemon.log

What's far more likely that happened was that the power went off on the server while it was in the middle of a write, and that daemon.log file became corrupted. Since it's only a log file, I'm inclined to attempt re-creating the file and seeing if that resolves my issue.

Can I successfully do that without causing the system to panic?

 

[Chris Moore]

Are you able to get SMART status from the disks in FreeNAS so the system can monitor their health?

If you change where the log is stored, it should fix your log file issue, but you still have a disk controller issue and if you don't have a UPS, you should really fix that too.
http://doc.freenas.org/11/system.html#system-dataset

 

[Castertr0y357]

Are you able to get SMART status from the disks in FreeNAS so the system can monitor their health?

but you still have a disk controller issue and if you don't have a UPS, you should really fix that too.
http://doc.freenas.org/11/system.html#system-dataset

I didn't mess with the cabling since I don't have enough free time to re-cable all the disks. I've had disk failures before, and I'm always notified pretty quickly. The only time I've ever run into multiple disk failures is when I was using Seagate disks in the system.

I still wouldn't call it a disk controller issue, but I am looking to replace the raid card with an HBA. As far as a UPS goes, I'm also working on that too. Since I work with this stuff on a budget, I don't get a lot of money to put into this.

 

[kdragon75]

in the middle of a write, and that daemon.log file became corrupted

ZFS is copy on write. The file may not have the expected contents but it would not be corrupted on the file system. Nor would that cause checksum errors.

Since I work with this stuff on a budget

Dell Perc H310 cards are great. I'm using one that I got for $30 USD. You do need to cross flash it but that only take 20 min. As you have not mentioned your full system design I will just warn about tower cases with server HBAs; they often have airflow requirements the exceed what is found in most tower cases. Also this air is intended to be channeled across high heat components like HBAs. If you running a tower an extra fan MAY be needed. Monitor temps and proceed as indicated.

 

[Castertr0y357]

I was actually able to read the contents of the file itself, up until it had the IO message in it. When I recreated the file, and destroyed the old one, I no longer had errors, and I haven't seen any other checksum errors. I have hit previous data errors that have been caused due to power failures during write, so it would make sense that this could have been a similar cause.

I do have a Norco server chassis that I am using for this, so I have plenty of airflow, but I'll look into those Dell HBAs as a replacement. Ideally, I'd like to get enough to allow me to fill out the rest of my HDD bays. I should also look for a UPC to help in the event of power outages, but that's going to be a bit of time.

I do appreciate the help though. Thanks.

 

[Chris Moore]

I have hit previous data errors that have been caused due to power failures during write, so it would make sense that this could have been a similar cause.

You shouldn't have data errors due to a power loss. That must be a feature of the RAID card you are using.

Ideally, I'd like to get enough to allow me to fill out the rest of my HDD bays.

You only need one SAS controller. They are good for up to 256 drives. After that, you need SAS expanders like this one:
https://www.ebay.com/itm/Intel-24-P...der-Card-RES2SV240-PBAE91267-203/401561865884

How many bays does your chassis have?