zpool import I/O error

Tweako

Cadet
Joined
May 14, 2017
Messages
6
Hello,

I get a zpool I/O error on my RocketRaid volume which are 4 Samsung EcoGreen 1,5TB disks on my Highpoint RocketRAID 2310.

A while back i get the status:
One or more devices has experienced an error resulting in data corruption. Applications may be affected.
A few hours later I get the message:
Unscheduled system reboot
And now I have the Alert:
The volume RocketRaid state is UNKNOWN

I have read some forum posts and tried to import the zpool but then I"ll get:
Code:
root@freenas:~ # zpool import RocketRaid
cannot import 'RocketRaid': I/O error
		Destroy and re-create the pool from
		a backup source.


So I'll hope I can get some help with this :(.

Here is some information:
Code:
root@freenas:~ # zpool status
  pool: ZFS_storage
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 04:15:28 with 0 errors on Sun Oct 28 03:15:30 2018
config:
		NAME											STATE	 READ WRITE CKSUM
		ZFS_storage									 ONLINE	   0	 0	 0
		  raidz1-0									  ONLINE	   0	 0	 0
			gptid/5a19fa81-3bf2-11e7-9c49-bc5ff4f53749  ONLINE	   0	 0	 0
			gptid/5afabe95-3bf2-11e7-9c49-bc5ff4f53749  ONLINE	   0	 0	 0
			gptid/5bcdfe42-3bf2-11e7-9c49-bc5ff4f53749  ONLINE	   0	 0	 0
			gptid/5c901953-3bf2-11e7-9c49-bc5ff4f53749  ONLINE	   0	 0	 0
			gptid/5d578c9d-3bf2-11e7-9c49-bc5ff4f53749  ONLINE	   0	 0	 0
errors: No known data errors
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:36 with 0 errors on Sat Nov 17 03:47:36 2018
config:
		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da1p2	 ONLINE	   0	 0	 0
errors: No known data errors


Code:
root@freenas:~ # zpool import
   pool: RocketRaid
	 id: 16390382078416128707
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
		some features will not be available without an explicit 'zpool upgrade'.
 config:
		RocketRaid									ONLINE
		  gptid/a01600bd-d2f3-11e7-bb4b-bc5ff4f53749  ONLINE


Code:
root@freenas:~ # camcontrol devlist
<KINGSTON SA400S37120G SBFK61K1>   at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD20EZRX-00DC0B0 80.00A80>	at scbus3 target 0 lun 0 (pass1,ada1)
<WDC WD20EZRX-00D8PB0 80.00A80>	at scbus4 target 0 lun 0 (pass2,ada2)
<WDC WD20EZRX-00D8PB0 80.00A80>	at scbus5 target 0 lun 0 (pass3,ada3)
<WDC WD20EZRX-07D8PB0 80.00A80>	at scbus6 target 0 lun 0 (pass4,ada4)
<WDC WD20EZRX-00D8PB0 80.00A80>	at scbus7 target 0 lun 0 (pass5,ada5)
<HPT DISK 0_0 4.00>				at scbus9 target 0 lun 0 (pass6,da0)
<SanDisk Ultra Fit 1.00>		   at scbus10 target 0 lun 0 (pass7,da1)


Code:
root@freenas:~ # gpart show
=>	   40  234441568  ada0  GPT  (112G)
		 40		 88		- free -  (44K)
		128	4194304	 1  freebsd-swap  (2.0G)
	4194432  230247168	 2  freebsd-zfs  (110G)
  234441600		  8		- free -  (4.0K)
=>		40  3907029088  ada1  GPT  (1.8T)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834688	 2  freebsd-zfs  (1.8T)
  3907029120		   8		- free -  (4.0K)
=>		40  3907029088  ada2  GPT  (1.8T)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834688	 2  freebsd-zfs  (1.8T)
  3907029120		   8		- free -  (4.0K)
=>		40  3907029088  ada3  GPT  (1.8T)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834688	 2  freebsd-zfs  (1.8T)
  3907029120		   8		- free -  (4.0K)
=>		40  3907029088  ada4  GPT  (1.8T)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834688	 2  freebsd-zfs  (1.8T)
  3907029120		   8		- free -  (4.0K)
=>		40  3907029088  ada5  GPT  (1.8T)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834688	 2  freebsd-zfs  (1.8T)
  3907029120		   8		- free -  (4.0K)
=>		40  8790343600  da0  GPT  (4.1T)
		  40		  88	   - free -  (44K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  8786149200	2  freebsd-zfs  (4.1T)
  8790343632		   8	   - free -  (4.0K)
=>	  40  30031176  da1  GPT  (14G)
		40	  1024	1  freebsd-boot  (512K)
	  1064  30030144	2  freebsd-zfs  (14G)
  30031208		 8	   - free -  (4.0K)


Code:
root@freenas:~ # gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 234441607
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   efimedia: HD(1,GPT,a4e50f41-d6f6-11e8-9035-bc5ff4f53749,0x80,0x400000)
   rawuuid: a4e50f41-d6f6-11e8-9035-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 117886550016 (110G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   efimedia: HD(2,GPT,a4ea103c-d6f6-11e8-9035-bc5ff4f53749,0x400080,0xdb94b00)
   rawuuid: a4ea103c-d6f6-11e8-9035-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 117886550016
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 234441599
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 120034123776 (112G)
   Sectorsize: 512
   Mode: r0w0e0
Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,5a0401b8-3bf2-11e7-9c49-bc5ff4f53749,0x80,0x400000)
   rawuuid: 5a0401b8-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 1998251360256 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,5a19fa81-3bf2-11e7-9c49-bc5ff4f53749,0x400080,0xe8a08800)
   rawuuid: 5a19fa81-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251360256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,5ae393e3-3bf2-11e7-9c49-bc5ff4f53749,0x80,0x400000)
   rawuuid: 5ae393e3-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998251360256 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,5afabe95-3bf2-11e7-9c49-bc5ff4f53749,0x400080,0xe8a08800)
   rawuuid: 5afabe95-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251360256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,5bb6d8ff-3bf2-11e7-9c49-bc5ff4f53749,0x80,0x400000)
   rawuuid: 5bb6d8ff-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 1998251360256 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,5bcdfe42-3bf2-11e7-9c49-bc5ff4f53749,0x400080,0xe8a08800)
   rawuuid: 5bcdfe42-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251360256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,5c75eda2-3bf2-11e7-9c49-bc5ff4f53749,0x80,0x400000)
   rawuuid: 5c75eda2-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada4p2
   Mediasize: 1998251360256 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,5c901953-3bf2-11e7-9c49-bc5ff4f53749,0x400080,0xe8a08800)
   rawuuid: 5c901953-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251360256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada4
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada5p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,5d406751-3bf2-11e7-9c49-bc5ff4f53749,0x80,0x400000)
   rawuuid: 5d406751-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada5p2
   Mediasize: 1998251360256 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,5d578c9d-3bf2-11e7-9c49-bc5ff4f53749,0x400080,0xe8a08800)
   rawuuid: 5d578c9d-3bf2-11e7-9c49-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251360256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029119
   start: 4194432
Consumers:
1. Name: ada5
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 8790343639
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   efimedia: HD(1,GPT,9fd0522e-d2f3-11e7-bb4b-bc5ff4f53749,0x80,0x400000)
   rawuuid: 9fd0522e-d2f3-11e7-bb4b-bc5ff4f53749
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da0p2
   Mediasize: 4498508390400 (4.1T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   efimedia: HD(2,GPT,a01600bd-d2f3-11e7-bb4b-bc5ff4f53749,0x400080,0x20bb1ff50)
   rawuuid: a01600bd-d2f3-11e7-bb4b-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4498508390400
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 8790343631
   start: 4194432
Consumers:
1. Name: da0
   Mediasize: 4500655964160 (4.1T)
   Sectorsize: 512
   Mode: r1w1e2
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30031215
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,4635a317-3bf0-11e7-8945-bc5ff4f53749,0x28,0x400)
   rawuuid: 4635a317-3bf0-11e7-8945-bc5ff4f53749
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 1063
   start: 40
2. Name: da1p2
   Mediasize: 15375433728 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   efimedia: HD(2,GPT,46540b11-3bf0-11e7-8945-bc5ff4f53749,0x428,0x1ca3940)
   rawuuid: 46540b11-3bf0-11e7-8945-bc5ff4f53749
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 15375433728
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 30031207
   start: 1064
Consumers:
1. Name: da1
   Mediasize: 15376000000 (14G)
   Sectorsize: 512
   Mode: r1w1e2
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
What's puzzling me here is that you have a pool called ZFS_storage which seems to contain the 4 disks you're talking about.

Then there's the other pool that's having trouble... RocketRaid... which appears to be only a single disk.

Where is your real problem? with this single-disk volume or with the other pool using the 4 disks?

Or is it actually something entirely different and you have presented 4 other disks attached to the RocketRaid card as a single disk, using the RAID function of the card instead of FreeNAS to make a single volume? (this would be the 4.1TB da0 disk?)
 

Tweako

Cadet
Joined
May 14, 2017
Messages
6
My ZFS_storage is fine. I having trouble with my RocketRAID pool.
Which are 4 disks on a Highpoint RocketRAID controller. Freenas sees this as one disk.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
You're not doing it right if FreeNAS can't see each of the disks individually... you will not get much support in this forum using hardware RAID with FreeNAS (but I'm sure you will get plenty of advice to not do that).

You will need to look at the options in the RAID BIOS to see what it's telling you about the RAID set.

If the one (RAID) disk presented to FreeNAS is dead, your pool might be toast.

You could try zpool status -v RocketRaid , which may be able to get the details of the missing/corrupt files.

You would possibly need to do as suggested by zpool import and import the volume using the ID:
zpool import 16390382078416128707

before the zpool status will get you something interesting.

You could also try zpool import with -f or -F options to force it to import...
 
Last edited:

Tweako

Cadet
Joined
May 14, 2017
Messages
6
You're not doing it right if FreeNAS can't see each of the disks individually... you will not get much support in this forum using hardware RAID with FreeNAS (but I'm sure you will get plenty of advice to not do that).

You will need to look at the options in the RAID BIOS to see what it's telling you about the RAID set.

If the one (RAID) disk presented to FreeNAS is dead, your pool might be toast.

You could try zpool status -v RocketRaid , which may be able to get the details of the missing/corrupt files.

You would possibly need to do as suggested by zpool import and import the volume using the ID:
zpool import 16390382078416128707

before the zpool status will get you something interesting.

You could also try zpool import with -f or -F options to force it to import...

Yeah, I know I was doing it wrong...

I have checked the raid config and everything looks fine to me. I think the problem is not the raid but it's freenas.

Thanks for the information. I'll try this later today.
 

Tweako

Cadet
Joined
May 14, 2017
Messages
6
Update:
I tried the following commands with Freenas 11.0 STABLE.

Code:
[root@freenas ~]# zpool status -v RocketRaid                                                                                       
cannot open 'RocketRaid': no such pool                                                                                             
[root@freenas ~]# zpool import 16390382078416128707                                                                                 
cannot import 'RocketRaid': pool may be in use from other system, it was last accessed by  (hostid: 0x290b4bc9) on Tue Nov 20 13:34:
10 2018                                                                                                                             
use '-f' to import anyway                                                                                                           
[root@freenas ~]# zpool import -f 16390382078416128707                                                                             
cannot import 'RocketRaid': I/O error                                                                                               
        Destroy and re-create the pool from                                                                                         
        a backup source.
root@freenas:~ # zpool import -F RocketRaid
cannot import 'RocketRaid': pool may be in use from other system, it was last accessed by  (hostid: 0x290b4bc9) on Tue Nov 20 13:34:10 2018
use '-f' to import anyway
root@freenas:~ # zpool import -f -F RocketRaid
cannot import 'RocketRaid': I/O error
        Destroy and re-create the pool from
        a backup source. 
 

Tweako

Cadet
Joined
May 14, 2017
Messages
6
Still not working...

I booted Windows and checked the raid partition with different tools and everything looks fine.
My last options are trying a other motherboard and raid card.

Does anyone have any other suggestions?
Is there no way to see what the I/O error is?
 
Top