Drive Failure, won't boot unless I remove the drive

[stupes]

Ps. I read the manual when I set up, 4 years ago, but one does not tend to retain that info without rehearsal ...

 

[Chris Moore]

Here is that output, I am a bit unsure what to do next. Is my drive dead? or Does it need reformatted or something? TIA:

Based on the output of your zpool status -v there are only two drives involved in the storage pool and they are in a "striped set", not in any kind of redundant pool type. If you had redundancy, you wouldn't have the errors you do with files being marked as damaged or lost. The third drive that you appear to think was part of the pool was never involved in the storage in any way and we don't know for sure if it is good or bad.

I imagine that the one with all the bad sectors is the cause of your data loss, but I have no way to know for sure without further testing.

If you don't want to loose the data stored in this NAS, I would suggest getting that data moved somewhere else. If these drives are four years old and one has a significant number of bad sectors, it is time it was replaced and the pool definitely needs to be reconfigured.

 

[stupes]

Agreed thanks. Getting the data off will be number 1 priority. I get what you mean about Raid0 now from your article above. Striped does not equal redundant! I guess I picked it because it was the only option available at the time. My data has always been stored in multiple places and tbh performance is not a priority for me. Time for a bit of re-architecting.

 

[stupes]

Excuse me for this, just one last question. I get that Raid 0 is non redundant now, but I wondered why by 2x2TB disks only give a 2.3TB volume (and not a 4TB)?

 

[danb35]

I wondered why by 2x2TB disks only give a 2.3TB volume (and not a 4TB)?

Where are you seeing that? And if it's the "available" column on the Storage page, you need to add the "Used" amount to it.

 

[droeders]

PS. Only having room for 3 disks doesn't give you enough room to build a proper array of any kind.

What about a 2-disk (or 3-disk) mirror?

 

[Chris Moore]

What about a 2-disk (or 3-disk) mirror?

I suppose that my statement could be considered a matter of opinion, while a mirror (2 way or 3 way) is perfectly legitimate, I personally feel it is wasteful of resources. If I wanted 12 TB of usable storage, for example, I would never consider using a mirror of two 12 TB drives. To me, that would be an insufficient level of redundancy (I have had two drives in the same vdev fail at the same time) and a 3 way mirror would be wasteful as you would expend 36 TB of storage to have two disks of redundancy and realize less than 12 TB of actual usable storage.
What I would do is use 6 x 4 TB drives in RAIDz2, which would give me two drives of redundancy, and it would give me 14 TB of capacity with about 11 TB of capacity after allowing for the 20 % space reservation for copy-on-write.
To me, it makes more sense to buy a larger chassis and mount more drives. Not everyone thinks like me, and that is fine. I didn't come here to argue. I have a 48 bay chassis for my primary home NAS and a 24 bay chassis for my backup NAS and until about a week ago I had the 48 bay chassis completely filled with drives. Now there are only 28 in there.

 

[stupes]

I only need 2TB ;) I was thinking I could get something to meet that requirement with 3x 2TB drives including redundancy, and this is what I will be interrogating the manual to work out hopefully.

 

[pro lamer]

I would never consider using a mirror of two 12 TB drives. To me, that would be an insufficient level of redundancy (I have had two drives in the same vdev fail at the same time)

I, a noob, would since from what I've read mirror rebuilds quite quick comparing to raidz1 or raidz2. But I wolud consider only if it wasn't Especially SMR which I've heard/read are slow to write (context: lots of data; anyway I've read in our forums that SMR are discouraged for NASes at all).

3x 2TB drives including redundancy

Since it means raidz1 - some in our forums consider 2TB per disk the upper bound for raidz1 and some consider 1TB per disk as this bound... (because of the 2nd disk failure risk higher for larger drives and resilvering raidz1 slower than rebuilding a mirror).

Thus using 3x2TB drives in raidz1 may limit your upgrade paths - it's not recommended to replace such drives with drives bigger than the 1/2TB bound in raidz1 (which might be an upgrade path in other setups). EDIT: If the motherboard allows.

I guess the OP is happy with the SSD as a boot drive so I hesitate advising switching to USB boot - USB boot might not be as reliable as SSD... Some in our forums are happy with USB boot drives anyway...

On the other hand:

old HP Proliant ML115 box with 4 sata ports and 1 IDE

maybe some additional HBA card can be considered? (what generation server is this? Does it have more than one 5.25" bay and a 2x5.25" -> 3x3.5" adapter can be bought and mount?)

 

[stupes]

I guess the OP is happy with the SSD as a boot drive so I hesitate advising switching to USB boot - USB boot might not be as reliable as SSD... Some in our forums are happy with USB boot drives anyway...

I had multiple USB drive failures so went with a small SSD SATA drive - I wish I had used the IDE port for that now perhaps.

maybe some additional HBA card can be considered? (what generation server is this? Does it have more than one 5.25" bay and a 2x5.25" -> 3x3.5" adapter can be bought and mount?)

2x 5.25 available. HBA? Is that not a Fibre Card?

 

[Chris Moore]

2x 5.25 available. HBA? Is that not a Fibre Card?

No, a SAS HBA has nothing to do with fiber.

I only need 2TB ;) I was thinking I could get something to meet that requirement with 3x 2TB drives including redundancy, and this is what I will be interrogating the manual to work out hopefully.

If you use three 2 TB drives in RAIDz1, it will give you about 2.7 TB of usable space after reserving 20% for the copy on write feature of ZFS. If you use RAIDz2, with four 2 TB drives, it will give you almost exactly the same capacity but with two drives of redundancy. The recommendation has been (for years) that drives larger than 1 TB should not be used in RAIDz1, but if you are only doing this as a backup and you have other backups of the same data, I suppose the risk of data loss is minimal and RAIDz1 would be acceptable even with 2 TB drives.

 

[Chris Moore]

I, a noob, would since from what I've read mirror rebuilds quite quick comparing to raidz1 or raidz2.

This is something that is only sometimes true. No drive, regardless of the kind of pool it is in, can rebuild faster than the drive is able to write data. If a drive in a RAIDz pool can write data at 100MB/s, and a drive in a mirror can write at the same speed, the only way the RAIDz rebuild is going to run slower is if the system is under-powered such that the computer can't keep up with the calculations needed for processing the parity. If you computer is sufficiently powerful to process the parity calculations, the RAIDz pool will rebuild just as fast as a mirror. I can resilver a drive in my RAIDz2 array in around an hour. The long rebuilds that some people experience is due to trying to build a low-power system that doesn't have enough processing capability.

 

[pro lamer]

I can resilver a drive in my RAIDz2 array in around an hour.

Good to hear this. I guess I must have assumed (and must have been not aware of this assumption) that mirror rebuilds relied on large block copies and raidz resilvers - on lots of IOPS. Which appears not to be true...

Sent from my mobile phone

 

[Chris Moore]

Good to hear this. I guess I must have assumed (and must have been not aware of this assumption) that mirror rebuilds relied on large block copies and raidz resilvers - on lots of IOPS. Which appears not to be true...

In ZFS, a mirror vdev is not recovered by simply copying the contents of one disk to the other. In any resilver ZFS 'walks the tree' and checks all the data against all the checksums, so the process of reading the data would be limited (in a mirror) by how fast the source disk can read. The data is then written back out to the target disk with very little processor overhead, but the speed that the target disk can write is still a limiting factor.
When rebuilding a RAIDz(1 , 2, or 3) vdev, to 'walk the tree' involves reading from all the functioning disks, which means each source disk is working less than the source disk in a mirror, because that work is divided among the number of disks in the vdev and they only need to read fast enough to keep up with the maximum write capability of the target disk. The limiting factor on speed is partly the speed that the target disk can write at but it is also the speed that parity can be recalculated. The file-system (ZFS) must read the data in and calculate what should be on the disk being replaced, then write that data out with the correct checksum data. So, processor utilization during a rebuild will usually be high and if the system has a low powered processor, that could be a bottleneck.
If you have a sufficiently powerful CPU that the calculation of parity and checksum data is not a bottleneck, the speed that the replacement drive is able to write data is the only limiting factor.
Lets look at total data stored though, because that is the big factor with RAIDz and a rebuild. If you have a single mirror of two 4 TB drives with 1 TB of data on it, the entire 1 TB of data must be read and written for the resilver to complete. If you have a RAIDz2 vdev with six dives, you need to read that entire 1TB of data but you are reading from five drives, so you only need to read somewhere around 256 GB of data from each drive because the entire 1 TB of data is not on any one drive. So it should be pretty obvious that 256 GB of data can be read from a drive faster than 1 TB of data can be read. These reads happen in parallel, so each drive may only be reading at 20 or 30 MB/s. Then when you are writing the parity and checksum data back out to the replacement drive, you are not needing to write the entire 1 TB either, only the fraction of 1 TB that is supposed to be on that drive, probably around 256 GB. It should be obvious that 256 GB of data will write out to a drive much faster than 1 TB of data. It would be my estimation that a system with six drives in RAIDz2 could recover from a drive replacement in one quarter the time that a system using a single mirror could recover as long as the quantity of data were equivalent between the two configurations.
People have made some erroneous assumptions about the benefits of mirror vdevs. Mirrors have their place, but for most home users I disagree that they are preferable to RAIDz2. The thing that usually entices the home user to use mirrors is the fact that they can increase capacity by simply adding two more disks. It makes expansion easy, but when you get to the point where you have six disks in mirrors, you have also reached a point where the capacity of two of those disks is being wasted. Planning ahead makes more sense to me.
I have suggested to people on the forum that have sought advice, that in a situation where the user already has four 8 TB drives, it would be better to use a couple of older 2 TB drives to complete a six drive RAIDz2 array, instead of using those drives in mirrors. That will give them an initial 6 ish TB of capacity and they can replace the 2 TB drives when funds allow to expand capacity to around 14 TB. It is better than locking yourself into a pool of mirrors. Options are many and which one to choose depends on what you want to be able to do with the storage in the future.

 

[stupes]

No, a SAS HBA has nothing to do with fiber.

If you use three 2 TB drives in RAIDz1, it will give you about 2.7 TB of usable space after reserving 20% for the copy on write feature of ZFS. If you use RAIDz2, with four 2 TB drives, it will give you almost exactly the same capacity but with two drives of redundancy. The recommendation has been (for years) that drives larger than 1 TB should not be used in RAIDz1, but if you are only doing this as a backup and you have other backups of the same data, I suppose the risk of data loss is minimal and RAIDz1 would be acceptable even with 2 TB drives.

Hi Chris

I have created a 3 disk raidz1 I believe, however I seem to have a lot more storage thank 2.7TB which concerns me as I may have done myself out of redundancy again ;)

Code:

[root@stu-nas1 ~]# zpool status -v | more
  pool: data1
 state: ONLINE
  scan: none requested
config:
		NAME											STATE	 READ WRITE CKSUM
		data1										   ONLINE	   0	 0 0
		  raidz1-0									  ONLINE	   0	 0 0
			gptid/97c83d74-d1fe-11e8-abcd-001e0bc68414  ONLINE	   0	 0 0
			gptid/9a598ecb-d1fe-11e8-abcd-001e0bc68414  ONLINE	   0	 0 0
			gptid/9bb4bea8-d1fe-11e8-abcd-001e0bc68414  ONLINE	   0	 0 0
errors: No known data errors
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:01:25 with 0 errors on Wed Oct 17 03:46:27 2018
config:
		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  ada3p2	ONLINE	   0	 0	 0
errors: No known data errors

On selecting the raid type I could only choose raidz from the GUI. and in the pools section of the GUI it reports 7.43TiB, 5.44TiB free, and if I click the drop down there it says,
used = 484.86 KiB
available = 3.51 TiB
Which is curious.

Am I in the correct ball park?

 

[Chris Moore]

The "Practical usable storage" number at the bottom is how much you should be able to put in the pool without going over the 80% point that I think we discussed above. Here is a link to this calculator: https://wintelguy.com/zfs-calc.pl

There are some things that you should read, because I don't see where they were discussed above:

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[stupes]

View attachment 26161

There are some things that you should read, because I don't see where they were discussed above:

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Thanks for this I read through these after you pasted them in earlier in the thread (that's why I've been away being busy, lol)

It seems that what I have then is good to go, I do not think I have COW set up, but I am ok with this. I did not see an option to set it up as I went along

 

[Chris Moore]

I do not think I have COW set up

That is a feature of the file system, you can't use ZFS and not use copy-on-write, and you can't use FreeNAS without using ZFS.

 

[stupes]

Understood, thanks again