SOLVED Can't get a drive online and happy, relatively new install

[Royce Rezendes]

Error Information:

I recently purchased new hardware and transferred my Working hard drives (sata) from a large pc tower into a much smaller NAS case. I have my installation running RaidZ1 (just movies and tv shows) with all the plugins I want running fine, but I cannot seem to get this one drive to cooperate. I have researched for like half a day or longer trying to resolve this and so far I have nothing to show for it. I have tried many zpool commands such as "zpool replace/online/offline/detach" etc... I did a replace operation at one point which most of the time that option doesn't show a drive on the list of replacements to choose from (this might be because I would be "replacing" it with itself?), the Replace succeeded but the drive wasn't online/wouldn't go online. I did a scrub, which didn't seem to change anything either. I have verified all my physical connections including reseating all drives and sata power/data cables. Last thing I did (just now) was Offline the drive and reboot the server, it now shows all drives online and Resilvering in progress (I'll edit this post with the results). It would be awesome if someone could help me figure this out.

Drive in question:
-Seagate ST3000DM001-1CH166 CC24
-connected to ada4 (sata)
-serial W1F2Y92S

smartctl -a:

I tried to get smartctl command to give me any valuable output but all I could get was "No Match." or "/dev/ada4: Unable to detect device type"

Debug.log: Following the Forum Rules post, I couldn't locate the debug file (maybe instructions are for an older version?). Here's the tail end of a huge file I found inside the archive named debug.log...

Code:

Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = iface:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv4: 200.200.200.6/24
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = autostart: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = vnet: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = nat: Disabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = mac: 02:42:f1:00:08:0b
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = status: Running
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = type: pluginjail
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = flags: allow.raw_sockets=true
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:221] cmd_pipe.__init__: leave
Mar 30 00:33:21 roycenas manage.py: [common.warden:460] warden_base.run: leave
Mar 30 00:33:21 roycenas manage.py: [freeadmin.navtree:593] Plugin plexmediaserver requested to be appended to plugins
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = id: 1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = host: plexmediaserver_1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = iface:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv4: 200.200.200.6/24
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = autostart: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = vnet: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = nat: Disabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = mac: 02:42:f1:00:08:0b
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = status: Running
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = type: pluginjail
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = flags: allow.raw_sockets=true
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:221] cmd_pipe.__init__: leave
Mar 30 00:33:21 roycenas manage.py: [common.warden:460] warden_base.run: leave
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = id: 1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = host: plexmediaserver_1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = iface:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv4: 200.200.200.6/24
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = autostart: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = vnet: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = nat: Disabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = mac: 02:42:f1:00:08:0b
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = status: Running
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = type: pluginjail
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = flags: allow.raw_sockets=true
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:221] cmd_pipe.__init__: leave
Mar 30 00:33:21 roycenas manage.py: [common.warden:460] warden_base.run: leave
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = id: 1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = host: plexmediaserver_1
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = iface:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv4: 200.200.200.6/24
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv4:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = alias-bridge-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = defaultrouter-ipv6:
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = autostart: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = vnet: Enabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = nat: Disabled
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = mac: 02:42:f1:00:08:0b
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = status: Running
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = type: pluginjail
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out = flags: allow.raw_sockets=true
Mar 30 00:33:21 roycenas manage.py: [common.cmd:210] cmd_pipe.__init__: out =
Mar 30 00:33:21 roycenas manage.py: [common.cmd:221] cmd_pipe.__init__: leave
Mar 30 00:33:21 roycenas manage.py: [common.warden:460] warden_base.run: leave

EDIT1:

Well, I feel a little silly now... Here's the result after the Resilver it automatically started:

Although all the drives are now Online, the error above is now concerning me. How can I resolve/clear this error?

 

[Royce Rezendes]

Well... I just ran a zpool clear on that disk and everything seems ok. Thanks for the help everyone lol.

 

[Robert Trevellyan]

I hope you realize that if ada5 fails you lose your entire pool.

 

[Royce Rezendes]

I was under the assumption that there are 3 separate entities there, a mirror of 2 drives, a raidz1 of 5 drives, and a "stripe" (one drive left over). Are you saying that the lone 4TB drive in the "stripe" is somehow attempting to stripe the data with my 15TB raidz1? If that's true, I need to reconfigure... I have the main raidz1 consisting of 5x3TB drives, I wanted to add the 4TB drive to it but it wouldn't allow me, can I partition the 4TB into a 3TB in order to use it as a 6th drive in my raidz1? I'm not too worried about losing this data as I said earlier it's just movies and tv shows.

 

[gpsguy]

Yes, that's correct. No, you can't fix it by trying to partition it. It needs to stay where it is - you can't move it to the RAIDz1 vdev. Once a vdev is created, you can't add disks to it, other than one exception. That exception, and applicable in your case, would allow you to add another disk and mirror it the lone 4TB drive. To do the latter, you'd need to do it from the command line. Search the forum for details on how to do it.

 

[Robert Trevellyan]

You have one pool made of 3 vdevs. In order for a pool to remain healthy, all its vdevs must remain healthy. If one vdev fails, the entire pool is lost. Also, you can't just add a drive to a vdev. This is all covered in cyberjock's guide. Thank goodness you're not too worried about losing this data.

 

[Royce Rezendes]

Thank you for educating me on that Robert. @gpsguy, would I be able to delete the volume and start over using 5x3TB drives and 1x4TB in a RaidZ1? Then I would just not include the smaller drives of course.

 

[gpsguy]

If you start over and don't want to start over, down the road, I'd suggest creating a new volume using the 6 disks in RAIDz2. The 4TB drive would function as a 3TB drive, but RAIDz2 is recommended for vdev's containing large drives. A concern, is that should a drive fail and you replace it, during the time the resilvering takes place, another drive might fail. RAIDz2 offers additional redundancy over RAIDz1.

If you wanted to keep the 2TB drives, you could create a second volume, containing a mirrored set.

 

[Royce Rezendes]

What is the secret method to allow the 4TB drive to be added together with the 3TB drives? I tried to add it and FreeNAS wouldn't allow it. I read about RaidZ and how exactly what you are explaining could happen but ultimately given the data I am storing, I feel fine with the risky RaidZ1. Do you mean a second Pool for the 2TB drives? Right now I have them setup as a mirror vdev correct? (Are vdev/volume basically same?)

 

[gpsguy]

There is no secret method that would have allowed you to add the 4TB drive to your existing 3TB drives, after the fact. That's why you ran into problems. But, now that it's been striped into the volume, you can't move it, without destroying your pool and starting over. That's why I said that you could add another 4TB drive (from the command line) and add the 2nd one as a mirror.

But, since you said you were willing to start delete the volume (all 3 vdevs) and start over, I offered another suggestion.

No, vdev and volume aren't the same thing. You currently have one volume (volMain) consisting of 3 vdev's (RAIDz1, mirror, and stripe (single disk)].

Yes, what I was suggesting, if you wanted to reuse the 2TB drives, was to create a second pool/volume, containing just the 2 disks in a mirror. Or live dangerously and stripe them. There wouldn't be any redundancy. Since this would be separate pool, if anything happened to it, it wouldn't affect the 6 disk volume.

 

[Royce Rezendes]

I was actually asking about what is the method to allow freenas to create the new RaidZ volume with the 4TB drive and the 3TB drives together from scratch (after I delete my pool and start over). Because what I was saying earlier is that freenas doesn't seem to simply allow you to add the 4TB drive with all the 3TB drives due to the size not matching, I tried this for a little bit and attempted to look up how it could be done but didn't find a definitive answer. Thanks for the clarification on volumes/vdevs.

 

[danb35]

To create a RAIDZ* vdev with mismatched disks, you'll need to go into "manual mode" in the volume manager. You also had to do that to add the single disk as a stripe.

 

[Royce Rezendes]

Awesome, thank you. Looks like I'll be ordering a few 3TB WD Reds and restarting again. I'm trying to figure out right now if this drive (same one from earlier) is failing, it shows Checksum numbers on the "Volume Status" window in the GUI. They eventually reappear when I clear them... I ran a short and long SMART test:


Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 10905 4101589432
# 2 Short offline Completed: read failure 90% 10905 4101589136

How can I tell if I should replace this drive?

 

[Royce Rezendes]

 

[danb35]

How can I tell if I should replace this drive?

Two consecutive SMART test failures is enough for me. The 184 offline uncorrectable sectors are just extra confirmation. You really need to set up regular SMART tests, too--the two you did today were the first it's had in over a year of nonstop operation. You also need to set up an email address for SMART alerts--you should have gotten an email a long time ago warning you about this disk based on the offline sectors.

The drive's dead. RMA it.

 

[Royce Rezendes]

Crap. Yeah I just built this NAS from a regular pc tower I had with JBOD. I saw the "check interval" in FreeNAS GUI S.M.A.R.T. section was default set to 30 minutes, this doesn't actually setup S.M.A.R.T. tests every 30 minutes?

 

[Ericloewe]

Crap. Yeah I just built this NAS from a regular pc tower I had with JBOD. I saw the "check interval" in FreeNAS GUI S.M.A.R.T. section was default set to 30 minutes, this doesn't actually setup S.M.A.R.T. tests every 30 minutes?

No, it polls the SMART data every 30 minutes.

 

[ser_rhaegar]

Crap. Yeah I just built this NAS from a regular pc tower I had with JBOD. I saw the "check interval" in FreeNAS GUI S.M.A.R.T. section was default set to 30 minutes, this doesn't actually setup S.M.A.R.T. tests every 30 minutes?

No that checks the SMART info every 30 minutes.

You need to setup tests in: Tasks -> SMART Tests

 

[danb35]

See the manual for more information on setting up SMART tests. But, again, you should have gotten an emailed warning as soon as the bad sectors came up (and I can't imagine that 184 of them came up at the same time). Make sure you've got email configured correctly on your server. If you've set up your server correctly, things like this shouldn't happen.