Defective Tank How to remove?

[Harold Roberts]

OK so I have a system with 3 tanks. Tank 2, has failed. Fine. it was just a small 2 drive tank with nothing important on it. I finally got the system up without it throwing a ton of errors. Now I just want to know how can I remove this tank so I can disconnect these drives (they are in an external cabinet) from the system. Everytime I have tried to just leave it off, and power up. It will do absolutely nothing.

Thanks for your help.

 

[joeschmuck]

Hey boss man, you have been a member long enough to know we need some basic details such as FreeNAS version, System Configuration (mostly related to physical drive connections), post the output of zpool status and zfs list and describe in detail what you have tried to remove your pool.

The lack of this data is likely the only reason you haven't received assistance already.

EDIT: Also you must be having some type of error message or lack of messages when you just leave the hard drives powered down, this is odd behavior so explain what is going on here as well. Remember, we cannot see what you see so you need to convey this to us.

 

[Harold Roberts]

I may have joined a while back, but I have no idea how to provide you the information you are requesting about zpool status, and zfs list. Do I do this from the console? from the web interface either way can you tell me what I need to do?

As far as configuration
I have 4 drives inside the PC in 1 pool (Tank-3), I have 5 drives in an external ESATA cabinet in another pool (Tank-1), I then have 2 drives in an external USB 3.0 Cabinet (Tank-2)(failed Tank) I will provide the FreeNas version this evening when I get home. I know that there is an update to 11.1 available.

As far as trying to remove. I have only left my device powered down. Currently the device is powered up, and I am able to access the console. I get a message about unable to determine free space or something along those lines.

Here are some of the emails I get at night if that helps you out any. I know that my boot pool is degraded. This happens every month. I have to just swap my USB drives and that goes away.

I am sure there are other issues as well, like it telling me my GPT table is corrupt. I have no clue what that means, and can find no information anywhere on how to clear it up. the system comes up (most of the time) so I just ignore it.

freenas.RobertsFam changes in mounted filesystems:
10d9
< Tank-2 /mnt/Tank-2 zfs rw,nfsv4acls 0 0

freenas.RobertsFam kernel log messages:
> CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2397.70-MHz K8-class CPU)
> ugen2.1: <Intel UHCI root HUB> at usbus2
> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
> ugen5.1: <Intel UHCI root HUB> at usbus5
> uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
> uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
> uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
> uhub2: 2 ports with 2 removable, self powered
> (aprobe1:siisch0:0:1:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe1:siisch0:0:1:0): CAM status: Uncorrectable parity/CRC error
> (aprobe1:siisch0:0:1:0): Error 5, Retries exhausted
> uhub3: 6 ports with 6 removable, self powered
> (aprobe2:siisch0:0:2:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe2:siisch0:0:2:0): CAM status: Unconditionally Re-queue Request
> (aprobe2:siisch0:0:2:0): Error 5, Retry was blocked
> (aprobe1:siisch0:0:1:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe1:siisch0:0:1:0): CAM status: Uncorrectable parity/CRC error
> (aprobe1:siisch0:0:1:0): Error 5, Retries exhausted
> umass0 on uhub3
> (aprobe2:siisch0:0:2:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe2:siisch0:0:2:0): CAM status: Unconditionally Re-queue Request
> (aprobe2:siisch0:0:2:0): Error 5, Retry was blocked
> (aprobe1:siisch0:0:1:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe1:siisch0:0:1:0): CAM status: Uncorrectable parity/CRC error
> (aprobe1:siisch0:0:1:0): Error 5, Retries exhausted
> (aprobe2:siisch0:0:2:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe2:siisch0:0:2:0): CAM status: Unconditionally Re-queue Request
> (aprobe2:siisch0:0:2:0): Error 5, Retry was blocked
> (aprobe1:siisch0:0:1:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe1:siisch0:0:1:0): CAM status: Uncorrectable parity/CRC error
> (aprobe1:siisch0:0:1:0): Error 5, Retries exhausted
> umass3 on uhub3
> ukbd0 on uhub4
> (aprobe2:siisch0:0:2:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe2:siisch0:0:2:0): CAM status: Unconditionally Re-queue Request
> (aprobe2:siisch0:0:2:0): Error 5, Retry was blocked
> (aprobe1:siisch0:0:1:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
> (aprobe1:siisch0:0:1:0): CAM status: Uncorrectable parity/CRC error
> (aprobe1:siisch0:0:1:0): Error 5, Retries exhausted
> run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
> (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (probe0:umass-sim1:1:0:0): Retrying command
> (probe1:umass-sim2:2:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe1:umass-sim2:2:0:0): CAM status: CCB request completed with an error
> (probe1:umass-sim2:2:0:0): Retrying command
> (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (probe0:umass-sim1:1:0:0): Retrying command
> (probe1:umass-sim2:2:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe1:umass-sim2:2:0:0): CAM status: CCB request completed with an error
> (probe1:umass-sim2:2:0:0): Retrying command
> (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (probe0:umass-sim1:1:0:0): Retrying command
> (probe1:umass-sim2:2:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe1:umass-sim2:2:0:0): CAM status: CCB request completed with an error
> (probe1:umass-sim2:2:0:0): Retrying command
> (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (probe0:umass-sim1:1:0:0): Retrying command
> (probe1:umass-sim2:2:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe1:umass-sim2:2:0:0): CAM status: CCB request completed with an error
> (probe1:umass-sim2:2:0:0): Retrying command
> (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (probe0:umass-sim1:1:0:0): Error 5, Retries exhausted
> (probe1:umass-sim2:2:0:0): INQUIRY. CDB: 12 00 00 00 24 00
> (probe1:umass-sim2:2:0:0): CAM status: CCB request completed with an error
> (probe1:umass-sim2:2:0:0): Error 5, Retries exhausted
> ada0: 953869MB (1953525168 512 byte sectors)
> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
> da1 at umass-sim3 bus 3 scbus12 target 0 lun 0
> da1: <Generic Flash Disk 8.00> Removable Direct Access SPC-2 SCSI device
> da1: Serial Number 53BCF28B
> da1: 40.000MB/s transfers
> da1: 61000MB (124928000 512 byte sectors)
> da1: quirks=0x2<NO_6_BYTE>
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #4 Launched!
> SMP: AP CPU #7 Launched!
> Timecounter "TSC-low" frequency 1198848516 Hz quality 1000
> GEOM: da1: the secondary GPT table is corrupt or invalid.
> GEOM: da1: using the primary only -- recovery suggested.
> ums0 on uhub4
> bge0: link state changed to UP

-- End of security output --

Checking status of zfs pools:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Tank-1 32.62T 2.19T 30.44T - 33% 60% 1.00x ONLINE /mnt
Tank-3 24.06T 1.60M 24.06T - 0% 0% 1.00x ONLINE /mnt
freenas-boot 59.5G 2.20G 57.3G - - 3% 1.00x DEGRADED -

pool: freenas-boot
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: resilvered 8.99M in 0h1m with 0 errors on Wed Dec 20 19:25:06 2017
config:

NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
da0p2 DEGRADED 0 0 0 too many errors
da1p2 DEGRADED 0 0 1 too many errors

errors: No known data errors

Local system status:
3:01AM up 7:42, 0 users, load averages: 0.33, 0.21, 0.18

-- End of daily output --

 

[joeschmuck]

In reading your response I feel like FreeNAS is not for you. FreeNAS is not a set and forget system and you cannot ignore error messages. I hate turning someone away from FreeNAS but it really may be best for you. Have you considered something like a Qnap or Synology NAS?

Basically I'm willing to help you figure this out and remove your faulty tank but you need to address the other failing issues.

To run the commands I listed above you can use the GUI and open the Shell (left side pane) and then enter the command. You will need to cut & paste the results using code tags to maintain proper text format.

But I also need to know what happens when you try to remove Tank-2 and what the error message(s) is/are. I need you to explain to me all the steps you do like I'm an idiot. Trust me, this method works because we remove assumptions so I know exactly what you did. Be very descriptive even though it does take a bit of time to type all that data in.

The very first thing you really need to do is read the User Guide from cover to cover and doing this twice would be ideal but at least once will let you know what is in the guide and it will help you in maintaining your system and with the basic operations.

Lets fix the Boot Pool first as this could be causing you issues... Reference the FreeNAS User Guide to backup/restore the configuration file if you are not familiar with this.

1) Backup the Configuration file then power down.
2) Remove your current USB Flash Drives and set to the side.
3) Insert a single unused USB Flash Drive and install FreeNAS 11.0-U4 to this drive.
4) Boot up your system and cancel the configuration setup.
5) Restore you configuration file.
6) Reboot and your system should come back to life and you will no longer have boot pool errors. If you do have boot pool errors then you have a problem with your configuration data and this is not a good thing at all and you are going to need to perform a clean installaton again and manually configure your system, not a fun thing if you have jails.

 

[Harold Roberts]

I can't figure out how to cut and paste within the shell. Ctrl-C doesn't work. Here is the zpool

Here is the zfs list

My freenas version is: FreeNAS-11.0-U2 (e417d8aa5)

As far as trying to remove the tank. I have selected it and right clicked. but there is no option. I have no idea how to get rid of it, that is why I am asking this forum on how to do it.

It doesn't show up in any of the commands you requested but here in this screen shot you can see it is listed

As for the issue with the Boot pool. I have asked on here how to resolve this, and everyone gave me some suggestions. Bad thumbdrives.

I have tried all kinds of thumbdrives, using the suggestions on this forum. I now buy 2 brand new Sandisk 64gb Thumbdrives every month, swap them out and then give the others to people or use them on Linux or Windows if I have to actually get on that crappy system.

As for getting rid of FreeNAS, sure I have thought about it. I have gone thru many many other NAS type systems, and keep returning to FreeNAS because it used to be easy to use. Since all of this crap started being added zpools instead of allowing me to use my Hardware RAID 10 correctly, I have to create it with software, and I loose more with software than my hardware raid it seems.

I am a Unix/Linux person. I am trying to learn BSD, and understand FreeNAS, but when I try to read stuff about zpool, and zfs and everything else I can't find any information that makes it simple to understand. FreeNAS seems to me to have gone to the Engineers only. Even the Systems Engineer that I work with at Work has stated that he will never have it in our organization because it is way to complicated.

I just want some help to understand how to resolve these issues, and be able to use this system to is fullest abilities. and YES I have read the user guide more than 2 times, how do you think I was able to get this set up?

 

[joeschmuck]

Screen captures work great too!

Your solution seems to be fairly easy. Tanks-1 and Tank-3 look to be good. Do you have the room to install a single SSD in your FreeNAS computer to use as the boot device? This will solve the USB reliability issue, although I doubt your USB flash drives are all actually damaged, they probably just became corrupt. Stop buying 64GB USB Flash drives, they cost too much, an 8GB drive is fine. And only use a single drive, no need to use mirrors, just make a backup of your config file and if you have to restore it, no problem and it's quick.

I suspect that your freenas-v1.db file is corrupt which is the configuration file. The only way to resolve this is to start from a clean install, do not restore your configuration file. During this process ensure that your Tank-2 is disconnected before you start this. What leads me to this conclusion is Tank-2 is not listed on the outputs I requested, only the FreeNAS GUI is confused and that data resides in the configuration database file. I assume that you have tried to remove Tank-2 via the GUI and you get some error message or it isn't even an option.

So the downside to rebuilding is if you have any jails or plugins, these may need to be destroyed and recreated for the GUI to recognize them properly. Once you get everything back to normal, create a backup of the configuration data.

I think I've covered everything here and I wish you good luck. If you have another question, please ask.

P.S. I'd recommend upgrading to FreeNAS 11.0-U4 at this time. You can use 11.1 but it's hot off the press so if there are minor issues with it, I'm sure you would not want to deal with those. Wait for 11.1-U1 to come out before upgrading, assuming you want to that is.

 

[danb35]

As far as trying to remove the tank. I have selected it and right clicked. but there is no option. I have no idea how to get rid of it, that is why I am asking this forum on how to do it.

It isn't a tank, it's a pool. The "standard" (i.e., most common, but by no means necessary) name for a ZFS pool is tank, but a pool is nonetheless a pool, not a tank. The fact that you've decided to call all your pools Tank-n is somewhat unusual, but wouldn't hurt anything.

With that said, have you tried just detaching that pool from the GUI? Click on Tank-2 in the list of volumes, and then click the Detach button at the bottom (it would have a red X on it; it's the button to the far left in the pic below).

What's happening is that your configuration database thinks there should be another pool, but it can't get any current information about it (because it isn't actually there any more). If the GUI will let you detach the pool in this way, it will remove it from the list, clear any errors, and you'll be good to go.

I suspect that your freenas-v1.db file is corrupt which is the configuration file.

I don't think there's been any evidence of that shown. What's happening is that the disks comprising Tank-2 have died, so they were removed from the system, but not from the GUI.

 

[Harold Roberts]

@danb35 Thank you for explaining that to me. That is what I couldn't figure out how to do. When I created the devices, I just selected all of the defaults.

I tried to detach it using the steps you outlined, and I get this message:

Code:

Request Method:	POST
Request URL:	http://192.168.1.40/storage/detach/2/
Software Version:	FreeNAS-11.0-U2 (e417d8aa5)
Exception Type:	CallTimeout
Exception Value:	
Call timeout
Exception Location:	/usr/local/lib/python3.6/site-packages/middlewared/client/client.py in call, line 233
Server time:	Fri, 22 Dec 2017 09:18:07 -0800
Traceback

Environment:

Software Version: FreeNAS-11.0-U2 (e417d8aa5)
Request Method: POST
Request URL: http://192.168.1.40/storage/detach/2/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  39.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volume_detach
  897.					 cascade=form.cleaned_data.get('cascade', True))
File "./freenasUI/storage/models.py" in delete
  385.					 systemdataset=systemdataset,
File "./freenasUI/storage/models.py" in _delete
  348.				 n.stop(svc)
File "./freenasUI/middleware/notifier.py" in stop
  266.			 return c.call('service.stop', what, {'onetime': onetime}, **kwargs)
File "./freenasUI/middleware/notifier.py" in stop
  266.			 return c.call('service.stop', what, {'onetime': onetime}, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  233.			 raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /storage/detach/2/
Exception Value: Call timeout



Request information
GET
No GET data

POST
Variable	Value
__all__	''
cascade	'on'
__form_id	'form_VolumeExport'
FILES
No FILES data

COOKIES
Variable	Value
fntreeSaveStateCookie	'root%2Croot%2F173%2Croot%2F58%2Croot%2F58%2F59%2F60%2Croot%2F58%2F59%2Croot%2F38%2F40%2Croot%2F8%2Croot%2F59%2F100%2Croot%2F59%2F90%2Croot%2F59%2F97%2Croot%2F59%2F60%2F61%2Croot%2F59%2F98%2Croot%2F59'
csrftoken	'18sc8a1fmbYwyjQ6lmFrU8EcadtITMQcpDFtCmYrMRcCPnJmpSEO741gdU3oUanp'
sessionid	'jnq6y6aojrx6dp4kuu6h6bu1z6ae8tjq'
META
Variable	Value

Not sure what that means, but it is still listed there.

I am in the process of copying everything off this NAS on to external 4 TB drives, and I can then be destructive with the configuration and setting it up correctly. The Tank (volume) that failed was the only drive that I can lose without hurting my feeelings. Everything from it was backed up nightly to one of the other volumes anyways.

The reason I don't want to loose this data is this is my history. I have saved everything I have come across since I first started working with PC's even before the internet existed, when all we had were BBS's. The files on this NAS are the backups of every single PC, I have ever owned. from DOS, to Windows, to Linux, Unix, and SCOunix. Some are still tar'd others I have expanded. It is also the backup of every email I have ever sent or received personally or at every job I have worked at. It is also High Resolution copies of every picture of my family, and ancestors, as well as Signatures and anything I could find in mine and my fathers research.

I know some of this may never be used again. Like the 5 1/4 floppy backup of Kingdom of Kroz. or the 8 inch floppy backups, the Superdisk drives, or the Zip/jaz drives but to me this is my legacy.

@joeschmuck. That is why I use FreeNAS. it appears to be the hardest program to backup my life onto.

 

[joeschmuck]

@danb35 Thank you for explaining that to me. That is what I couldn't figure out how to do.

Seriously, you didn't try to remove the pool already? I thought you said you had tried or maybe I am misunderstanding what you are writing. If you didn't try these steps previously then per my previous suggestion, read the user guide, all of this is in there.

even before the internet existed, when all we had were BBS's.

I remember those days very well and I use to run a BBS, had to get a second phone line installed. We have come a long ways since the mid/late 1980's.

I am in the process of copying everything off this NAS on to external 4 TB drives, and I can then be destructive with the configuration and setting it up correctly.

While we all promote having a backup of your data, the process of rebuilding the FreeNAS boot media will not destroy your data on Tank-1 nor Tank-3. But you could destroy those and start all over from scratch if you like. The jails you have however might not work, assuming you have something in a jail.

 

[Harold Roberts]

I have no idea what a jail is, so I am going to say I don't have anything. The NAS is currently only used for file storage.

I would love to be able to use it for more, but have never been able to understand how to configure it.

As for my NAS Boot Pool. It is installed on 64gb Sandisk USB drives. The reason why is this:

Code:

A minimum of 8 GB of space on the boot device is required. However, 32 GB is recommended to provide room for future additions and boot environments. When using mirrored boot devices, it is best to use devices of the same size. If the device sizes are different, the mirror is limited to the size of the smaller device.

This is directly from the Installation guide (see I did read it). I used many 32gb drives but they were degrading as well. and the price of 64gb's became nothing so I started buying them.

I have them configured in a mirror. After I replaced about my second dozen set I asked on here: https://forums.freenas.org/index.php?threads/usb-mirror-install-issues.55464/#post-387288

What I can do to stop this. As you can see from the post someone said it was my system dataset pool. I verified that it was on a tank and not my boot drives, and that was it. No one else even made suggestions. This was back in June. I figured that no one wanted to help a newbie like me so I dropped the subject and continued to swap the drives every month, until Oct. when Tank-2 failed. at that point I could not get the system to come up, and figured there was no way anyone here was going to help me, so I shutdown the system and left it turned off until the other day when I powered it up, and asked about how to remove the tank.

Even now looking at my USB drives. The red activity lights on both of them are going on and off like crazy. Here is a short video to show you how active they are, The HDD light is because I am copying data off the data drives at this time. https://photos.app.goo.gl/AQOJTmtvboWLdY8p1

 

[joeschmuck]

I speak no lies here and my advice is sound, a single 8GB USB Flash drive is all you need but larger is fine too, anything larger than 16GB is a waste of your money especially for how you use your system. Stay clear of small plastic USB 3.0 devices, get a USB 2.0 if you can find one or a metal case USB 3.0 device. USB 2.0 is the best way to go. It may be slower however they do not have heat issues. The best way to go is using a single internal SSD connected to your SATA port if your hardware supports it.

Since you do not have any jails and are using this as a simple NAS, recovery is easy. Do as I state below and you will be fine.

0) If you have any user accounts or special configurations like Active Directory, write all this data down so you can restore it. If this sounds greek to you then ignore this step.
1) Remove your current USB Flash Drives and set to the side.
2) Insert a single unused USB Flash Drive and install FreeNAS 11.0-U4 to this drive.
3) Boot up your system and follow the installation procedure to import your pools and set up your system.
4) Reconfigure your Shares.
5) Once your system is re-configured and operational make a backup of your configuration file should you run into any corruption again.

I wish I could just have my sister or newphew or niece come over to your house and punch a few key and make it all good for you but that would be a bit odd to ask them that. They live in Reno, Sparks, and Fernley. I use to live in Reno as well 37 years ago and worked at the Pick Hobson Casino, it's no longer there of course. Downtown has changed so much that I barely recognized it 2 years ago when I was there.