HELP: 8.2 p1 flash died - the primary GPT table is corrupt

[Jeff Buddington]

Team-

Please help. Had my FreeNAS system for a long time successfully but I am stuck on 8.2 p1 and now my boot flash died. Fresh install and boot. First time I can auto import my ZFS pool called GenePool.

This is a pool with two RAID5 groups with 6 1TB drives and 6 2TB drives. Most of the drives have been failed, successfully replaced and resilvered over a period of time. There are only a few original drives remaining. These show up in gpart status as OK. The drives I repartitioned as identically as I could and successfully resilvered show up as corrupt GPT table.

Code:

[root@gp1] ~# gpart status
  Name  Status  Components
  da0s1      OK  da0
  da0s2      OK  da0
  da0s3      OK  da0
  da0s4      OK  da0
da0s1a      OK  da0s1
ada1p1  CORRUPT  ada1
ada1p2  CORRUPT  ada1
ada2p1  CORRUPT  ada2
ada2p2  CORRUPT  ada2
ada4p1  CORRUPT  ada4
ada4p2  CORRUPT  ada4
ada5p1      OK  ada5
ada5p2      OK  ada5
ada6p1  CORRUPT  ada6
ada6p2  CORRUPT  ada6
ada7p1  CORRUPT  ada7
ada7p2  CORRUPT  ada7
ada8p1      OK  ada8
ada8p2      OK  ada8
ada9p1      OK  ada9
ada9p2      OK  ada9
ada10p1  CORRUPT  ada10
ada10p2  CORRUPT  ada10
ada11p1      OK  ada11
ada11p2      OK  ada11

Once successful autoimport, after a reboot, I get the following error messages:

Code:

GEOM: ada0: the primary GPT table is corrupt or invalid.
GEOM: ada0: using the secondary instead -- recovery strongly advised.
GEOM: ada1: the primary GPT table is corrupt or invalid.
GEOM: ada1: using the secondary instead -- recovery strongly advised.
GEOM: ada2: the primary GPT table is corrupt or invalid.
GEOM: ada2: using the secondary instead -- recovery strongly advised.
GEOM: ada3: the primary GPT table is corrupt or invalid.
GEOM: ada3: using the secondary instead -- recovery strongly advised.
GEOM: ada4: the primary GPT table is corrupt or invalid.
GEOM: ada4: using the secondary instead -- recovery strongly advised.
GEOM: ada6: the primary GPT table is corrupt or invalid.
GEOM: ada6: using the secondary instead -- recovery strongly advised.
GEOM: ada7: the primary GPT table is corrupt or invalid.
GEOM: ada7: using the secondary instead -- recovery strongly advised.
GEOM: ada10: the primary GPT table is corrupt or invalid.
GEOM: ada10: using the secondary instead -- recovery strongly advised.
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS filesystem version 4
ZFS storage pool version 15
ZFS WARNING: Unable to attach to ada10.
ZFS WARNING: Unable to attach to ada4.
ZFS WARNING: Unable to attach to ada7.
ZFS WARNING: Unable to attach to ada6.
ZFS WARNING: Unable to attach to ada1.
ZFS WARNING: Unable to attach to ada2.
ZFS WARNING: Unable to attach to ada10.
ZFS WARNING: Unable to attach to ada4.
ZFS WARNING: Unable to attach to ada7.
ZFS WARNING: Unable to attach to ada6.
ZFS WARNING: Unable to attach to ada1.
ZFS WARNING: Unable to attach to ada2.
GEOM: ada0: the primary GPT table is corrupt or invalid.
GEOM: ada0: using the secondary instead -- recovery strongly advised.
GEOM: ada3: the primary GPT table is corrupt or invalid.
GEOM: ada3: using the secondary instead -- recovery strongly advised.

Then the Pool Will not Mount or work, says not available.

Here is an example of a good gpart:

Code:

[root@gp1] ~# camcontrol identify ada5
pass5: <Hitachi HDT721010SLA360 ST6OA31B> ATA-8 SATA 2.x device
pass5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 
protocol              ATA/ATAPI-8 SATA 2.x
device model          Hitachi HDT721010SLA360
firmware revision    ST6OA31B
serial number        STF604MR22KR1P
WWN                  5000cca349dd5a3c
cylinders            16383
heads                16
sectors/track        63
sector size          logical 512, physical 512, offset 0
LBA supported        268435455 sectors
LBA48 supported      1953525168 sectors
PIO supported        PIO4
DMA supported        WDMA2 UDMA6
media RPM            7200
 
Feature                      Support  Enabled  Value          Vendor
read ahead                    yes    yes
write cache                    yes    yes
flush cache                    yes    yes
overlap                        no
Tagged Command Queuing (TCQ)  no    no
Native Command Queuing (NCQ)  yes        32 tags
SMART                          yes    yes
microcode download            yes    yes
security                      yes    no
power management              yes    yes
advanced power management      yes    no    0/0x00
automatic acoustic management  yes    no    254/0xFE    128/0x80
media status notification      no    no
power-up in Standby            yes    no
write-read-verify              no    no
unload                        no    no
free-fall                      no    no
data set management (TRIM)    no

Here is one showing corrupted:

Code:

Geom name: ada11
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada11p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Mode: r1w1e0
  rawuuid: 95943d58-7763-11e0-b4ec-485b39c1d8a6
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: swap-ada3
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada11p2
  Mediasize: 998057319936 (930G)
  Sectorsize: 512
  Mode: r0w0e0
  rawuuid: 959b5561-7763-11e0-b4ec-485b39c1d8a6
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: disk3
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada11
  Mediasize: 1000204886016 (932G)
  Sectorsize: 512
  Mode: r1w1e1

Code:

[root@gp1] ~# camcontrol identify ada0
pass0: <ST32000542AS CC34> ATA-8 SATA 2.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 
protocol              ATA/ATAPI-8 SATA 2.x
device model          ST32000542AS
firmware revision    CC34
serial number        5XW16XD9
WWN                  5000c5002a6872a4
cylinders            16383
heads                16
sectors/track        63
sector size          logical 512, physical 512, offset 0
LBA supported        268435455 sectors
LBA48 supported      3907029168 sectors
PIO supported        PIO4
DMA supported        WDMA2 UDMA6
media RPM            5900
 
Feature                      Support  Enabled  Value          Vendor
read ahead                    yes    yes
write cache                    yes    yes
flush cache                    yes    yes
overlap                        no
Tagged Command Queuing (TCQ)  no    no
Native Command Queuing (NCQ)  yes        32 tags
SMART                          yes    yes
microcode download            yes    yes
security                      yes    no
power management              yes    yes
advanced power management      yes    no    0/0x00
automatic acoustic management  yes    no    0/0x00    254/0xFE
media status notification      no    no
power-up in Standby            yes    no
write-read-verify              yes    no    0/0x0
unload                        no    no
free-fall                      no    no
data set management (TRIM)    no

 

[Jeff Buddington]

OK- Everything Still Down. Trying to figure it out..

So I did a gpart recover on most of the devices except two.. Didn't help with the attach failures. But now only the two drives are showing primary GPT table is corrupt or invalid. These drives were not partitioned correctly true however the zfs data is intact and the previous system used to read them fine. I need to erase these two drives, partition them properly and resilver them.

Here is a zpool status from before the crash of the primary flash drive when the system was working...

[root@GP01] ~# zpool status
pool: GenePool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
GenePool DEGRADED 0 0 0
raidz1 ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
gpt/disk3 ONLINE 0 0 0
gpt/disk4 ONLINE 0 0 0
gpt/ada5 ONLINE 0 0 0
raidz1 DEGRADED 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
12734268283310371124 FAULTED 0 0 0 was /dev/ada0
ada3 ONLINE 0 0 0
ada11 ONLINE 0 0 0
ada7 ONLINE 0 0 0

errors: No known data errors
[root@GP01] ~# zpool replace GenePool 12734268283310371124 ada2

I can't do a zpool status right now because after a reboot it says no pools.

Why the attach failures now? If I wipe the Flash Drive Completely clean with a new install of 8.2 P1 then use the Web GUI to auto import the volume then the entire GenePool will come up successfully. You can see that over time and different motherboards and controller orders the actual volume names were different than the actual adaX numbers as these get renumbered dynamically on boot. However once I reboot then it never will come up again.

It's pretty obvious that I partitioned these two drives bad:

Code:

[root@gp1] ~# gpart show ada0
=>        34  3907029101  ada0  GPT  (1.8T) [CORRUPT]
          34  3907029101        - free -  (1.8T)
 
[root@gp1] ~# gpart show ada3
=>        34  3907029101  ada3  GPT  (1.8T) [CORRUPT]
          34  3907029101        - free -  (1.8T)
 
[root@gp1] ~# gpart show ada1
=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  3902834703    2  freebsd-zfs  (1.8T)
 
[root@gp1] ~# gpart show ada2
=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47K)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  3902834703    2  freebsd-zfs  (1.8T)

Good:

Code:

Geom name: ada1
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Mode: r1w1e0
  rawuuid: 678e23c0-dc2a-11e0-960a-485b39c1d8a6
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: gpt/ada8
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada1p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Mode: r0w0e0
  rawuuid: 7ff8654c-dc2a-11e0-960a-485b39c1d8a6
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: gpt/ada8
  length: 1998251367936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: ada1
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Mode: r1w1e1

Bad:

Code:

[root@gp1] ~# gpart list ada0
Geom name: ada0
state: CORRUPT
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Consumers:
1. Name: ada0
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Mode: r0w0e0

Code:

GEOM: ada0: the primary GPT table is corrupt or invalid.
GEOM: ada0: using the secondary instead -- recovery strongly advised.
GEOM: ada3: the primary GPT table is corrupt or invalid.
GEOM: ada3: using the secondary instead -- recovery strongly advised.
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS filesystem version 4
ZFS storage pool version 15
ZFS WARNING: Unable to attach to ada10.
ZFS WARNING: Unable to attach to ada4.
ZFS WARNING: Unable to attach to ada7.
ZFS WARNING: Unable to attach to ada6.
ZFS WARNING: Unable to attach to ada1.
ZFS WARNING: Unable to attach to ada2.
ZFS WARNING: Unable to attach to ada10.
ZFS WARNING: Unable to attach to ada4.
ZFS WARNING: Unable to attach to ada7.
ZFS WARNING: Unable to attach to ada6.
ZFS WARNING: Unable to attach to ada1.
ZFS WARNING: Unable to attach to ada2.
GEOM: ada0: the primary GPT table is corrupt or invalid.
GEOM: ada0: using the secondary instead -- recovery strongly advised.
GEOM: ada3: the primary GPT table is corrupt or invalid.
GEOM: ada3: using the secondary instead -- recovery strongly advised.

 

[Jeff Buddington]

Forgot to show the new status:

Code:

[root@gp1] ~# gpart status
  Name  Status  Components
  da0s1      OK  da0
  da0s2      OK  da0
  da0s3      OK  da0
  da0s4      OK  da0
da0s1a      OK  da0s1
ada1p1      OK  ada1
ada1p2      OK  ada1
ada2p1      OK  ada2
ada2p2      OK  ada2
ada4p1      OK  ada4
ada4p2      OK  ada4
ada5p1      OK  ada5
ada5p2      OK  ada5
ada6p1      OK  ada6
ada6p2      OK  ada6
ada7p1      OK  ada7
ada7p2      OK  ada7
ada8p1      OK  ada8
ada8p2      OK  ada8
ada9p1      OK  ada9
ada9p2      OK  ada9
ada10p1      OK  ada10
ada10p2      OK  ada10
ada11p1      OK  ada11
ada11p2      OK  ada11

Still unable to attach like above.

 

[cyberjock]

Well, I have no clue what is going on. I read your post and it sounds like you've done alot of "bad" things and now you are paying the price. In particular:

The drives I repartitioned as identically as I could and successfully resilvered show up as corrupt GPT table.

• If you are truely "repartitioning" your drives I don't even know what to say...

• You later say "here's a good and bad" and I see nothing bad and only good one.

• Later you show a zpool status from before, and yet I see whole disks being used. That *clearly* was not done by FreeNAS as we have *never* done full disks. So either you did them from the CLI or from another OS. In either case, you shouldn't be using that pool on this server. Also the zpool status shows gpt/disk1. Sorry, it should give a GPTID.

Right now I have no explanation for why you are trying to fix the GPT tables on stuff that was whole disk. It shouldn't have a partition table at all.

Regardless, I don't want to touch this system with a 10 foot pole. You've definitely done at least two things that aren't recommended(at the least) and have compounded the problem trying to fix whatever wasn't broken because you thought you should.

Right now I'm doubting anyone else is going to unless you're going to stop and start explaining why the heck your system is nothing near the established norms for FreeNAS. Because my bet is all this non-standardization you've done is exactly why you are in the mess you are in right now.

 

[Jeff Buddington]

OK- So it's not as bad as I make it sound (or perhaps it is). I don't do a really good job posting to forums but I could still really use some help. Let me try to explain the series of events that brought me to this..It all really started because there was no way with the early GUI to replace a failed drive.

1. A long time ago I had a healthy system provisioned through the web GUI with 6x1TB drives. Really happy. Added 6x2TB drives to the pool. Sailing along smoothly.
2. One day one of the drives completely fails. No apparent way to replace the drive through the web GUI on older releases. So I logged into the shell, created a GPART with whole drive, (didn't know how to create the other partitions). Then I manually did a zpool replace. Then I would detach the lost drive. Pool came back up perfectly. Everything worked great.
3. Happened again several times (#2 here).
4. System hardware died. Put flash drive into a better system with a much better motherboard. Move drives. More SATA ports. New controllers, new drive bays. System ran great for a long time.
5. Another drive failed. Because of new hardware and moves of drive order the previous ada0,ada1, etc. numbers no longer matched the original drive orders, etc. So when the drive failed, the system would say ada0, ada1, etc. were online then the failed drive would say was ada1 :-O
6. To "avoid confusion" when I partitioned the drive I found an article that showed me via cli how to partition the drives with the swap and proper partitions so I created them with the label gpt/disk1 - I thought this naming would maintain a clear label despite hardware failure and if the drives needed to be reordered later.
7. As long as the original flash drive was alive the system would happily boot up and never had any problems.
8. The original flash drive died. Now as long as the flash drive is completely clean the whole system will boot up and I can successfully import and retrieve data from the pool and a scrub shows completely clean.
9. Reboot the working system. Errors are as above. Failed to attach ada1, etc. Only resolution is to create a totally clean new flash and boot the system again.

Now having said this I am trying to come up with a plan for healthyness. My idea is as follows:

1. Create clean flash. Boot up system. Locate one of the two drives that are "full drive partition" currently "ada0" and "ada3". Replace with an empty drive. Partition with a swap, data identical to other drives.
2. zpool replace the identifier number with the "ada0". Wait for resilver to complete.
3. Repeat #1 and #2 for "ada3"
4. Create a clean flash. Boot up and import pool. gpart recover any primary partitions showing corrupt. At this point there should be no corrupt primary partitions and everything should be running.
5. Hope for the best and reboot the system.

 

[Jeff Buddington]

Who can help? @cyberjock There has got to be a way to fix this? Created a brand new FreeNAS 9.2.1.5 flash drive. Booted it up, imported pool GenePool. Imports perfectly no problems. If I had a really really good UPS the system would run for a long, long, time.

The issue is if I reboot the system now it will never mount the pool again. Scrubbing the pool with 9.2.1.5...

Code:

[default@freenas] /% zpool status
  pool: GenePool
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub in progress since Wed Jun 25 10:29:13 2014
        41.0G scanned out of 14.0T at 99.6M/s, 40h42m to go
        0 repaired, 0.29% done
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    GenePool                                        ONLINE      0    0    0
      raidz1-0                                      ONLINE      0    0    0
        gptid/94dd14ec-7763-11e0-b4ec-485b39c1d8a6  ONLINE      0    0    0
        gptid/951285f1-7763-11e0-b4ec-485b39c1d8a6  ONLINE      0    0    0
        gptid/955d00c9-7763-11e0-b4ec-485b39c1d8a6  ONLINE      0    0    0
        gptid/959b5561-7763-11e0-b4ec-485b39c1d8a6  ONLINE      0    0    0
        ada10                                      ONLINE      0    0    0  block size: 512B configured, 4096B native
        ada4                                        ONLINE      0    0    0
      raidz1-1                                      ONLINE      0    0    0
        ada0                                        ONLINE      0    0    0
        ada3                                        ONLINE      0    0    0
        ada7                                        ONLINE      0    0    0  block size: 512B configured, 4096B native
        ada6                                        ONLINE      0    0    0  block size: 512B configured, 4096B native
        ada1                                        ONLINE      0    0    1  block size: 512B configured, 4096B native
        ada2                                        ONLINE      0    0    0  block size: 512B configured, 4096B native
 
errors: No known data errors

 

[Jeff Buddington]

Seems that when the system imports the pool from a blank flash drive, it only creates something called GEOM_ELI for 4 original surviving drives from the original FreeNAS Gui created pool.. After reboot all the other drives not listed here error with "unable to attach, and primary GPT corrupt".

Code:

ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada11p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software

 

[danb35]

If the pool is imported and a scrub is running, it seems like a good idea to let it finish.

 

[joeschmuck]

My very strong advice is, if you are able to access your files in your pool, copy off whatever you deem important first, then copy off the remaining files. If you have to create a stack of DVDs, that is okay.

Next, destroy the pools and create a clean FreeNAS 9.2.1.3 boot flash drive. Build it up from scratch and create the new pools properly. If you would rather run 8.2.x that is fine too. I have no idea what your system hardware is.

 

[Ericloewe]

In the future, follow the manual's instructions to the letter instead of making it up as you go along. Cyberjock says he has no clue what's going on - that alone should scare you.

 

[Jeff Buddington]

Team- Thanks for the follow up. @ericloewe LOL I like your comment about what @cyberjock said. Point taken. Yes,the scrub completed successfully with no major errors. resilvered a few megabytes due to the CRC error / abrupt power off when system failed.

I can access the files and mount/import the pool / share the files once I boot a clean 9.2.1.5 USB drive each time I have to power cycle the system. I literally made a stack of USB flash drives ready to go by number for this purpose. This will have to do until I can afford to build a clean new system with 4TB drives. This time I will build a system with only 4 Drives per raidz instead of 6, and not mix drive sizes. Also now the Web GUI is much better and hopefully if a drive fails in the new system it would be possible to easily replace and resilver it with just the web GUI. Is there a process for this that's well documented for this I hope? I haven't looked yet.

Is it possible to remove a raid-z from the pool now? In this way I could offload my files gradually until I freed enough space to remove the 6x1TB raidz from the pool. Then create a new system using those drives. Copy from the 6x2TB raidz pool over the network to the new system. Then offload remaining files to External Drives and such, erase and add the 6xTB raidz to the "new system". In the past you could add raidz to the pool but unfortunately never remove them even if you freed up enough space and scrubbed to pool. Sure would be nice to do this in the future.

This is a success story to the power of ZFS / Freenas to retain data despite so many drive failures (they just don't make drives like they used to anymore 1YR/2YR warranty) and newbie actions while learning this amazing system through experience and trial / error. The very latest code imports the drives from the oldest software and it all works. Even though wrong partitioning, etc. I wouldn't trust my data and system backups to anything else ;-D

 

[Whattteva]

Team- Thanks for the follow up. @ericloewe LOL I like your comment about what @cyberjock said. Point taken. Yes,the scrub completed successfully with no major errors. resilvered a few megabytes due to the CRC error / abrupt power off when system failed.

If you don't care about your data at all or you have 100% solid backup, then this doesn't apply to you. But if you valued your data at all, then I suggest you get a decent UPS as power brownouts are somewhat of a common occurrence. For the peace of mind and stability it confers, it's worth every penny.

I can access the files and mount/import the pool / share the files once I boot a clean 9.2.1.5 USB drive each time I have to power cycle the system. I literally made a stack of USB flash drives ready to go by number for this purpose. This will have to do until I can afford to build a clean new system with 4TB drives. This time I will build a system with only 4 Drives per raidz instead of 6, and not mix drive sizes. Also now the Web GUI is much better and hopefully if a drive fails in the new system it would be possible to easily replace and resilver it with just the web GUI. Is there a process for this that's well documented for this I hope? I haven't looked yet.

Any sort of RAID implementation generally require identical sizes (better yet, identical make and models) if you expect it to behave with any sort of stability. RAID exists to give servers a redundancy system and makes several assumptions that server systems typically have. Sure, you can mix-and-match stuff and have it still work, but that's just asking for trouble. Last I checked, FreeNAS is not yet capable of resizing zpools on the fly (it may now, someone else may confirm this for you). But, if it doesn't, sadly your option would be to copy the files somewhere and recreate a new pool with reduced size and copy your files back in. Then again, you do have to do this anyway, as your present pool state right now is highly questionable. As the other guy said, cyberjock doesn't even have a clue, you should be scared.
About resilvering, there are stickies that deal with all that information.

Is it possible to remove a raid-z from the pool now? In this way I could offload my files gradually until I freed enough space to remove the 6x1TB raidz from the pool. Then create a new system using those drives. Copy from the 6x2TB raidz pool over the network to the new system. Then offload remaining files to External Drives and such, erase and add the 6xTB raidz to the "new system". In the past you could add raidz to the pool but unfortunately never remove them even if you freed up enough space and scrubbed to pool. Sure would be nice to do this in the future.

I think you need to read up how RAID works. RAID works by striping data across all drives with several drives reserved for parity data. You can't just remove drives as you please because the data is split across all of them. Well, you CAN remove several drives depending on your redundancy configuration, but you'll be operating the pool at a degraded state, which is really risky.

This is a success story to the power of ZFS / Freenas to retain data despite so many drive failures (they just don't make drives like they used to anymore 1YR/2YR warranty) and newbie actions while learning this amazing system through experience and trial / error. The very latest code imports the drives from the oldest software and it all works. Even though wrong partitioning, etc. I wouldn't trust my data and system backups to anything else ;-D

Glad it worked out that well for you. Countless many other brave souls prior to you who didn't quite read the manual lost their entire pool.
Consider yourself lucky and hopefully, this will be a good learning experience for you moving forward. Cheers!

 

[danb35]

Once a vdev has been added to a pool, it remains impossible to remove it and retain an intact pool. You can add more vdevs to a pool, and you can increase the capacity of RAIDZ vdevs by replacing all the drives in the vdev one at a time.

 

[joeschmuck]

So you don't have a way to backup all your data, is that correct? Because if you can backup all your data, you can fix all your issues quickly and don't need to wait for your new 4TB drives. Once all the data you want to retain has been backed up, destroy your pool and leave in the four largest capacity drives you have, remove the others. Load up a fresh copy of FreeNAS 9.2.1.6-RC2 and create a new pool out of those four drives. Finally finish setting up FreeNAS. Reboot and make sure it works. Shutdown and then power up again for good measure. It should be working fine, if not then you need to troubleshoot your problem. You might end up destroying your pool and loading up an older version of FreeNAS, then creating a new pool under that version. Now once you can afford 4TB drives, you can swap those out one at a time and let it resilver. No gain in capacity will be noted until you have swapped out the final drive with a 4TB drive.

Going this route should bring you to a stable platform once again.