Disks from volume "disappeared", even though the system still sees them.

[dmt0]

Unfortunately I don't have another controller to try out.

Switched around ada3 and ada4 (interchanged SATA cables). Now the disk that used to be ada4 (the only one alive), appears as ada3. The other 3 are "UNKNOWN".
So that at least rules out the connectors and order within the controller.

And than, when I look at messages, all the disks appear, with their serials, etc.

...
Oct 29 09:18:31 freenas kernel: da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
Oct 29 09:18:31 freenas kernel: da0: <WD My Book 1130 1012> Fixed Direct Access SCSI-6 device
Oct 29 09:18:31 freenas kernel: da0: Serial Number 574341575A31333538303832
Oct 29 09:18:31 freenas kernel: da0: 40.000MB/s transfers
Oct 29 09:18:31 freenas kernel: da0: 2861556MB (732558336 4096 byte sectors: 255H 63S/T 45599C)
Oct 29 09:18:31 freenas kernel: ada0 at ata0 bus 0 scbus0 target 1 lun 0
Oct 29 09:18:31 freenas kernel: ada0: <CF Card Ver5.04> ATA-5 device
Oct 29 09:18:31 freenas kernel: ada0: Serial Number 14EF070C0A0800484997
Oct 29 09:18:31 freenas kernel: ada0: 100.000MB/s transfers (UDMA5, PIO 512bytes)
Oct 29 09:18:31 freenas kernel: ada0: 7279MB (14909328 512 byte sectors: 16H 63S/T 14791C)
Oct 29 09:18:31 freenas kernel: ada1 at ata2 bus 0 scbus1 target 0 lun 0
Oct 29 09:18:31 freenas kernel: ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:18:31 freenas kernel: ada1: Serial Number WD-WMAZA3416616
Oct 29 09:18:31 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:18:31 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:18:31 freenas kernel: ada2 at ata2 bus 0 scbus1 target 1 lun 0
Oct 29 09:18:31 freenas kernel: ada2: <WDC WD20EARS-22MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:18:31 freenas kernel: ada2: Serial Number WD-WMAZA4748524
Oct 29 09:18:31 freenas kernel: ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:18:31 freenas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:18:31 freenas kernel: ada3 at ata3 bus 0 scbus2 target 0 lun 0
Oct 29 09:18:31 freenas kernel: ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:18:31 freenas kernel: ada3: Serial Number WD-WCAZA3050492
Oct 29 09:18:31 freenas kernel: ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:18:31 freenas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:18:31 freenas kernel: ada4 at ata3 bus 0 scbus2 target 1 lun 0
Oct 29 09:18:31 freenas kernel: ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:18:31 freenas kernel: ada4: Serial Number WD-WCAZA6613270
Oct 29 09:18:31 freenas kernel: ada4: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:18:31 freenas kernel: ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:18:31 freenas kernel: SMP: AP CPU #1 Launched!
Oct 29 09:18:31 freenas kernel: cpu1 AP:
Oct 29 09:18:31 freenas kernel: ID: 0x01000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
Oct 29 09:18:31 freenas kernel: lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
Oct 29 09:18:31 freenas kernel: timer: 0x000200ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
Oct 29 09:18:31 freenas kernel: GEOM: new disk ada0
Oct 29 09:18:31 freenas kernel: GEOM: new disk ada1
Oct 29 09:18:31 freenas kernel: GEOM: new disk ada2
Oct 29 09:18:31 freenas kernel: GEOM: new disk ada3
Oct 29 09:18:31 freenas kernel: GEOM: new disk ada4
Oct 29 09:18:31 freenas kernel: Root mount waiting for: GRAID3
Oct 29 09:18:31 freenas last message repeated 2 times
Oct 29 09:18:31 freenas kernel: GEOM_RAID3: Force device MainRaidZraid3 start due to timeout.
Oct 29 09:18:31 freenas kernel: GEOM_RAID3: Device raid3/MainRaidZraid3 launched (2/3).
Oct 29 09:18:31 freenas kernel: GEOM: raid3/MainRaidZraid3: corrupt or invalid GPT detected.
Oct 29 09:18:31 freenas kernel: GEOM: raid3/MainRaidZraid3: GPT rejected -- may not be recoverable.
Oct 29 09:18:31 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
...

Would any of this be visible if the controller was kaput?

Everything worked fine until I've changed the following:
1. Added one more disk
2. Upgraded FreeNAS from 8.00 to 8.02



Investigating the first one:
Experiment 1:
Remove the disk that was added to the original setup - that being ada1.
Guess what - one of the disks reappeared:
ada2 shifted to ada1 (presumably) and is not visible (that's the disk that was never formatted)
ada3 shifted to ada2 and is now visible
ada4 shifted to ada3 and is visible as it was

The following appears in messages:

...
Oct 29 09:37:44 freenas kernel: da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
Oct 29 09:37:44 freenas kernel: da0: <WD My Book 1130 1012> Fixed Direct Access SCSI-6 device
Oct 29 09:37:44 freenas kernel: da0: Serial Number 574341575A31333538303832
Oct 29 09:37:44 freenas kernel: da0: 40.000MB/s transfers
Oct 29 09:37:44 freenas kernel: da0: 2861556MB (732558336 4096 byte sectors: 255H 63S/T 45599C)
Oct 29 09:37:44 freenas kernel: ada0 at ata0 bus 0 scbus0 target 1 lun 0
Oct 29 09:37:44 freenas kernel: ada0: <CF Card Ver5.04> ATA-5 device
Oct 29 09:37:44 freenas kernel: ada0: Serial Number 14EF070C0A0800484997
Oct 29 09:37:44 freenas kernel: ada0: 100.000MB/s transfers (UDMA5, PIO 512bytes)
Oct 29 09:37:44 freenas kernel: ada0: 7279MB (14909328 512 byte sectors: 16H 63S/T 14791C)
Oct 29 09:37:44 freenas kernel: ada1 at ata2 bus 0 scbus1 target 1 lun 0
Oct 29 09:37:44 freenas kernel: ada1: <WDC WD20EARS-22MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:37:44 freenas kernel: ada1: Serial Number WD-WMAZA4748524
Oct 29 09:37:44 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:37:44 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:37:44 freenas kernel: ada2 at ata3 bus 0 scbus2 target 0 lun 0
Oct 29 09:37:44 freenas kernel: ada2: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:37:44 freenas kernel: ada2: Serial Number WD-WCAZA3050492
Oct 29 09:37:44 freenas kernel: ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:37:44 freenas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:37:44 freenas kernel: ada3 at ata3 bus 0 scbus2 target 1 lun 0
Oct 29 09:37:44 freenas kernel: ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:37:44 freenas kernel: ada3: Serial Number WD-WCAZA6613270
Oct 29 09:37:44 freenas kernel: ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:37:44 freenas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:37:44 freenas kernel: SMP: AP CPU #1 Launched!
Oct 29 09:37:44 freenas kernel: cpu1 AP:
Oct 29 09:37:44 freenas kernel: ID: 0x01000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
Oct 29 09:37:44 freenas kernel: lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
Oct 29 09:37:44 freenas kernel: timer: 0x000200ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
Oct 29 09:37:44 freenas kernel: GEOM: new disk ada0
Oct 29 09:37:44 freenas kernel: GEOM: new disk ada1
Oct 29 09:37:44 freenas kernel: GEOM: new disk ada2
Oct 29 09:37:44 freenas kernel: GEOM: new disk ada3
Oct 29 09:37:44 freenas kernel: Root mount waiting for: GRAID3
Oct 29 09:37:44 freenas last message repeated 3 times
Oct 29 09:37:44 freenas kernel: GEOM_RAID3: Force device MainRaidZraid3 start due to timeout.
Oct 29 09:37:44 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
Oct 29 09:37:44 freenas kernel: GEOM_RAID3: Device MainRaidZraid3 destroyed.
Oct 29 09:37:44 freenas kernel: ct_to_ts([2011-10-29 16:37:15]) = 1319906235.000000000
Oct 29 09:37:44 freenas kernel: start_init: trying /sbin/init
Oct 29 09:37:44 freenas kernel: GEOM: ada2: the secondary GPT table is corrupt or invalid.
Oct 29 09:37:44 freenas kernel: GEOM: ada2: using the primary only -- recovery suggested.
Oct 29 09:37:44 freenas kernel: ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable,
...

Experiment 2
Added back ada1.
Removed ada4 - the only one that was working.
Now:
ada1 is visible
ada2 is not visible (never formatted)
ada3 is visible
(all in their usual places)

Did a nonverbose log this time:

...
Oct 29 09:55:44 freenas kernel: ada0: <CF Card Ver5.04> ATA-5 device
Oct 29 09:55:44 freenas kernel: ada0: 100.000MB/s transfers (UDMA5, PIO 512bytes)
Oct 29 09:55:44 freenas kernel: ada0: 7279MB (14909328 512 byte sectors: 16H 63S/T 14791C)
Oct 29 09:55:44 freenas kernel: ada1 at ata2 bus 0 scbus1 target 0 lun 0
Oct 29 09:55:44 freenas kernel: ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:55:44 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:55:44 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:55:44 freenas kernel: ada2 at ata2 bus 0 scbus1 target 1 lun 0
Oct 29 09:55:44 freenas kernel: ada2: <WDC WD20EARS-22MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:55:44 freenas kernel: ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:55:44 freenas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:55:44 freenas kernel: ada3 at ata3 bus 0 scbus2 target 0 lun 0
Oct 29 09:55:44 freenas kernel: ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 09:55:44 freenas kernel: ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 09:55:44 freenas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 09:55:44 freenas kernel: SMP: AP CPU #1 Launched!
Oct 29 09:55:44 freenas kernel: Root mount waiting for: GRAID3
Oct 29 09:55:44 freenas kernel: Root mount waiting for: GRAID3
Oct 29 09:55:44 freenas kernel: GEOM_RAID3: Force device MainRaidZraid3 start due to timeout.
Oct 29 09:55:44 freenas kernel: GEOM_RAID3: Component ada3 (device MainRaidZraid3) broken, skipping.
Oct 29 09:55:44 freenas kernel: GEOM_RAID3: Device MainRaidZraid3 is broken, too few valid components.
Oct 29 09:55:44 freenas kernel: GEOM_RAID3: Device MainRaidZraid3 destroyed.
Oct 29 09:55:44 freenas kernel: GEOM: ada3: the secondary GPT table is corrupt or invalid.
Oct 29 09:55:44 freenas kernel: GEOM: ada3: using the primary only -- recovery suggested.
Oct 29 09:55:44 freenas kernel: GEOM: ada1: the secondary GPT table is corrupt or invalid.
Oct 29 09:55:44 freenas kernel: GEOM: ada1: using the primary only -- recovery suggested.
Oct 29 09:55:44 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
...

Experiment 3
removed ada2, now the pool is in DEGRADED state and I can see the data.

...
Oct 29 10:43:37 freenas kernel: ada0: <CF Card Ver5.04> ATA-5 device
Oct 29 10:43:37 freenas kernel: ada0: Serial Number 14EF070C0A0800484997
Oct 29 10:43:37 freenas kernel: ada0: 100.000MB/s transfers (UDMA5, PIO 512bytes)
Oct 29 10:43:37 freenas kernel: ada0: 7279MB (14909328 512 byte sectors: 16H 63S/T 14791C)
Oct 29 10:43:37 freenas kernel: ada1 at ata2 bus 0 scbus1 target 0 lun 0
Oct 29 10:43:37 freenas kernel: ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 10:43:37 freenas kernel: ada1: Serial Number WD-WMAZA3416616
Oct 29 10:43:37 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 10:43:37 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 10:43:37 freenas kernel: ada2 at ata3 bus 0 scbus2 target 0 lun 0
Oct 29 10:43:37 freenas kernel: ada2: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 10:43:37 freenas kernel: ada2: Serial Number WD-WCAZA3050492
Oct 29 10:43:37 freenas kernel: ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 10:43:37 freenas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 10:43:37 freenas kernel: ada3 at ata3 bus 0 scbus2 target 1 lun 0
Oct 29 10:43:37 freenas kernel: ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 29 10:43:37 freenas kernel: ada3: Serial Number WD-WCAZA6613270
Oct 29 10:43:37 freenas kernel: ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Oct 29 10:43:37 freenas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 29 10:43:37 freenas kernel: SMP: AP CPU #1 Launched!
Oct 29 10:43:37 freenas kernel: cpu1 AP:
Oct 29 10:43:37 freenas kernel: ID: 0x01000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
Oct 29 10:43:37 freenas kernel: lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
Oct 29 10:43:37 freenas kernel: timer: 0x000200ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
Oct 29 10:43:37 freenas kernel: GEOM: new disk ada0
Oct 29 10:43:37 freenas kernel: GEOM: new disk ada1
Oct 29 10:43:37 freenas kernel: GEOM: new disk ada2
Oct 29 10:43:37 freenas kernel: GEOM: new disk ada3
Oct 29 10:43:37 freenas kernel: Root mount waiting for: GRAID3
Oct 29 10:43:37 freenas last message repeated 2 times
Oct 29 10:43:37 freenas kernel: GEOM_RAID3: Force device MainRaidZraid3 start due to timeout.
Oct 29 10:43:37 freenas kernel: GEOM_RAID3: Component ada2 (device MainRaidZraid3) broken, skipping.
Oct 29 10:43:37 freenas kernel: GEOM_RAID3: Device MainRaidZraid3 is broken, too few valid components.
Oct 29 10:43:37 freenas kernel: GEOM_RAID3: Device MainRaidZraid3 destroyed.
Oct 29 10:43:37 freenas kernel: GEOM: ada2: the secondary GPT table is corrupt or invalid.
Oct 29 10:43:37 freenas kernel: GEOM: ada2: using the primary only -- recovery suggested.
Oct 29 10:43:37 freenas kernel: GEOM: ada1: the secondary GPT table is corrupt or invalid.
Oct 29 10:43:37 freenas kernel: GEOM: ada1: using the primary only -- recovery suggested.
...

ada1 -> ada1
ada2 -> X
ada3 -> ada2
ada4 -> ada3
And note that the one drive that was always working does not have the GPT issue.

Output from gpart show:

=> 6 732558325 da0 GPT (2.7T)
6 4194304 1 freebsd-swap (16G)
4194310 728364021 2 freebsd-ufs (2.7T)

=> 63 14909265 ada0 MBR (7.1G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 11004336 - free - (5.2G)

=> 34 3907029101 ada3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 0 1930257 ada0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)

=> 34 3907029101 ada2 GPT (1.8T) [CORRUPT]
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

=> 34 3907029101 ada1 GPT (1.8T) [CORRUPT]
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834703 2 freebsd-zfs (1.8T)

Output from zpool status:

pool: MainOne
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
MainOne DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ada1p2 ONLINE 0 0 0
5964774385088512135 UNAVAIL 0 0 0 was /dev/ada2p2
ada2p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0

errors: No known data errors

Two questions I have here:
1. Is it that I can not have all 4 disks present, while things do work when there are only 3. Or is it the GPT issue? Or is it the combination of the two factors?
Controller does not seem to be an issue at least because it works fine until the first reboot. So I believe it's got to be the Software.
2. What happened to the secondary GPT table on the two drives and how to recover it? Searching around shows that gmirror might use the secondary gpt for its purposes and that error can be ignored. Is that true?

 

[ProtoSD]

1. Is it that I can not have all 4 disks present, while things do work when there are only 3. Or is it the GPT issue?

I haven't seen this is so long I didn't recognize it, and I don't have a real technical answer about it, but...

I have found with certain older disk controllers, and this doesn't have to do with 48-bit addressing because the ones I had trouble with I KNOW had 48-bit addressing, that there seems to be a limit to the total amount of disk space they can deal with. If you go over that limit, then strange unpredictable things happen, like you're seeing. Maybe someone else here has a better explanation, but I think you are going to need a different controller to get things to work with 4 disks.

 

[dmt0]

A breakthrough!
Booted from gparted live usb, launched gdisk utility and restored the corrupted GPT label.
Hooked up all four disks and I can see them all!
I can see the data.
No GPT errors in messages.
Now I'd only have to resilver the replaced ada2.

So the conclusion is that it was the GPT corruption that was causing this - not letting me see more than 3 disks at once.
And the questions is what caused the corruption...

Initially 3 drives were reused from the old system and only 2 of them had the issue. The third one did not have the issue, but later was replaced with a blank one.
So it may or may not be the old version of FreeNAS...

 

[dmt0]

Except I'm completely confused as to what the replace command should be in this situation:

pool: MainOne
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
MainOne DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
ada1p2 ONLINE 0 0 0
5964774385088512135 UNAVAIL 0 0 0 was /dev/ada2p2
ada3p2 ONLINE 0 0 0
ada4p2 ONLINE 0 0 0

errors: No known data errors

zpool replace MainOne ada2
Gives error No such device in pool.

EDIT:
zpool replace MainOne 5964774385088512135 ada2
is the right command

 

[dmt0]

Finished resilvering, done the other commands as per 'Unofficial FAQ' in protosd's sig.
Did a reboot - no GPT errors.

Everything is fine:

> zpool status -v
pool: MainOne
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
MainOne ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0
ada4p2 ONLINE 0 0 0

errors: No known data errors

Except that the GUI does not see the Serial of the disk that was replaced. Name is " " blank. Serial is "Unknown". Doesn't bother me, but I'd say it's a bug.


Summary:
- Something causes GPT errors (corrupt secondary GPT header) after the first restart after installing 8.02 (and it looks like I'm not the only one (http://forums.freenas.org/showthread.php?3554-GPT-Corrupted)
- One possible good prevention measure could be for FreeNAS to command to zero out beginning and end of hard drives during creation of new volumes. That could be a defensive step, but perhaps not a solution.
- Because of GPT errors on 2 disks I could not have 4 disks at a time, but only 3 disks at a time - weird stuff
- GPT errors are easily resolved using gdisk utility, which might be useful to have on the FreeNAS distro itself - would save a lot of trouble downloading and booting from Gparted and similar linux distro.
- Something is not completely ironed out with drive replacement (even when done solely through command line)


Huge thanks to protosd for taking time and giving me guidance and pointers!