You are barking at the wrong tree. You are deviating from my original issue. Read the heading of the thread. Nothing about raid. So why are you annoyed at me trying to give you a use case. There are other cases, where this can be done similarly Example, in a SAN Frame where you can grow a LUN from 100GB to 300GB. I wanted to know if FreenNAS pool can grow when that is done. When freenas (zfs) cannot do it you seems to have found something else and complaing that I am repeating samething again. Answer should have been zfs is not designed to do that. I told why I asked the question. It worked for someone else, I thought it could be a bug. That is why I started the thread.
Cyberjock has taken a long story and distilled it down to the root cause. You are doing something that is outside the design parameters. ZFS and FreeNAS do not do what you want, at least not automatically. The only people who run into problems like this are people who disregard ZFS design guidelines and attempt to do things like what you're doing. Cyberjock is perhaps a bit more intolerant of that than I am, but fundamentally I agree with him. ZFS is designed to be your RAID controller. It is not intended to run on top of a lower level RAID controller. It is not intended to be used on a SAN.
You can certainly expand your disk, but to do it you may have to do some manual work. No, I am not going to help you do it.
A2 => Isn't it true that Algorithm will try to balance the balance all the available vdevs, so that at the end it behaves like a stripe, after may writes? I have done, my own testing with adding a new vdev to a existing vdev. (Before adding both vdevs were empty). After that all my rw-io's went went equally to both disks. This made me believe, algorithm tends to create a stripe (balaced-io) at the end.
That bears almost no resemblance to your original question:
Q2: No way to concatenate vdev devices, it only does stripe? Is that correct?
The answer to your question is INCORRECT. In a pessimistic situation, where one vdev is 100% full and the other is 0% full, vdev writes are "concatenated". In an ideal situation, where both vdevs are 0% full, writes are "striped". In the real world, where no pool meets those conditions except under rare circumstances, the algorithm actually ends up doing something intelligent. Given random usage patterns, the load may eventually appear to be roughly balanced. However, striping has a very specific meaning in the storage world, as does concatenation, and ZFS does not actually do either one.