Unbalanced usage of ZFS tank

[vielhak]

Hi everyone,

we have 4 almost identical FreeNAS systems (9.10-x) with one zfs tank each (consist of 3 or 4 mirrors).
We use iSCSI to present 3 LUNs (1TB each) from each system to 6 HyperV Server (Windows 2012R2).

2 of the systems use their storage totally balanced (377G+378G+378G), e.g.

Code:

NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
tank									2.72T  1.11T  1.61T		 -	84%	40%  1.35x  ONLINE  /mnt
  mirror								 928G   377G   551G		 -	84%	40%
	gptid/5bfd263a-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -
	gptid/5c263646-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -
  mirror								 928G   378G   550G		 -	84%	40%
	gptid/6a5e22f6-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -
	gptid/6a876fe4-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -
  mirror								 928G   378G   550G		 -	84%	40%
	gptid/7a7fde16-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -
	gptid/7aaa7421-9cc8-11e6-8228-90e2bac96788	  -	  -	  -		 -	  -	  -

Two are unbalanced, e.g. (167G+272G+271G+258G)

Code:

NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
tank									3.62T   969G  2.68T		 -	73%	26%  1.52x  ONLINE  /mnt
  mirror								 928G   167G   761G		 -	66%	17%
	gptid/69799f98-da71-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
	gptid/46520b03-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
  mirror								 928G   272G   656G		 -	75%	29%
	gptid/46825eae-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
	gptid/46aeee79-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
  mirror								 928G   271G   657G		 -	76%	29%
	gptid/46db9775-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
	gptid/47083f4c-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
  mirror								 928G   258G   670G		 -	76%	27%
	gptid/4735a3f7-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -
	gptid/47630b6c-d982-11e6-ac25-90e2bac96a0c	  -	  -	  -		 -	  -	  -

Every setting, every format (CSV in HyperV) are identical. All pools are created at once (no later expansion, etc). The unbalanced pool above was just recreated some days ago and after migrating the VMs to the LUNs, it got unbalanced.

Any ideas?

 

[dlavigne]

That's interesting. It's probably worth creating a bug report at bugs.freenas.org so a dev can look at it. If you do, post the issue number here.

 

[vielhak]

Thanks... we will try.

https://bugs.freenas.org/issues/20298

 

[Ericloewe]

Are the drives all identical? Different batches or something?

 

[vielhak]

Yes, every SSD is the same, e.g.

Code:

[root@fnas-01] ~# MegaCli -pdlist -a0 | grep Inq
Inquiry Data: S2RFNX0H601203L	 Samsung SSD 850 EVO 1TB				 EMT02B6Q
Inquiry Data: S2RFNX0H521081F	 Samsung SSD 850 EVO 1TB				 EMT02B6Q
Inquiry Data: S2RFNX0H517532E	 Samsung SSD 850 EVO 1TB				 EMT02B6Q
Inquiry Data: S2RFNX0H518363F	 Samsung SSD 850 EVO 1TB				 EMT02B6Q
Inquiry Data: S2RFNX0H521242K	 Samsung SSD 850 EVO 1TB				 EMT02B6Q
Inquiry Data: S2RFNX0H517601N	 Samsung SSD 850 EVO 1TB				 EMT02B6Q

On the one system we see some kind of unbalanced access pattern on the mirrors now:

Code:

tank									2.72T   807G  1.93T		 -	85%	28%  1.77x  ONLINE  /mnt
  mirror								 928G   313G   615G		 -	85%	33%
	gptid/f0d173fd-8ba2-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -
	gptid/f10038c4-8ba2-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -
  mirror								 928G   265G   663G		 -	85%	28%
	gptid/1646469a-8ba3-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -
	gptid/1675fb8f-8ba3-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -
  mirror								 928G   229G   699G		 -	86%	24%
	gptid/3ca892b6-8ba3-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -
	gptid/3cd805c6-8ba3-11e6-ad71-90e2bac966d4	  -	  -	  -		 -	  -	  -

da0 gptid/f0d173fd-8ba2-11e6-ad71-90e2bac966d4, one disk of first mirror

da2, gptid/1646469a-8ba3-11e6-ad71-90e2bac966d4, one disk from second mirror:

da4, gptid/3ca892b6-8ba3-11e6-ad71-90e2bac966d4, one disk of 3rd mirror:

As you can see the busy-% of the 3 disk mirrors is not very balanced... this contradicts the conclusion in the ticket (above), doesn't it? THe third mirror is mostly idle... why does ZFS does not choose this vdev for (more) writing? It looks like on 01/15 something happened... but there were no significant changes to the system.