Frustrated with write performance

Status
Not open for further replies.

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
Hey Everyone,

I've been fighting with FreeNAS for some time now. I've tried many different zVol setups and I max out at 30-40MBps write speeds (which was RaidZ).

Config: HP DL380 G7 + LSI 9211-8i in IT mode (P20 firmware), 32GB RAM, 8x512GB Samsung Pro 850 SSDs.
Current zVol config: 7x512 SSDs in raidz2 and 1x512 SSD cache/stripe.

I've tried multiple zVol configurations and network configurations (including LaCP and bonding). I'm currently at 1Gbps non-bonded, no ling aggregation.

Any thoughts would be appreciated.

Thanks,
Todd
 
Joined
Dec 29, 2014
Messages
1,135
You didn't mention how you are using FreeNAS. What are the client machines, and how are they accessing FreeNAS?
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
My apologies. It is being used for NFS shares that my ESX hosts use. vSphere 6.7
 
Joined
Dec 29, 2014
Messages
1,135
What does the output of a "zpool status" command look like? I use my FreeNAS for my ESX hosts with NFS, and I can tell you that having a SLOG (separate dedicated log disk, ZFS intent log is how that appears in the GUI) is HUGE! ESX does synchronous write when using NFS, and that is likely what is dragging your performance down. I know that was it for me. I added an Intel Optane card as an SLOG, and my write performance went up 4-6X. My storage network is on 10G, and I can reliably get 4-6G write through when moving big VM's using Vmotion.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Can you provide the output of zpool list -v and place that in code tags just place the word code inside a set of [] brackets.
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
Code:
zpool status:
root@freenas01:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:01:08 with 0 errors on Wed Jun 27 06:46:08 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da8p2	 ONLINE	   0	 0	 0

errors: No known data errors

  pool: zVol01
 state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		zVol01										  ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/c89e97c7-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/c9193aad-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/c981c41c-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/c9ea8f7c-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/ca580449-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/cab938d1-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
			gptid/cb3ce1e6-7e44-11e8-9e69-d485647a4050  ONLINE	   0	 0	 0
		cache
		  gptid/cbc7b478-7e44-11e8-9e69-d485647a4050	ONLINE	   0	 0	 0

errors: No known data errors

zpool list -v:
root@freenas01:~ # zpool list -v
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot							14.9G  1.65G  13.2G		 -	  -	11%  1.00x  ONLINE  -
  da8p2								 14.9G  1.65G  13.2G		 -	  -	11%
zVol01								  3.22T  3.77M  3.22T		 -	 0%	 0%  1.00x  ONLINE  /mnt
  raidz2								3.22T  3.77M  3.22T		 -	 0%	 0%
	gptid/c89e97c7-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/c9193aad-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/c981c41c-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/c9ea8f7c-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/ca580449-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/cab938d1-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
	gptid/cb3ce1e6-7e44-11e8-9e69-d485647a4050	  -	  -	  -		 -	  -	  -
cache									   -	  -	  -		 -	  -	  -
  gptid/cbc7b478-7e44-11e8-9e69-d485647a4050   477G	37K   477G		 -	 0%	 0%
root@freenas01:~ #
 
Last edited:
Joined
Dec 29, 2014
Messages
1,135
Here is what mine looks like.

Code:

		NAME											STATE	 READ WRITE CKSUM
		RAIDZ2-I										ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/bd041ac6-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bdef2899-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bed51d90-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bfb76075-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c09c704a-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c1922b7c-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c276eb75-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c3724eeb-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/a1b7ef4b-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a2eb419f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a41758d7-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a5444dfb-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a6dcd16f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a80cd73c-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a94711a5-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/aaa6631d-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
		logs
		  gptid/01054460-64dc-11e8-ac23-e4c722848f30	ONLINE	   0	 0	 0
		spares
		  gptid/4abff125-23a2-11e8-a466-e4c722848f30	AVAIL


The key item there is the "logs" one. You should be able to delete the cache, then re-add that disk as a log. I suspect that will help your performance. It is only going to go so far because the disks you are using are SATA, so they are a bit slower. I think it would be worth your while to get a single faster SAS SSD to use as the log, and then use the rest for storage. Logs is probably one of the few things where you wouldn't want all the disks in a pool/vdev to be the same.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
You forgot the code tags...
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
Here is what mine looks like.

Code:

		NAME											STATE	 READ WRITE CKSUM
		RAIDZ2-I										ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/bd041ac6-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bdef2899-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bed51d90-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bfb76075-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c09c704a-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c1922b7c-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c276eb75-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c3724eeb-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/a1b7ef4b-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a2eb419f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a41758d7-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a5444dfb-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a6dcd16f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a80cd73c-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a94711a5-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/aaa6631d-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
		logs
		  gptid/01054460-64dc-11e8-ac23-e4c722848f30	ONLINE	   0	 0	 0
		spares
		  gptid/4abff125-23a2-11e8-a466-e4c722848f30	AVAIL


The key item there is the "logs" one. You should be able to delete the cache, then re-add that disk as a log. I suspect that will help your performance. It is only going to go so far because the disks you are using are SATA, so they are a bit slower. I think it would be worth your while to get a single faster SAS SSD to use as the log, and then use the rest for storage. Logs is probably one of the few things where you wouldn't want all the disks in a pool/vdev to be the same.
Let me flip it to logs and see...
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Basically with synchronous writes, each write must be confirmed on your pool (twice once for the ZIL and once for the actual write) before ESXi will send the next one. As you can see this is quite slow. With an SSD SLOG (Separate zil LOG) the write only need to get as far as the SSD before its confirmed and the next one is sent. Then its flushed (from RAM not the SLOG) to disk with the transaction group in an optimized pattern (sequentially).

Yes you read that correctly, we only write to the SLOG and (hopefully) never read from it. We only read from the SLOG when the system crashes and we didn't have a chance to write the TXG (transaction group) to the pool. Think of it as a backup for RAM not a cache.
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
Here is what mine looks like.

Code:

		NAME											STATE	 READ WRITE CKSUM
		RAIDZ2-I										ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/bd041ac6-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bdef2899-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bed51d90-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/bfb76075-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c09c704a-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c1922b7c-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c276eb75-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
			gptid/c3724eeb-9e63-11e7-a091-e4c722848f30  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/a1b7ef4b-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a2eb419f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a41758d7-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a5444dfb-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a6dcd16f-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a80cd73c-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/a94711a5-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
			gptid/aaa6631d-3c2a-11e8-978a-e4c722848f30  ONLINE	   0	 0	 0
		logs
		  gptid/01054460-64dc-11e8-ac23-e4c722848f30	ONLINE	   0	 0	 0
		spares
		  gptid/4abff125-23a2-11e8-a466-e4c722848f30	AVAIL


The key item there is the "logs" one. You should be able to delete the cache, then re-add that disk as a log. I suspect that will help your performance. It is only going to go so far because the disks you are using are SATA, so they are a bit slower. I think it would be worth your while to get a single faster SAS SSD to use as the log, and then use the rest for storage. Logs is probably one of the few things where you wouldn't want all the disks in a pool/vdev to be the same.
What kind of controller are you using, that you have so many drives hooked up?
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Code:
Code tags preserve spacing making it easier to read console output.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
What kind of controller are you using, that you have so many drives hooked up?
My guess is he is using a SAS expander in conjunction with his controller in a rack enclosure.
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
Well I'm over 50MB/s now, which is better, but I get 100MB/s read performance (I know read is faster than write), but these SSDs should perform better than this :(
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Well I'm over 50MB/s now, which is better, but I get 100MB/s read performance (I know read is faster than write), but these SSDs should perform better than this :(
I'm currently at 1Gbps
No, that sounds about right. 125MB/s is your theoretical max speed.
 
Joined
Dec 29, 2014
Messages
1,135
No, that sounds about right. 125MB/s is your theoretical max speed.

What are you using to test the write speed from the ESXi side? Kdragon's math is correct. 1Gb (small b) is 1 gigabit, which equates to 125MB megabytes.
 

tlauder

Cadet
Joined
Jul 2, 2018
Messages
9
What are you using to test the write speed from the ESXi side? Kdragon's math is correct. 1Gb (small b) is 1 gigabit, which equates to 125MB megabytes.
Just doing storage vMotions. Read perf is good (100MB/s), write is around 50MB/s
 
Joined
Dec 29, 2014
Messages
1,135
Just doing storage vMotions. Read perf is good (100MB/s), write is around 50MB/s

Bear in mind that Vcenter also throttles the resources used in a Vmotion. I don't get close to maxing things out unless I have multiple migrations going at the same time. I recall having to tweak some Vcenter options to allow more resources to be used. Unfortunately I don't remember exactly what options those were as it has been a couple of years since I did it.
 
Status
Not open for further replies.
Top