Physical vs. Virtual FreeNAS + iSCSI vs. NFS: performance testing results.

Status
Not open for further replies.

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Thanks for the info. I will remake it as you suggested. What I was using was just based off forum posts.
Double check with glabel status that nvme devices are getting a gpt-id. They weren't at some point but I don't know it that's been fixed.
 

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
thanks @soulburn Interesting regarding the network throughput. I'm aware of the benefits of over provisioning visa-vi wear. Definitely agree there.
However..
The transaction group (txg) will be limited by the speed of your network link.
What would the reasoning for a ESXi all-in-one box, where VMXNET3 drivers are used?
Since network speed becomes more or less 'infinity' ...are we back to the [sorry for really clumsy use of terminology] general rule of thumb of 1/8 of <hmhm> times 5 or would something else come into play?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
thanks @soulburn Interesting regarding the network throughput. I'm aware of the benefits of over provisioning visa-vi wear. Definitely agree there.
However..

What would the reasoning for a ESXi all-in-one box, where VMXNET3 drivers are used?
Since network speed becomes more or less 'infinity' ...are we back to the [sorry for really clumsy use of terminology] general rule of thumb of 1/8 of <hmhm> times 5 or would something else come into play?
My understanding was that @soulburn is running FreeNAS on-the-metal, not virtualized, and it is serving as a datastore for other ESXi hosts. If this is true, then network speed is the bottleneck and the usual transaction-group ZIL SLOG size calculations apply.

Or not! I may have misunderstood...
 

soulburn

Contributor
Joined
Jul 6, 2014
Messages
100
thanks @soulburn Interesting regarding the network throughput. I'm aware of the benefits of over provisioning visa-vi wear. Definitely agree there.
However..

What would the reasoning for a ESXi all-in-one box, where VMXNET3 drivers are used?
Since network speed becomes more or less 'infinity' ...are we back to the [sorry for really clumsy use of terminology] general rule of thumb of 1/8 of <hmhm> times 5 or would something else come into play?

My understanding was that @soulburn is running FreeNAS on-the-metal, not virtualized, and it is serving as a datastore for other ESXi hosts. If this is true, then network speed is the bottleneck and the usual transaction-group ZIL SLOG size calculations apply.

Or not! I may have misunderstood...

Yes @Spearfoot is totally correct. I originally was going to use this box for an "all-in-one" for my homelab, but I ultimately decided to push this bare metal hardware box out to a production environment, since it runs so fast. As such I installed FreeNAS on the bare hardware and didn't virtualize it. So yeah. what @Spearfoot said applies. Good point though, @Dice. I would be interested in testing various SLOG sizes on an "all-in-one" box in the future. I'll do that on my next FreeNAS build.
 

soulburn

Contributor
Joined
Jul 6, 2014
Messages
100
Double check with glabel status that nvme devices are getting a gpt-id. They weren't at some point but I don't know it that's been fixed.

Just FYI glabel status does not show a gptid for the nvme device. Also, can you please tell me why that is important, @Mlovelace?
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Just FYI glabel status does not show a gptid for the nvme device. Also, can you please tell me why that is important, @Mlovelace?
The gptid is used to add a drive (really a partition) to a zpool in freeNAS because it is persistent to the partition. When freeNAS adds the device labels for a drive (daX or adaX), it does so on a first come first served basis (more or less); and doesn't necessarily correlate to the sata port or sas channel it's connected to. The problem with adding a device to a zpool in freeNAS by the daX/adaX label, is that the daX/adaX label can change to a different physical drive; however gptid of the partition on the drive regardless of it's daX/adaX label does not change.

It's why creating a pool in the gui uses the gptid's by default.
Code:
  pool: pebcak
state: ONLINE
  scan: scrub repaired 0 in 0h59m with 0 errors on Thu Sep 22 07:09:36 2016
config:

		NAME											STATE	 READ WRITE CKSUM
		pebcak										  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/679198fa-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/685ccd37-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/6930de7a-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/69fad7bc-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/6abffd58-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6b8aeb0b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/6c552a9a-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6d1e9201-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/6de8223b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6eab2101-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/6f69e4ec-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/702385a3-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/70ddf160-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/719cac13-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/725c2621-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/730d224b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/73d024c7-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/74897d7f-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-10									 ONLINE	   0	 0	 0
			gptid/b6e4769f-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/b79f1ad9-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
		  mirror-11									 ONLINE	   0	 0	 0
			gptid/c214f79c-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/c2d50f95-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
		  mirror-12									 ONLINE	   0	 0	 0
			gptid/cd3a77ce-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/cdf82110-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0

errors: No known data errors



That being said, you've only got one nvme device so I think you're g2g.
 

soulburn

Contributor
Joined
Jul 6, 2014
Messages
100
The gptid is used to add a drive (really a partition) to a zpool in freeNAS because it is persistent to the partition. When freeNAS adds the device labels for a drive (daX or adaX), it does so on a first come first served basis (more or less); and doesn't necessarily correlate to the sata port or sas channel it's connected to. The problem with adding a device to a zpool in freeNAS by the daX/adaX label, is that the daX/adaX label can change to a different physical drive; however gptid of the partition on the drive regardless of it's daX/adaX label does not change.

It's why creating a pool in the gui uses the gptid's by default.
Code:
  pool: pebcak
state: ONLINE
  scan: scrub repaired 0 in 0h59m with 0 errors on Thu Sep 22 07:09:36 2016
config:

		NAME											STATE	 READ WRITE CKSUM
		pebcak										  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/679198fa-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/685ccd37-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/6930de7a-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/69fad7bc-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/6abffd58-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6b8aeb0b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/6c552a9a-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6d1e9201-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/6de8223b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/6eab2101-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/6f69e4ec-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/702385a3-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/70ddf160-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/719cac13-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/725c2621-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/730d224b-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/73d024c7-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
			gptid/74897d7f-c924-11e5-90b2-001b2134afcc  ONLINE	   0	 0	 0
		  mirror-10									 ONLINE	   0	 0	 0
			gptid/b6e4769f-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/b79f1ad9-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
		  mirror-11									 ONLINE	   0	 0	 0
			gptid/c214f79c-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/c2d50f95-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
		  mirror-12									 ONLINE	   0	 0	 0
			gptid/cd3a77ce-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0
			gptid/cdf82110-0261-11e6-b6d3-0007432c72e0  ONLINE	   0	 0	 0

errors: No known data errors



That being said, you've only go one nvme device so I think you're g2g.

Oh yeah that's a dumb question on my part! WTF do you do when there is more than 1 nvme drive? I'm sure the devs must be aware of this and will fix it.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Oh yeah that's a dumb question on my part! WTF do you do when there is more than 1 nvme drive? I'm sure the devs must be aware of this and will fix it.
There was a thread back in June talking about the lack of gptid's for nvme devices. You might want to file a bug report about it just in case there hasn't been one yet. If they are already working on it then they'll tag it as a duplicate, no harm no foul.
 
Joined
Mar 22, 2016
Messages
217
Would that mean then that you could not create a FreeNAS system based on NVMe devices?
 

kspare

Guru
Joined
Feb 19, 2015
Messages
507
fwiw here's my 2 cents.

I'm running 3 of these boxes now. The sweet spot I found for SLOG was to use Intel 750 Nvme. I run 256gb of ram with 1.2tb for L2Arc and 400gb for SLOG. I also use the chelsios 10gb nics as I never did get the intel 520's to run as fast. we're currently running about 24 customer terminal servers on one server, and 43 on the other.

We also used iscsi for awhile but don't forget vmware also really loves NFS.

We chose nfs for one simple but very important reason. replicating.

You can replicate your storage with iscsi to another server, but nfs makes it *very* easy.

Your replicated data can be setup as a mounted store that you never access except in the event of an emergency. it's pretty slick.
 

soulburn

Contributor
Joined
Jul 6, 2014
Messages
100
fwiw here's my 2 cents.

I'm running 3 of these boxes now. The sweet spot I found for SLOG was to use Intel 750 Nvme. I run 256gb of ram with 1.2tb for L2Arc and 400gb for SLOG. I also use the chelsios 10gb nics as I never did get the intel 520's to run as fast. we're currently running about 24 customer terminal servers on one server, and 43 on the other.

We also used iscsi for awhile but don't forget vmware also really loves NFS.

We chose nfs for one simple but very important reason. replicating.

You can replicate your storage with iscsi to another server, but nfs makes it *very* easy.

Your replicated data can be setup as a mounted store that you never access except in the event of an emergency. it's pretty slick.

Thanks for the feedback. I am still testing this because I can make the server slow down quite a bit if I really hammer on it. I still have some tweaking to do so I appreciate that you've listed your specs. Do you not miss the benefits of VAAI since you're using NFS?
 

kspare

Guru
Joined
Feb 19, 2015
Messages
507
Thanks for the feedback. I am still testing this because I can make the server slow down quite a bit if I really hammer on it. I still have some tweaking to do so I appreciate that you've listed your specs. Do you not miss the benefits of VAAI since you're using NFS?

Let me put that question back to you. What benefits do you think you will be missing out on? It hasn't affected me at all.
 

soulburn

Contributor
Joined
Jul 6, 2014
Messages
100
Let me put that question back to you. What benefits do you think you will be missing out on? It hasn't affected me at all.

UNMAP to free up storage, XCOPY for vMotion between ESXi hosts, and Warn & Stun for safety.
 

vrod

Dabbler
Joined
Mar 14, 2016
Messages
39
Very interesting thread, I'm currently in the planning myself of whether a physical or virtual box is the way to go.

However with NFS, UNMAP becomes obsolete as VMware is not handling the filesystem (VMFS). Since it's NFS, Freenas will be handling it and won't need the UNMAP command (or similar) to free up space. However there is probably some other goodies of using iscsi, I've just found NFS to be an easier choice in terms of only using the space which is needed. No need to worry about dead space.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
However with NFS, UNMAP becomes obsolete as VMware is not handling the filesystem (VMFS). Since it's NFS, Freenas will be handling it and won't need the UNMAP command (or similar) to free up space.
o_O You have some reading to do...
 

vrod

Dabbler
Joined
Mar 14, 2016
Messages
39
o_O You have some reading to do...
Why? I've worked with VMware for several years and the UNMAP command has always been something you had to run manually in order to free up dead space. It basically creates big files just with 0's and deletes them again. Done it many times to free up dead space on SAN's (both FC and iSCSI). An example:

You have a 100G iSCSI/FC LUN which are formatted as VMFS5 in ESXi. You have another NFS datastore of the same size. You create 2 identical VM's, both consuming 50G of space. Both datastores will in Freenas report as 50% being used. You now delete those 2 VM's. FreeNAS will immediately know that the NFS share just had 50G of files deleted, however this will not be the case on the iSCSI/FC LUN since the data blocks from the block device will not be zero'ed. Therefore FreeNAS won't know if the space is still really in use or not.

So hence, for NFS the UNMAP is obsolete because it's never needed. FreeNAS know that the space is "free" because it handles the file system. You can read up on the UNMAP command here: http://blogs.vmware.com/vsphere/2012/04/vaai-thin-provisioning-block-reclaimunmap-in-action.html
 

jdong

Explorer
Joined
Mar 14, 2016
Messages
59
NFS doesn't need UNMAP to handle deletions, but thin provisioned VMs deleting data does not free space on the NFS data store the way that iSCSI+UNMAP does...


Sent from my iPhone using Tapatalk
 

vrod

Dabbler
Joined
Mar 14, 2016
Messages
39
How will the UNMAP command help you on reclaiming space from VMDK files (or qcow2 for that matter)? It's the Guest OS which will need to free up that space and as far as I know it's not aware of the underlying hypervisor OS and its Datastores. That's why you run tools like sdelete or so..
 

jdong

Explorer
Joined
Mar 14, 2016
Messages
59
How will the UNMAP command help you on reclaiming space from VMDK files (or qcow2 for that matter)? It's the Guest OS which will need to free up that space and as far as I know it's not aware of the underlying hypervisor OS and its Datastores. That's why you run tools like sdelete or so..
Because the guest OS's virtual SCSI driver also exposes an UNMAP command which the guest filesystem driver uses to mark blocks free, either continuously or routinely. That gets passed through as iSCSI UNMAPS.


Yeah you can zero free blocks too and let NFS either punch holes or just compress a run of zeroes but that's relatively crude compared to what VAAI/UNMAP offers. And not all filesystems have a concept of zeroing free blocks. ZFS, ironically, doesn't. Nor does BTRFS on Linux. But both of those support issuing TRIM/UNMAP commands to the storage controller.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Well... The point was that if you wanted to leverage any VAAI primitive via freeNAS you would have to use iSCSI.

Other vendors have built VAAI plugins for their NFS mounts, but I don't see freeNAS creating one without a large user request.
 
Status
Not open for further replies.
Top