Slow mirrored read performance, but not a network or single disk issue

[bubba173]

Hello,

I am free to FreeNAS and so far really like the product.

However, I try I am experiencing unexpectedly slow read performance from my HDDs.

Here my setup:

# FreeNas on VM within Hyper-V -> all installed perfectly fine and the setup was smooth
# 4x 4TB WD Re Enterprise disks (single disk sequential read performance = 170 megabytes/second) set up as 4way mirror in order to test striped read performance across all mirrors [the disks are passed through, not virtual disks nor is the controller passed through, just the physical drives]
# 1x Virtual Hard Disk that is stored on a server NVME drive (over 2 gigabyte/second read performance) for testing purposes -> set up in FreeNas as single stripe
# 10 Gbe network connected to 10Gbe Switch and Windows Lan
# Sharing over SMB

I see the following results:

# I stored some larger test files (I am interested in sequential read performance) on the NVME based virtual disk and then shared the dataset over SMB and tested to read sequentially.
-> The read speed is fantastic, with compression turned on it constantly hits the 10 gigabit/second mark, thus fully saturating the network, perfect, I could not wish for more. However, this was just to exclude any network bottlenecks, no bottlenecks, big check mark here!!!

# Next, I also stored files on the 4way mirror that is built on top of the 4x 4TB HDDs. Here, with compression enabled the read speed only reaches around 450 megabytes/second. Excluding any overhead the theoretical throughput should be 4x170 = 680 megabytes/second. I was at the very least hoping to gain a read performance of close to 600 megabytes/second especially with compression turned on because I have allocated quite some beefy CPUs to the virtual machine.

I know many will say that this is already a pretty good throughput. I agree, but it won't hurt to aim higher, right? So my question here is:

Are there any tunable parameters I can try out? What about the sector size? Where can I see what the sector size is that FreeNas set for the ZFS for the drives? Obviously the network can be excluded as bottleneck as described above. So I am very keen on learning what tunable parameters there are for mirrors and HDDs in particular.

Hoping to hear from some of you. By the way, I am Matt and as said I am pretty new to Freenas, and so far I really love the platform and also the community here.

Thanks for your ideas and suggestions.

 

[sfcredfox]

4x 4TB WD Re Enterprise disks (single disk sequential read performance = 170 megabytes/second) set up as 4way mirror in order to test striped read performance across all mirrors [the disks are passed through, not virtual disks nor is the controller passed through, just the physical drives]

Sounds like you created a four drive stripe, not mirror right? If they were mirrored, you'd only be getting the aggregate performance of the number of mirrored vdevs you created. If you created a four drive stripe, you'd expect roughly four drives of performance.

Since we're talking SMB, hit the forum for some threads about samba performance and being single threaded. I don't know if that went away with these later versions, but at one time, samba was constrained to a single core and people had throughput issues with CPU, not drive throughput. Might be worth double checking. That was why a couple of smart people said fewer higher ghz CPUs were better for file sharing than many-core processors with a lower clock rate. Again, not sure if that's still the case.

Not sure what your overall goal and disk config will be, but you can compare a couple of disk configs to see which one will give you closer to what you're looking for (mirrors/*Z1/Z2). *People don't like Z1 in some cases because of the rebuild time, see cyberjock's post

 

[bubba173]

No, I am absolutely certain I set it up as a mirror. 4 way mirror so to speak and I checked, all 4 discs are read from concurrently. But the throughput on each disk was only slightly above 100 megabytes per second while the drives can do 170 on sequential reads.

Also i excluded network as a bottleneck. I have a 10gbe setup and tested by passing through an nvme disk and got close to 10gigabit reads from freenas.

Everything works perfectly well except that each disk only reads at 100 megabytes per second instead of utilizing full capacity.

So I am looking for micro optimizations here such as sector size or so.

Sounds like you created a four drive stripe, not mirror right? If they were mirrored, you'd only be getting the aggregate performance of the number of mirrored vdevs you created. If you created a four drive stripe, you'd expect roughly four drives of performance.

Since we're talking SMB, hit the forum for some threads about samba performance and being single threaded. I don't know if that went away with these later versions, but at one time, samba was constrained to a single core and people had throughput issues with CPU, not drive throughput. Might be worth double checking. That was why a couple of smart people said fewer higher ghz CPUs were better for file sharing than many-core processors with a lower clock rate. Again, not sure if that's still the case.

Not sure what your overall goal and disk config will be, but you can compare a couple of disk configs to see which one will give you closer to what you're looking for (mirrors/*Z1/Z2). *People don't like Z1 in some cases because of the rebuild time, see cyberjock's post

 

[sfcredfox]

No, I am absolutely certain I set it up as a mirror.

Well, this would not be ideal for performance.

If you run a

Code:

zpool list -v

and see:

Code:

Poolname
  mirror
	deviceid
	deviceid
	deviceid
	deviceid

...then you're only giving the system one device to read/write instead of four. Remember, a mirror is one device that copies the data to another device for redundancy. Just a guess, but if you got 450MB/s read out of a single mirror, I'm betting ARC helped you out, and that's not the actual read performance from that pool.

If you want better throughput, try the suggested pool configs from above.

 

[sfcredfox]

4 way mirror so to speak and I checked, all 4 discs are read from concurrently.

Furthermore, that's not what that means. It means one device that mirrors (copies) it's data to other devices in case of failure.

If you want the system to read from four devices simultaneously, spreading the load evenly between them, you want a stripe. I suggest ensuring you're tracking the differences and nomenclature for ZFS:
http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/

 

[sfcredfox]

Lastly, read the manual and some beginners guides, they'll explain vdevs, ZFS raid levels, building pools, etc.

 

[bubba173]

I need to disagree with you on most points here, based on factual evidence:

As mentioned, I definitely set up a mirror with the 4 disks.

Then, I copied a folder, 120 gigabytes in size, (memory allocated to Freenas is only 16gb), so the read performance has nothing to do with ARC.

So based on the evidence I see and now post here, the reads are definitely across all the disks. You can clearly see this in Reporting below:

I am open to suggestions to tweak certain mirror parameters, but claiming this is not a mirror and reads are not performed on all disks concurrently is factually wrong.

Well, this would not be ideal for performance.

If you run a

Code:

zpool list -v

and see:

Code:

Poolname
  mirror
	deviceid
	deviceid
	deviceid
	deviceid

...then you're only giving the system one device to read/write instead of four. Remember, a mirror is one device that copies the data to another device for redundancy. Just a guess, but if you got 450MB/s read out of a single mirror, I'm betting ARC helped you out, and that's not the actual read performance from that pool.

If you want better throughput, try the suggested pool configs from above.

 

[bubba173]

I prefer not to get pedantic about terminology. A mirror copies incoming data onto all the devices, that are defined in the mirror, in identical fashion. Let's not disagree on word choices. Then, for reads, different mirrors behave differently. Some mirrors read from a single disk that currently has the least I/O, other mirrors read from the fastest disk, again others can read from all disks concurrently to improve read throughput. That seems to clearly be the case with FreeNas (I posted the screenshots above).

As I said in my initial post, I do not want a pure stripe because I want at least a minimum amount of redundancy. Hence my initial choice of a mirror, especially given FreeNas can read from all mirrored disks at the same time to satisfy a single-threaded read request.

Furthermore, that's not what that means. It means one device that mirrors (copies) it's data to other devices in case of failure.

If you want the system to read from four devices simultaneously, spreading the load evenly between them, you want a stripe. I suggest ensuring you're tracking the differences and nomenclature for ZFS:
http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/

 

[bubba173]

Thank you, but I find this pretty condescending. I think I described my issue perfectly well, with supporting figures, screenshots and numbers. So far, I am not sure your claims are correct because most of them are disproved by the screenshots I posted. Then What does ZFS and some of the other terms have to do with my particular issue? You are basically assuming I do not understand a thing about NAS or file system/servers when I only mentioned I am new to FreeNas. My problem is not solved by reading the manual. I consulted the manual where necessary, already, and I could not extract help with my particular use case. That is why I came here to ask. Being brushed aside with a comment "go and read the manual" seems quite useless.

Lastly, read the manual and some beginners guides, they'll explain vdevs, ZFS raid levels, building pools, etc.

 

[sfcredfox]

Hey bro, I put zero judgement/condescension into my posts to people, just pure opinions/recommendations based on a good few years of experience. I was once a beginner too, and didn't appreciate it, so I don't bust balls. If you infer that from my recommendations, sorry, but not implied. If you want to see condescending posts, go ask some of these questions to cyberjock or a few of the other true experts. I don't claim to be an expert, and I don't treat people that don't know as much as me like they're stupid. We're all here to help each other make sweet storage systems and learn stuff.

My recommendation to read that material is genuine and most of the people who ask beginner questions (like a guy who joined the forum on Friday) would get really good clarifications to some of these early questions, I know I did. Cyberjock as a PPT that would help you out, I just couldn't get it to load.

I won't argue with you about the config. There are tons of posts on here regarding performance, and I'm fairly confident there aren't any that will recommend a single mirror. Just my opinion.

 

[bubba173]

Did you even look at my screen shots or read my requirements and initial problem? Apparently not.

Your insisting claim, I quote you: "...then you're only giving the system one device to read/write instead of four. Remember, a mirror is one device that copies the data to another device for redundancy. Just a guess, but if you got 450MB/s read out of a single mirror, I'm betting ARC helped you out, and that's not the actual read performance from that pool."

Is entirely incorrect. Here again the screen shots:

So with all due respect, your talk about beginners, documentations, does unfortunately not help me whatsoever. Apparently you seem to not know how a mirror in FreeNas works. And that is 100% fine. I was asking people who know and who have used Tunables in order to tweak performance of mirrored disks.

I appreciate the good intention and spirit in which it was written. I simply did not appreciate the tone in which it was worded, but we can agree to disagree on this.

Thanks again for your attempt to help.

Hey bro, I put zero judgement/condescension into my posts to people, just pure opinions/recommendations based on a good few years of experience. I was once a beginner too, and didn't appreciate it, so I don't bust balls. If you infer that from my recommendations, sorry, but not implied. If you want to see condescending posts, go ask some of these questions to cyberjock or a few of the other true experts. I don't claim to be an expert, and I don't treat people that don't know as much as me like they're stupid. We're all here to help each other make sweet storage systems and learn stuff.

My recommendation to read that material is genuine and most of the people who ask beginner questions (like a guy who joined the forum on Friday) would get really good clarifications to some of these early questions, I know I did. Cyberjock as a PPT that would help you out, I just couldn't get it to load.

I won't argue with you about the config. There are tons of posts on here regarding performance, and I'm fairly confident there aren't any that will recommend a single mirror. Just my opinion.

 

[garm]

Ya there are some factual errors in this thread. But sticking to the point, I think (won’t do the math now) that this is an I/O issue. A birds eye view of ZFS mirrors is that you get the throughput of all the disks but the I/O of one. There is little performance gain from having a wide mirror, what you do get is high redundancy. You should try to increasing your I/O by destroying your pool and rebuilding it with dual mirror vdevs of two disks each and rerun your tests.

 

[bubba173]

Thanks, I can try that but won't throughput not then be cut in half? For reads, with your suggestion only 2 disks will be concurrently read from, is that a correct guess? Currently all 4 disks can be concurrently read.

But I will play along and will test and report back.

Thanks

Ya there are some factual errors in this thread. But sticking to the point, I think (won’t do the math now) that this is an I/O issue. A birds eye view of ZFS mirrors is that you get the throughput of all the disks but the I/O of one. There is little performance gain from having a wide mirror, what you do get is high redundancy. You should try to increasing your I/O by destroying your pool and rebuilding it with dual mirror vdevs of two disks each and rerun your tests.

 

[garm]

Well, think of it like this. A single block can only be read once. So it doesn’t matter that there are 4 copies of the same block. What you want is a steady stream of sequential blocks. So the performance you care about it that of the pool, not individual vdevs. To be simplistic, imagine reading a file off the pool that is evenly split between two vdevs. First block is read from vdev1 and while it’s in flight vdev2 is already ready with the second block, and so on. A ZFS pools actual I/O can be far beyond that of the individual disks by stacking lots of mirror vdevs. A rule of thumb is that regardless of the type of vdev deployed, to increase performance you increas the number of vdevs in a pool.

 

[bubba173]

I tested with a 2x2x4TB mirror and read throughput performance is slightly below the one of a 4x4TB mirror.

Also, I am not sure I agree with your statement above. There are 4 copies of the same, identical data, on each disk. Technically, a read request can structure the request by reading about equally sized chunks of the total data request off all 4 drives at the same time. Technically there is zero problem doing so. And FreeNas in fact seems to exactly behave that way with a 4x XTB mirror setup. The screenshot clearly shows that my 4x4TB mirror reads at almost an identical throughput from all 4 drives at the same time. The 2x2x4TB mirror does so, too, albeit at less throughput per disk.

Edit: I could not attach the screenshot, received an error, not sure why...

Well, think of it like this. A single block can only be read once. So it doesn’t matter that there are 4 copies of the same block. What you want is a steady stream of sequential blocks. So the performance you care about it that of the pool, not individual vdevs. To be simplistic, imagine reading a file off the pool that is evenly split between two vdevs. First block is read from vdev1 and while it’s in flight vdev2 is already ready with the second block, and so on. A ZFS pools actual I/O can be far beyond that of the individual disks by stacking lots of mirror vdevs. A rule of thumb is that regardless of the type of vdev deployed, to increase performance you increas the number of vdevs in a pool.

 

[garm]

Well what do you know, real world experience beats book reading ^^ then I have no idea why you can’t push your drives to specification. What is the channel throughout on your disk controller? Is it the same interface as your NVMe?

 

[bubba173]

No nvme drives my definition are always on the pcie bus on which physical disks are never connected. The Hdds are on the sata3 ports. Each one has 6gigabit per second throughput hence no bottleneck here. I can only conclude that freenas does not optimize the read throughput in this particular setup. I checked, a pure stripe gets to almost 640 megabytes per second for all 4 drives. So, mirrored reads happen concurrently, just not in an optimized fashion. I am willing to bet the split of the job is not implemented optimally. Just a guess.

Anyway, thanks for your attempt to help.

Well what do you know, real world experience beats book reading ^^ then I have no idea why you can’t push your drives to specification. What is the channel throughout on your disk controller? Is it the same interface as your NVMe?

 

[garm]

So increasing the vdev count from one to four shows a 50% performance increase, but two vdevs show a slight decrease? Something is iffy with those numbers. How do you perform the tests?

 

[bubba173]

No the previous post described a stripe across 4 disks. I only performed this test to see whether FreeNas is capable of reading from 4 four drives and at what combined throughput. I only did that for test purposes.

My remaining problem is that FreeNas does not make full use of available throughput resources on the disk when configured in a 4 way mirror. Again, the purpose of this 4 way mirror is resiliency while at the same time maximizing read throughput. So far no other configuration has reached the same read throughput with 4 mechanical disks while at the same time providing resiliency.

So increasing the vdev count from one to four shows a 50% performance increase, but two vdevs show a slight decrease? Something is iffy with those numbers. How do you perform the tests?

 

[sretalla]

I think it might be useful for you to read this article:
https://constantin.glez.de/2010/06/04/a-closer-look-zfs-vdevs-and-performance/

Particularly the section titled:
Mirrored Performance Considerations

Basically it outlines why you are seeing more than 1 disk worth of performance, but less than 4.