Understanding %busy / gstat

Status
Not open for further replies.

dddza

Dabbler
Joined
Apr 24, 2012
Messages
25
Hello,

System:

FreeNAS-9.2.1.6-RELEASE-x64 (ddd1e39)
Intel(R) Xeon(R) CPU E5-2603 v2 @ 1.80GHz
32GB RAM
X9DRi-LN4+/X9DR3-LN4+
AOC-SAS2LP-H8IR Controller + BBU
16 x WD20EFRX WD RED 2TB 3.5" SATA3 64MB CACHE

I'm having a serious problem with the speed of my storage. There is quite a bit happening read/write to the unit but I am confused as to how this relates to fixing things.

Please know this is a temporary setup and I am just using FreeNAS (in this instance) while the correct hardware is provisioned for a proper FreeNAS/ZFS setup. For now, we use a AOC-SAS2LP-H8IR controller with RAID10 across 16 x WD 2TB Red disks. FreeNAS simply handles iSCSI to the extent.

%busy is constantly showing:

v5jDx2w.png


What I really do not get is that the read/write does not appear high at all yet performance is incredibly slow. I did migration to this machine which resulted in data writing at 1Gbps (network speeds) and %busy was no where near these values.

p8jzYQd.png


How do I understand this problem?

Thank you in advance.
 
Last edited:

dddza

Dabbler
Joined
Apr 24, 2012
Messages
25
Shortly after my post, here is a typical example. Note how the read speeds are _much_ higher with very similar write speeds yet %busy is half and performance is back.

This will continue up / down at different time intervals.


3rRlZL9.png


Of course I may just be reading this all wrong so again, any assistance in understanding this would be greatly appreciated.
 
D

dlavigne

Guest
Which version of FreeNAS? Are you using the original iSCSI (istgt) or the new experimental kernel iSCSI?
 

dddza

Dabbler
Joined
Apr 24, 2012
Messages
25
Which version of FreeNAS? Are you using the original iSCSI (istgt) or the new experimental kernel iSCSI?

Sorry about that - OP updated to include all the details :)

The iSCSI target is out the box, nothing custom at all.
 
D

dlavigne

Guest
Since this is a test system, it would be interesting to switch to the experimental iSCSI to see if it improves the performance.
 

dddza

Dabbler
Joined
Apr 24, 2012
Messages
25
iSCSI has something to do with gstat results? I thought that's entirely disk related ?
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
I am experiencing same dilemma on my X10S7-F based system.
This is just a guess but I think the SATA link is stuck at 1.5Gbs and not 3Gbs or 6Gbs for SATA 3.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I am experiencing same dilemma on my X10S7-F based system.
This is just a guess but I think the SATA link is stuck at 1.5Gbs and not 3Gbs or 6Gbs for SATA 3.

This is why it's a good idea to test your hardware before setting up your pool...
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
I have performed some test on the pool itself but also on individual drives.
I am currently running badblocks command on single WD RED 4TB drives and I am getting similar results.
I just can't make sense why %busy is maxed out then.
Which parameter does gstat use to calculate the sata link bandwidth?
What relevant piece of information can I use to pinpoint the true hardware or software bottleneck?
Does anybody know how %busy parameter is being calculated?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
gstat isn't calculating the "SATA link bandwidth." Modern disks are capable of about 1.2-1.5Gbit/s, but a current controller can do 6Gbit or 12Gbit/s per channel, so if you had a 3.5" SATA disk on a late generation SAS controller, it'd never show more than 10% busy if it was relying on the link bandwidth.

It is trivial to max out a spinny rust disk by sending it a few hundred operations per second that require seeks. Assuming a 512 byte sector size and 150 operations, you can max it out while transferring only 77 KBytes/sec.

Think about that for awhile.

gstat attempts to give you an idea of percent busy, but it isn't strictly accurate. What it is reporting is the percentage of time the disk actually has an outstanding request. This is about the closest thing you could easily look at to determine how busy the disk actually is short of actually asking the drive for its own estimation of how busy it is.

It is deceptive in some ways because it is most accurately reflecting how much seek traffic the disk can sustain, but you could easily double the throughput to your disk by making all the same requests but just doubling the size of the requests.

In general I would expect the results to be less-useful on a RAID controller because there is a lot of room for multiple operations to be farmed out to multiple drives in ways that are not exactly predictably consistent. I would kind-of expect, for example, if you were doing primarily writes and had configured your RAID card for write-back mode that it'd show a consistently very low utilization percentage right up to the point where the array was actually at its max capacity, at which point it'd start to redline near 100%.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
I agree for the most part, however the SSD doesn't show high throughput either.
I am using the LSI SAS2308 that is on the motherboard and it is flashed with firmware in IT mode. There is no setup I am aware of that will tell me the state of the controller neither a way to adjust its setting.
For the other controllers on the motherboard, it seems I am getting similar results as well.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Again, this kind of thing is very much more difficult to debug on an installed system...
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
If it is difficult to debug on an installed system, them what would be the best approach in thinning out root cause of slow performance on a uninstalled system.
By "installed" do you mean a system without any pools?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
We typically build systems in the shop, let them burn in and do any debugging for a few months, then ship them, rack them, and put them into production.

This makes it easy to open the thing up and putter with hardware while the hardware is still undergoing burn-in and qualification, which is the stage in life that you should be discovering these sorts of things.

There's a little bit of information in

http://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

If you have to work around a pool on an installed system you can't do write tests.

It is fine to use a FreeNAS system image to help qualify your system, but you need to remember that FreeNAS isn't intended for the purpose so it has no special tools to help you do this. As such, you should expect to need to write some scripts and spend substantial time.

It is always tempting to just buy some parts, throw them all together, and then load up an OS and hope it all works out. That doesn't get you a reliable system. When Dell or HP or Supermicro build a system for you, a system engineer with familiarity with the hardware picks components that are known to be compatible usually for a platform being sold by the thousands, and the assembly guys have tests and stuff that they run to make sure that what's shipped is a correctly-functioning platform. When you roll your own, YOU are responsible for all of that, yet most of the time people don't bother. If you didn't test your box for a month before putting it into production ... fail.

Now there are a bunch of things that are wrong or undesirable about your setup and I haven't gone into those. Hopefully your "correct hardware" omits the crappy RAID controller and the CPU that is actually worse than the E5-2609 which I've frequently referred to as a contemptible CPU.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
We typically build systems in the shop, let them burn in and do any debugging for a few months, then ship them, rack them, and put them into production.

This makes it easy to open the thing up and putter with hardware while the hardware is still undergoing burn-in and qualification, which is the stage in life that you should be discovering these sorts of things.

This is why I bought Server grade hardware so I don't have to perform X, Y, Z... extensive series of test.

It is always tempting to just buy some parts, throw them all together, and then load up an OS and hope it all works out. That doesn't get you a reliable system. When Dell or HP or Supermicro build a system for you, a system engineer with familiarity with the hardware picks components that are known to be compatible usually for a platform being sold by the thousands, and the assembly guys have tests and stuff that they run to make sure that what's shipped is a correctly-functioning platform.

Even high quality equipment that have passed stringent quality test may exhibit undesirable behaviour. 100% reliability doesn't exist and even the stress you put on you hardware before putting them into production are not going to be tested 100%. It isn't possible.

Now there are a bunch of things that are wrong or undesirable about your setup and I haven't gone into those.

Why not, obviously something is bothering you, and yet I don't understand what it is. I am not taking an offensive stand, I just try to understand the reasoning behind your judgement.
If you can't justify your remark what the point having this forum at all.
Let's go through what is wrong and undesirable, and maybe what is wrong to you isn't for me.
It's like driving a Formula 1 car on a 30 mph road, or turtle on a Greyhound race.
My point is, I bought my equipment based on many criteria I had in mind, the rest from pieces of information here and there through this forum, other forum, well not the entire web obviously, but made a sizeable effort to pick what I believe would make a good candidate, and of course based on my own experience.
This forum is all about helping people, and pointing out errors so that they can be remediated, if not for me it should serve ground for the next generation of noobs and more seasoned persons after me.

Hopefully your "correct hardware" omits the crappy RAID controller and the CPU that is actually worse than the E5-2609 which I've frequently referred to as a contemptible CPU.

Same comments. Beside, what makes the E3-1241-V3 worse than the E5-2609? What base for comparison are you refering to?

Same about the crappy RAID controller? Beside not being on a PCIe daughter board it seems to have decent specification according to the tech brief, however and I agree, there is no detail specifications about the device itself, and under which condition it behaved, either in IR or IT mode. All I know is that it has 8 SATA 3 interfaces and has 8 PCIe 3.0 lanes to the processor. Which means that there is plenty of room to fully exercise SATA 3 links.

If Freenas is not build to perform such series of test, then what better platform would you recommend. If it was possible, then why not provide us with a set of scripts that we could all use, then we could all try to conform to a Freenas open-standard for testing.
 
Last edited:

jamiejunk

Contributor
Joined
Jan 13, 2013
Messages
134
jgreco is one of the good guys on the forums. I think people like him just get frustrated answering the same questions over and over. Or more specifically tired of people trying to fight them on using hardware that isn't really upto the task for ZFS. Not that yours isn't (honestly I just skimmed over your details).

Unfortunately there isn't a real guide on this stuff. (although sometimes you'll find a sticky like the one on SAS HBAs).
You need to spend months reading this forum and other stuff on the web to try and figure out what the hell kind of hardware to get. Or if you can afford it get something from IX systems and let them deal with the headaches. They already have all this hammered out.

A good bet is to do research, and then post what you are thinking about buying, what your workload is etc and ask for suggestions. That's what I did and I still didn't get it 100% right.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
This is why I bought Server grade hardware so I don't have to perform X, Y, Z... extensive series of test.

What would make you think it doesn't need extensive testing?

Buying "server grade" hardware merely means that you have picked bits of hardware that are optimal for server use. "Server grade" hardware will, for example, typically have a high quality Ethernet chipset rather than a Realtek. "Server grade" hardware will lack superfluous hardware such as audio. "Server grade" hardware should have IPMI and should log ECC errors and in general have good servery properties.

That doesn't mean that there aren't manufacturing defects. "Server grade" hardware isn't magically 100% problem-free. "Server grade" hardware can come dead on arrival. A BIOS can get corrupted. A pin can get bent. A connector can be seated not-quite-right. A power supply can be marginal. A card might not be seated quite right. Or you might have assembled the whole darn thing in a non-ESD-safe environment and caused subtle damage. Even many "professional" shops don't take ESD seriously... while some of us have gone to some expense to set up ESD-safe shops. "Server grade" doesn't buy you hard drives that are 100% guaranteed to be reliable - and that's by far the biggest liability for a NAS.

The guidance I've provided for burn-in suggests that a thousand hours of testing is a good starting point. However, it isn't uncommon to see a server here on the bench for months undergoing various stress testing. The time to discover problems is before you put the gear into production, if at all possible.


Even high quality equipment that have passed stringent quality test may exhibit undesirable behaviour. 100% reliability doesn't exist and even the stress you put on you hardware before putting them into production are not going to be tested 100%. It isn't possible.

"My car's seat belts cannot save me in 100% of accidents." Got it. Are you implying it's fine not to wear the seat belts?

Why not, obviously something is bothering you, and yet I don't understand what it is. I am not taking an offensive stand, I just try to understand the reasoning behind your judgement.
If you can't justify your remark what the point having this forum at all.
Let's go through what is wrong and undesirable, and maybe what is wrong to you isn't for me.
It's like driving a Formula 1 car on a 30 mph road, or turtle on a Greyhound race.
My point is, I bought my equipment based on many criteria I had in mind, the rest from pieces of information here and there through this forum, other forum, well not the entire web obviously, but made a sizeable effort to pick what I believe would make a good candidate, and of course based on my own experience.
This forum is all about helping people, and pointing out errors so that they can be remediated, if not for me it should serve ground for the next generation of noobs and more seasoned persons after me.

I'm going to thank you to not tell me what the purpose of this forum is. I believe I'm the second most active poster here, even if I've mostly been absent for the past half a year, and I believe I have several bits of key documentation to my credit that I've spent significant time and energy writing, maintaining, and updating. Those things aren't really for my benefit, they're for the benefit of everyone else. I think I've done a reasonable share of helping.

"maybe what is wrong to you isn't for me"? Seriously? Hey, fine. I'm not forcing you to read my posts or take my advice. It's not like I've been hanging out in the shop trying out lots of stuff over the years, debugging the problems, posting stickies. It's not like those of us who frequent the forum have some idea about what does and doesn't work. It's not like we haven't debated these issues many times, explained them even more. So since you imply I'm not helping, and since you seem like maybe you don't want to listen to the voice of experience, why you here, bro?

Same comments. Beside, what makes the E3-1241-V3 worse than the E5-2609? What base for comparison are you refering to?

The E5-2603 CPU you have in your originally described test system is The Slowest Piece Of Crap Xeon E5. No hyperthreading. No turbo boost. 1.8GHz. An E3-1230 is nearly twice that fast, with hyperthreading, with turbo boost. Honestly. There's this great new resource called Intel ARK. Avail yourself of the resource.

Same about the crappy RAID controller? Beside not being on a PCIe daughter board it seems to have decent specification according to the tech brief, however and I agree, there is no detail specifications about the device itself, and under which condition it behaved, either in IR or IT mode. All I know is that it has 8 SATA 3 interfaces and has 8 PCIe 3.0 lanes to the processor. Which means that there is plenty of room to fully exercise SATA 3 links.

Some unhelpful jerkoff wrote all about those LSI controllers in the hardware forum, especially touching upon your MFI-based RAID controller.

If Freenas is not build to perform such series of test, then what better platform would you recommend. If it was possible, then why not provide us with a set of scripts that we could all use, then we could all try to conform to a Freenas open-standard for testing.

Yes, I know it's very unfortunate that no one will summarize how to build, burn in, and test your FreeNAS system. And quite frankly yeah it sucks that I have not volunteered even more time to write a set of scripts that you could all use, because I'm so frelling lazy that I prefer to run a lot of tests by hand and/or just whip up a few quick scripts specific to my needs. Writing a comprehensive test suite is a very different thing and past efforts to that end were disappointing, so I haven't bothered for at least a decade.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Apollo,

Your hardware is pretty much straight out of the forum best practices list. You jumped into the middle of OP's thread and then poked jgreco. Depending on the device; hardware sigs don't always show (i.e tapatalk).

I'd just smile and move on. jgreco is far more valuable on your team. :)

Good Luck,
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Ah crap. I totally missed that. This is why thread hijacking is considered evil.

Thanks for catching that mjws00.

#needmorecoffee
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Ah crap. I totally missed that. This is why thread hijacking is considered evil.

Jgreco,

I am glad I did search through the forum and found this thread, otherwise I would have created a new redundant posting and then I would have been set on fire for not following the "Forum Guidelines" which by the way can be found here: ;)

http://forums.freenas.org/index.php?forums/forum-guidelines-read-before-posting.26/

By the way I was not familiar with the term "thread hijacking" within the context of a forum.

I hope we are good here. :D
 
Status
Not open for further replies.
Top