CPU Reporting Information Questions

[geekmaster64]

Hey there,

I searched the forums a bit and unfortunately, every page that I tried to open came up blank.

Attached, is a picture of my server doing a write workload. My questions are:

How can the CPU consumption be over 100%?

Based on the CPU Report, could this be impacting my write speeds?

IRQ - system resource consumption isn't that correct?

Thank you so much! My signature with system information is accurate.

 

[joeschmuck]

I searched the forums a bit and unfortunately, every page that I tried to open came up blank.

Not sure if you mean you page was physically blank or if you just couldn't find data that pertained to you.

Did a Google search for "freenas cpu over 100%" and came up with several hits. Here is the first hit: https://forums.freenas.org/index.php?threads/what-does-cpu-usage-mean-mine-is-often-100.61708/

Hope this helps.

 

[geekmaster64]

Not sure if you mean you page was physically blank or if you just couldn't find data that pertained to you.

Did a Google search for "freenas cpu over 100%" and came up with several hits. Here is the first hit: https://forums.freenas.org/index.php?threads/what-does-cpu-usage-mean-mine-is-often-100.61708/

Hope this helps.

Oh cool! Thank you.

Yes, literally the page was blank. Tried on different browsers and one of my servers just to rule something out. As a note, this was from the search at the top-right in the site, not with Google to query the site or anything else. Not sure what the deal was but I figured it was just a fluke.

Sorry to waste your time!

 

[joeschmuck]

No waste of my time, I was bored. The search engine within the forums is terrible. Please use something like Google and add "freenas" and maybe "freebsd" when you want to search for something.

 

[Chris Moore]

Yes, literally the page was blank. Tried on different browsers and one of my servers just to rule something out. As a note, this was from the search at the top-right in the site, not with Google to query the site or anything else. Not sure what the deal was but I figured it was just a fluke.

No, sometimes the forum search doesn't return links that work. It is a strange issue because it only happens sometimes.

 

[Chris Moore]

How can the CPU consumption be over 100%?

Code:

My FreeNAS Setup:
Dual E5606's
48GB ECC 10600R
1X Pool (iSCSI Shared)
3 VDev's, 2 Mirrored Drives in each: 2TB WD RE4 Drives
1X 250GB SSD L2ARC
1X 128GB SSD ZIL
Quad 1GBe with MPIO

I am going to tack this on here because if you change your signature later (for a new system) it won't make sense to someone down the road.

The answer is because you have a dual socket system and it is giving you 100% per each processor, 200% total.

 

[Chris Moore]

Based on the CPU Report, could this be impacting my write speeds?

Are you having problems with the write speed?
The way your CPU usage is displayed is different than I am accustomed to, so it appears high, but it may not be.
Are you doing deduplication or high compression?
What kind of network connectivity are you using? What is the use the NAS is put to?
I see it is mirrors with L2ARC and SLOG SSD drives. Are you using the system for iSCSI / virtualization?

 

[geekmaster64]

I am going to tack this on here because if you change your signature later (for a new system) it won't make sense to someone down the road.

The answer is because you have a dual socket system and it is giving you 100% per each processor, 200% total.

Thanks, based upon the earlier responses, My understanding is that 100% = 1 Core maxxed out, not socket. Can you confirm? I also changed the "Report CPU usage in percentage" checkbox so my CPU metrics are proper to the 100% is maxxing out the CPU's. See screenshots:

 

[geekmaster64]

Are you having problems with the write speed?
The way your CPU usage is displayed is different than I am accustomed to, so it appears high, but it may not be.
Are you doing deduplication or high compression?
What kind of network connectivity are you using? What is the use the NAS is put to?
I see it is mirrors with L2ARC and SLOG SSD drives. Are you using the system for iSCSI / virtualization?

Not really performance problems but it's a little slower than I expect with writes in the 40MB/s generating that screen-shot and maxxing out the disk activity on the hyper-visor. Yet, when I run crystal disk IO on the hyper-visor to the iSCSI volume without any VM's running I can push over 300MB/s, was just curious what others felt.

I'll answer your questions:

No dedupe, Lz4 standard.

4 1GBe for iSCSI MPIO connectivity to Hyper-V host. 4 1GBe connectivity on the host to match.

For Example: - (See screenshot for real-IP's)

Host:
10.0.0.1 connect iscsi session to 10.0.1.1, 10.0.2.2, 10.0.3.3, 10.0.4.4 on the FreeNAS Server
10.0.0.2 connect iscsi session to 10.0.1.1, 10.0.2.2, 10.0.3.3, 10.0.4.4 on the FreeNAS Server
10.0.0.3 connect iscsi session to 10.0.1.1, 10.0.2.2, 10.0.3.3, 10.0.4.4 on the FreeNAS Server
1 NIC on each is for mgmt 3 for iSCSI

For the profile, I am doing least queue depth instead of round-robin to enhance traffic pattern when it gets busy.

 

[Chris Moore]

for iSCSI MPIO connectivity to Hyper-V host.

Code:

1X 250GB SSD L2ARC
1X 128GB SSD ZIL

What kind of SSDs are these? Also, what interface are they connected to?

it's a little slower than I expect with writes in the 40MB/s generating that screen-shot

That might be able to be improved, if you want to invest in a little hardware.

when I run crystal disk IO on the hyper-visor to the iSCSI volume without any VM's running I can push over 300MB/s, was just curious what others felt.

Depending on the testing method involved here, this could be basically testing the speed of the RAM. FreeNAS uses system RAM for cach when the workload is not synchronous but will immediately commit to disk with a synchronous task, which is wat is usually generated by VMs. The SLOG (Separate ZFS Intent Log) device you have could be too slow to provide any assistance in this task, depending on exactly what you are using.

 

[geekmaster64]

Code:

1X 250GB SSD L2ARC
1X 128GB SSD ZIL

What kind of SSDs are these? Also, what interface are they connected to?


That might be able to be improved, if you want to invest in a little hardware.

Depending on the testing method involved here, this could be basically testing the speed of the RAM. FreeNAS uses system RAM for cach when the workload is not synchronous but will immediately commit to disk with a synchronous task, which is wat is usually generated by VMs. The SLOG (Separate ZFS Intent Log) device you have could be too slow to provide any assistance in this task, depending on exactly what you are using.

250GB L2ARC is a Samsung 850 Evo (not pro)
128GB is a consumer brand like the samsung with 550MB/s writes (can't recall the model off-hand) and they are both attached to a 3GB/s SAS2 backplane.

That makes sense with the memory, I set Crystalmark io to a 32GB file to 'help' overcome the memory but meh. Typically I see the ZIL at 80% disk activity in the reporting / disks/ screen, never above 80% consumption.

With all of that said, do you think I could potentially be maxxing out that 3GB/s SAS connection? It's a single port connection (Thank you HPe) to my HBA.

 

[Chris Moore]

With all of that said, do you think I could potentially be maxxing out that 3GB/s SAS connection? It's a single port connection (Thank you HPe) to my HBA.

No, I don't think it is the SAS connection. I think it is the 128GB SSD. I would know more if I could lookup the model, but I have an old SSD here that could barely make SATA-I speed despite being 'rated' as SATA-III.
If you are willing to spend a little money on it, you might try getting a pair of these and setting them as a striped set:
https://www.ebay.com/itm/Isilon-0B2...S600-IQ5400S-10000X-IQ32000X-SSD/273247200352

These guys are asking WAY too much for these. I would offer them $50 but don't pay more than $65. These are great drives. I have four of them in my server.

 

[geekmaster64]

No, I don't think it is the SAS connection. I think it is the 128GB SSD. I would know more if I could lookup the model, but I have an old SSD here that could barely make SATA-I speed despite being 'rated' as SATA-III.
If you are willing to spend a little money on it, you might try getting a pair of these and setting them as a striped set:
https://www.ebay.com/itm/Isilon-0B2...S600-IQ5400S-10000X-IQ32000X-SSD/273247200352

These guys are asking WAY too much for these. I would offer them $50 but don't pay more than $65. These are great drives. I have four of them in my server.

I think I found my culprit potentially (though I'm not writing near 182MB/s) - here is the SSD I have for my ZIL:

https://www.newegg.com/Product/Product.aspx?Item=9SIA65C3UM8680

 

[Chris Moore]

I think I found my culprit potentially (though I'm not writing near 182MB/s)

That will do it... An SSD for this needs to be super, SUPER fast at writing and it only ever gets read if there is a failure, so the read speed is trivial.

 

[geekmaster64]

That will do it... An SSD for this needs to be super, SUPER fast at writing and it only ever gets read if there is a failure, so the read speed is trivial.

:P

So in your opinion with a VM workload that isn't very write heavy, do you feel it could make more sense to not have a ZIL till I get a speedy one?

 

[Chris Moore]

So in your opinion with a VM workload that isn't very write heavy, do you feel it could make more sense to not have a ZIL till I get a speedy one?

Is it business critical or home use? If it is for home, the worst case is, it is slow, really slow. Still, you can probably deal until you get something better.

 

[geekmaster64]

Is it business critical or home use? If it is for home, the worst case is, it is slow, really slow. Still, you can probably deal until you get something better.

100% home use / lab. Sounds like it would be better than without until speedier comes along?

 

[Chris Moore]

100% home use / lab. Sounds like it would be better than without until speedier comes along?

It is hard to guess exactly, but it is probably twice as fast with this slow drive as it would be without it. If you get a faster drive, you could see it become 3 or 4 times faster. The amount of change depends on the exact composition of the work, but a fast drive will definitely be better.

 

[geekmaster64]

It is hard to guess exactly, but it is probably twice as fast with this slow drive as it would be without it. If you get a faster drive, you could see it become 3 or 4 times faster. The amount of change depends on the exact composition of the work, but a fast drive will definitely be better.

Thanks for the help. Good to know and I appreciate the recommendation.