ZFS RAID 10 poor performance

[Snufkin]

My system has the following specs :

8 GB RAM
2 x 160 GB SATA HDDs
2 x 200 GB SATA HDDs
Intel Core 2 Duo CPU (E5400)

And Freenas 9.2.1.x , not 100% sure of version number at this point.

Have setup these drives in a ZFS RAID 10 Volume and I'm accessing this using iSCSI. I get about 55MB/s when reading from the iSCSI drive (HDTune benchmark). I can see that each disk reads between 15 and 12 MB/s so they are not running at speed. (a single disk can do +-45MB/s) Aslo the CPU is below 40% and there is about 1GB RAM free with no swap space being used. Surely with this hardware I should be getting +- 90MB/s ...

What settings do I need to have a look at ? Also are there any test I can run locally on Freenas to check if there is something wrong with a disk or a benchmark to determine where a bottleneck is if any? (which I doubt)

Please advise ...

p.s : Just telling me to buy a overkill system is not advice. This setup will get 90MB/s on a Windows system with software raid which is the performance I am looking to get.

 

[cyberjock]

Like I said before.. this isn't Windows, and please stop trying to expect Windows. If you want Windows performance, go to Windows.

ZFS is NOT NTFS, and it is not even remotely correct to compare the two and expect the same.

Your hardware is NOT going to be good performing. That CPU alone is going to bottleneck you no matter what you want to argue.

Good luck (again).

 

[Snufkin]

But what you are saying does not make sense to me. Why do you say the CPU is the bottleneck if it's usage is less than 40% ? How can you justify buying better hardware when the current setup does not seem to be a bottleneck judging by the reports on the web interface.

If I saw 100% CPU usage , no free RAM and disks running at 100% their speed capacity I would agree with you.

 

[cyberjock]

FSBs can be bottlenecks. They do not affect the CPU usage meter. In fact, if you have a FSB you are basically screwed for a whole laundry list of reasons I won't discuss because it's been discussed to death. The bottom line is that you are going to be unhappy with the server for a bunch of reasons and need to get something newer.

 

[mjws00]

I'm not sure that 55MB/s isn't a reasonable speed for that hardware and iscsi on top of zfs. But there are certainly things you can test. Check the drive speeds using dd or iozone. Check the pool speed the same way. Check the network speed using iperf. Check a transfer with a light protocol like ftp. Compare it to cifs. If you throw a faster pool in, like an ssd for testing, you would have another set of points to compare.

Unfortunately there isn't any magic that will make slow gear fast. I have a similar rig that would max 1GBe all day under Windows... but got shelved for FN. Admittedly my concern was RAM, but I did manage ok speeds via smb on an ssd pool to keep disks out of the mix.

The defaults are pretty well optimized. So you're left with changing hardware or the approach.

Good luck.

 

[Ericloewe]

Am I imagining things, or did I explain the FSB issue just yesterday in this exact same scenario?

 

[cyberjock]

Nope, you explained it exactly yesterday. :D

 

[Ericloewe]

Nope, you explained it exactly yesterday. :D

In that case, I'll just leave this here:

Maximum FSB Bandwidth for a Core 2 Duo (if the motherboard and CPU support it): ~10,6GB/s
Maximum Memory Bandwidth for dual-channel DDR2: ~17GB/s
Memory Bandwidth for dual-channel DDR3 1600: ~24GB/s

@Snufkin: Notice the FSB can't even keep up with relatively slow RAM, much less keep up with it and everything else on the system.

 

[Apollo]

Just realized you have 160GB and 200GB Hard drives.
Are they SATA I or II, I assume they are very hold drive right?. They would have a detrimental impact on performance.

I assume you are accessing your iSCSI from a Windows machine, am I right? ( I don't have experience in the iSCSI area )
I would start by looking at your SSD RAID configuration throughput by running scrub on the volume. This should give you a good indication on speed, that you can monitor via Reporting in GUI, "zpool satus" or any other means available.

Then on your Windows machine, I would look at the Ethernet adapter.
If the following settings are available try to disable them:

- Adaptive Inter-Frame Spacing.
- Flow Control.
-Interrupt Moderation Rate.

Since I did these adjustments, I now get 937Mbit/s top throuput on my Gbit network and is sustainable.

Regarding Core 2 Duo performance, I did a quick comparison search versus my Athlon II 240 AMD processor running 2GB DDR2 RAM.
Doing a tomatoe to potatoe comparison, I would say they should be somewhat close in performance.
And I would say that your system should be able to push those SSD further than what you are currently achieving.

When running thoughput test on my Xeon system I was kind of disapointed too as it seems I was not able to exceed the 190MB/s'ish in any volume configurations(if memory serves). CPU was not even hitting the 100% bar either.

Give it a try and let's see what your results are.

 

[vuvanson]

today i'm install FreeNAS 9.2.1.9 on the System Supermicro X10SLL-F/RAM 4GB/E3-1230V3/4x4TB Seagate RAID5/IBM M1015, service iSCSI. and used another server setup OS windows server 2012 connect to NAS by LAN 1Gb/s point to point. I'm copy one file iso 3.5GB when start it's have Max speed 350MB/s and after that Min 140MB/s. i think with speed that is ok. But when i'm copy another file iso 2GB when start speed max 240MB/s, min 60MB/s. i don't know why. anyone help me how to config cache file for this system to get the highest speed copy.
Tks

 

[Apollo]

How are you monitoring disk troughput?
You mentioned 1Gb/s point to point, is that correct as it is not possible to achieve faster than 125MB/s or so.
Throughput as seen from zpool status replication will indicate compound speed from all the active drives including the redundancy of RAIDZ, which biased the true troughput of the data. That is my understanding.
Also, you will have the cache in the way of the transfer which skew the results too, but in a good way as you might be able to see optimal throughput of your network link, when data or portion of the data being accessed is in the cache.

THe other issue is Samba, as stated many time it is single threaded. Try setting up a ftp share and use something like Filezilla to allow multi client transfer.
You can use Iperf, but not quite useful as it biased behavior and doesn't help narrow down the true cause of the issue.
Replication is a good way to see what the max trhoughput can be.

On my Windows PC, I had to reduce or disable Interrupt handling on my Gbit ethernet interface. Having reduced interupt was in fact slowing my transfer significantly. Try disabling it and see how it behaves.
I can easily max out the Gbit lane right of the start and sustain it.
Disable the following if you can:

-Adaptive Inter-Frame Sapcing.
- Flow control.
Interrupt moderation rate,

What does it give you?

 

[Waco]

In that case, I'll just leave this here:

Maximum FSB Bandwidth for a Core 2 Duo (if the motherboard and CPU support it): ~10,6GB/s
Maximum Memory Bandwidth for dual-channel DDR2: ~17GB/s
Memory Bandwidth for dual-channel DDR3 1600: ~24GB/s

@Snufkin: Notice the FSB can't even keep up with relatively slow RAM, much less keep up with it and everything else on the system.

Do you really believe that having a FSB will slow down throughput over a dainty little gigabit link? Even worst case, let's assume both cores are trying to talk to RAM at the same time (which they don't need to)...that's still over 50X more bandwidth than he's getting out of his box (assuming slow DDR2/3).

Telling people to buy overkill hardware is a great way to not have to figure out how to fix things, but this is a configuration issue first and foremost. Apollo's suggestions are a good start. Checksum offload is another to disable on integrated NICs. I don't fully understand why CIFS on anything other than a Windows box is annoyingly slow but telling people to buy bigass servers is not the fix to a configuration issue. I wish I had the time to help fix the Samba server code...it really needs work. My array can do 1.5 GB/s sustained but it's still difficult to max out a gigabit link via CIFS. My crappy netbook hosting a CIFS share on an SSD? Line rate.

Anyway, rant off, but hopefully the OP got an answer that helped instead of just being berated.

 

[Ericloewe]

Do you really believe that having a FSB will slow down throughput over a dainty little gigabit link? Even worst case, let's assume both cores are trying to talk to RAM at the same time (which they don't need to)...that's still over 50X more bandwidth than he's getting out of his box (assuming slow DDR2/3).

Telling people to buy overkill hardware is a great way to not have to figure out how to fix things, but this is a configuration issue first and foremost. Apollo's suggestions are a good start. Checksum offload is another to disable on integrated NICs. I don't fully understand why CIFS on anything other than a Windows box is annoyingly slow but telling people to buy bigass servers is not the fix to a configuration issue. I wish I had the time to help fix the Samba server code...it really needs work. My array can do 1.5 GB/s sustained but it's still difficult to max out a gigabit link via CIFS. My crappy netbook hosting a CIFS share on an SSD? Line rate.

Anyway, rant off, but hopefully the OP got an answer that helped instead of just being berated.

Yes, I do, since it's been amply shown. Everything goes through it - multiple times. That eats up bandwidth real fast.

 

[Waco]

Yes, I do, since it's been amply shown. Everything goes through it - multiple times. That eats up bandwidth real fast.

I fully understand how IO works. You're talking about a multi-gigabyte per second bus with microsecond latency slowing down a 100 MB/s network connection. Sorry, but I've done enough multiprocessor communication to not run with that line.

Running a machine with a FSB does not mean you cannot get good performance from it - older hardware just means you need to spend some time tuning. Buying new hardware isn't always the best solution.

 

[Apollo]

Another piece of information which is a good indication is to run the gstat command.
If any of the Drive show a sustained high value in the %busy column bouncing around 100, then it is a good indication drives are the limitting factor.

 

[cyberjock]

I fully understand how IO works. You're talking about a multi-gigabyte per second bus with microsecond latency slowing down a 100 MB/s network connection. Sorry, but I've done enough multiprocessor communication to not run with that line.

Running a machine with a FSB does not mean you cannot get good performance from it - older hardware just means you need to spend some time tuning. Buying new hardware isn't always the best solution.

There's tons of examples in the forum of users that have had poor performance (despite having good CPUs) that are bottlenecked by the FSB. The reality is that any given byte of data has to come from the drives to the CPU, then go to RAM, then back and forth through ZFS, then back and forth again for your given file sharing protocols. That adds up extremely fast and FSBs quickly become a bottleneck, even with just Gigabit LAN.

Even if FSBs weren't a problem, the systems that ran FSBs are almost always using DDR2 memory. That stuff is crazy expensive to obtain, and it is so expensive it's often more cost effective to just buy a new system that is more modern. Add to that the fact that a high percentage (99%?) of systems that used FSBs also don't know how to boot from GPT, and you've got this line where there's so many treacherous ways for things to go wrong you're better off just avoiding going down that road to begin with. ;)