Terrible speeds - at wits end

[Dariss]

Hello all from the frigid north of Alaska.

Start off by saying I am very familiar with Linux, but very new to BSD.

Currently we just deployed older servers for FreeNAS 9.1 that are identical except the drive configuration. Both of these machines are barely putting out 2-10MB/s over iSCSI. Machine configs below:

Xeon 5110 @ 1.6GHZ
8GB RAM
Intel(R) PRO/1000 (x2)

Machine 1:
ATA Hitachi HDP72502 A52A (250G 10k RPM)(x15)
- RAIDZ1 5 Drives 1 Spare
- RAIDZ1 9 Drives

Machine 2:
2TB Seagate 7200RPM (x8)
- 2 RAIDZ1 3 drives 1 spare

I can understand the machine 2 running very slow because of the memory constraint, but the first machine should have enough power to at least push ~50MB/s over iSCSI hooked up to a gigabit switch right?

Maybe I underestimate how much memory ZFS uses but I thought the second machine would at least do 30 MB/s.

Where can I start troubleshooting (correctly)? I've lost my mind trying to figure this out already for the last couple of days.

Edit: IPerf Result of machine 1 to the iscsi target.

Code:

[  3] local 10.10.10.10 port 59946 connected with 10.10.10.17 port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0- 1.0 sec  54.5 MBytes  457 Mbits/sec
[  3]  1.0- 2.0 sec  53.6 MBytes  450 Mbits/sec
[  3]  2.0- 3.0 sec  58.1 MBytes  488 Mbits/sec
[  3]  3.0- 4.0 sec  55.1 MBytes  462 Mbits/sec
[  3]  4.0- 5.0 sec  54.6 MBytes  458 Mbits/sec
[  3]  5.0- 6.0 sec  53.5 MBytes  449 Mbits/sec
[  3]  6.0- 7.0 sec  53.1 MBytes  446 Mbits/sec
[  3]  7.0- 8.0 sec  54.4 MBytes  456 Mbits/sec
[  3]  8.0- 9.0 sec  55.6 MBytes  467 Mbits/sec
[  3]  9.0-10.0 sec  53.8 MBytes  451 Mbits/sec
[  3] 10.0-11.0 sec  52.5 MBytes  440 Mbits/sec
[  3] 11.0-12.0 sec  54.2 MBytes  455 Mbits/sec
[  3] 12.0-13.0 sec  57.0 MBytes  478 Mbits/sec
[  3] 13.0-14.0 sec  54.6 MBytes  458 Mbits/sec
[  3] 14.0-15.0 sec  51.9 MBytes  435 Mbits/sec
[  3] 15.0-16.0 sec  53.5 MBytes  449 Mbits/sec
[  3] 16.0-17.0 sec  55.9 MBytes  469 Mbits/sec
[  3] 17.0-18.0 sec  55.9 MBytes  469 Mbits/sec
[  3] 18.0-19.0 sec  54.2 MBytes  455 Mbits/sec
[  3] 19.0-20.0 sec  51.9 MBytes  435 Mbits/sec
[  3] 20.0-21.0 sec  52.9 MBytes  444 Mbits/sec
[  3] 21.0-22.0 sec  54.9 MBytes  460 Mbits/sec
[  3] 22.0-23.0 sec  55.6 MBytes  467 Mbits/sec
[  3] 23.0-24.0 sec  34.5 MBytes  289 Mbits/sec
[  3] 24.0-25.0 sec  54.0 MBytes  453 Mbits/sec
[  3] 25.0-26.0 sec  52.1 MBytes  437 Mbits/sec
[  3] 26.0-27.0 sec  56.5 MBytes  474 Mbits/sec
[  3] 27.0-28.0 sec  54.6 MBytes  458 Mbits/sec
[  3] 28.0-29.0 sec  53.4 MBytes  448 Mbits/sec
[  3] 29.0-30.0 sec  40.4 MBytes  339 Mbits/sec
[  3] 30.0-31.0 sec  54.2 MBytes  455 Mbits/sec
[  3] 31.0-32.0 sec  54.2 MBytes  455 Mbits/sec
[  3] 32.0-33.0 sec  56.8 MBytes  476 Mbits/sec
[  3] 33.0-34.0 sec  54.2 MBytes  455 Mbits/sec
[  3] 34.0-35.0 sec  54.9 MBytes  460 Mbits/sec
[  3] 35.0-36.0 sec  52.4 MBytes  439 Mbits/sec
[  3] 36.0-37.0 sec  56.4 MBytes  473 Mbits/sec
[  3] 37.0-38.0 sec  55.6 MBytes  467 Mbits/sec
[  3]  0.0-38.3 sec  2.01 GBytes  450 Mbits/sec

Looking into machine 2 further - network is only doing 18 Mbit on both interfaces (2 different subnets) (1 crossover direct and one through a switch)

 

[cyberjock]

Yeah, that machine is realyl old. More than likely the hardware can't push out much. ZFS needs some CPU power from a CPU that was made in the last 5 years or so for good performance(your CPU is from 2006). Once the i7s hit and the RAM limits increase dramatically(along with DDR3 support) then you are talking about systems that can better handle the load from ZFS. Also, 8GB of RAM is the minimum for ZFS, and more can make a huge difference. The check summing needs some CPU power, which a CPU with a 1066mhz FSB is going to have problems handling. You've also made the issue much worse because you appear to be using the lowest CPU from the 5100 series.

Also noteworthy is that your iperf test didn't even hit 1/2 of what it should have been close to. Normally you should be over 800Mb/sec and most likely over 900Mb/sec. You didn't hit 500Mb/sec.

While you might have a chance to get performance higher than what it is now, your time is probably better spent on getting newer machines with more RAM. You aren't likely to get performance anywhere near what you are hoping for without spending money on more RAM(which isn't cheap for the older stuff). And there's no saying how much speed you'll get before you just can't push it any more.

You might have better luck with UFS. And you are welcome to try it. But as old as that system is, I'd just look for something newer.

 

[Dariss]

Thank you for the response - what about 16GB RAM (these take 4GB sticks still) and double Xeeon 5140's at 2.33 GHz rather than a single 5110?

I can pick up the 5140's for next to nothing, and I'd rather just upgrade the ram considerably even if it is 300 bucks over buying a whole new system.

 

[cyberjock]

It's really hard to say. Your FSB is probably going to be your limiting factor. The performance increase from getting rid of the FSB is pretty amazing. I guess it depends on how much you want to risk trying to make the system faster. The components already are old. If I were building a system for long term use I'd definitely go with something that is newer vice trying to revive that old system.