SOLVED Strange AFP share 10GbE bandwidth to 4 different Mac clients

[VictorR]

I should open this by saying this is my first Freenas system. I have Linux and BSD experience, but it has been quite a while since I've been outside the Windows server world. So, in the sake of completeness, i will include (probably) too much information rather than leave things out.

We've got a new 45 Drives (mirrored 125GB SSD boot drives, dual 2.4GHz Xeon E5-2600 v3, 256GB RAM, 30 x WD Re 4TB drives, Rocket 750 HBA, 3x Intel dual 10 GbE NIC cards). Freenas 9.3-stable-201511280648 came pre-loaded on a USB stick that is inserted into the motherboard and used for booting. The NAS that will be used for as shared commercial film post-production/editing storage for large 2K, 4K, 6K resolution files when all is running well.

I created the initial testing volume as Z2 (3 x 10 drives). Then a dataset with 3 AFP shares. To simplify things, I turned on guest access and gave full permission to all on the volume, dataset, and shares(no need on volume and dataset, just making sure it wasn't a problem). AFP3 permissions were also checked (by default). [just realized I need to turn off lz4 on next volume test and final production]

I also initiated 3 of NIC ports(10.0.1.1, 10.0.2.1, 10.0.3.1) for direct NAS-to-Client connection via 15' CAT6 patch cables to Sonnet Twin 10G Thunderbolt to 10GbE ethernet converters into 4 Mac computers:
"Mac-1" = Late 2011 MacBook Pro(OS X 10.11.1, 2.5GHz i7, 8GB RAM)
"Mac-2" = Late 2013 Mac Pro (OS X 10.11.2, 2.7GHz 12-core Xeon E5, 64GB RAM)
"Mac-3" = Late 2013 Mac Pro (OS X 10.10.5, 2.7GHz 12-core Xeon E5, 64GB RAM)
"Mac-4" = Early 2014 Mac PRO (OS X 10.9.5, 3.5GHz 6-Core Xeon E5, 32GB RAM)

Now, 45 Drives in a test of a very similar, or exact same, NAS (with 10GbE switch) using Blackmagic's Disk Speed Test app for video bandwidth was able to get speeds of 724MB/sec write and 600 MB/sec read of an AFP share into a Mac Mini using the same Sonnet Twin 10G converter. Using iSCS shares via Studio Network Solutions' SANmp iSCSI initiator they go 825MB each way. And finally, with a Linux computer they were able to saturate the 10GbE line at ~1.1GB/sec

I first tested each Mac into the same NIC port using the same Sonnet converter, Thunderbolt & ethernet cable, and share. Then, I substituted a different Sonnet and repeated the test. Finally, I tried the each Mac using it's own Sonnet, cables, and share. All were done one at a time and accessed the shares as guest

My results for each of the 4 Macs were essentially(write/read):
Mac-1 = 350/280
Mac-2 = 134/388
Mac-3 = 767/453
Mac-4 = 199/0

What perplexes me the most is that wildly varying performance between different clients. Add to that, write speeds being different on each machine.

Mac-4's zero on read would seem to be a permission problem. Except, it was accessing the shares via the same "guest" account, and permissions as the others. So, that shouldn't be an issue.

I am at complete loss here. The "network" is so simple since they are direct-connect to the NAS ports, that I have trouble thinking it is the problem. If it was just one NIC card, perhaps there could be a problem with it. But, I have tried ports on all three with similar results. The internal raid transfers speeds seem fine when the 45 Drives tech remoted in and ran some tests for another issue. I think the internal speed was ~3-4GB/sec.

Maybe it's the Freenas installation, itself. I'm not expecting simple, quick, solution from everyone here. But maybe some of you have seen similar problems and could give me a "process of elimination" to try and narrow down the potential culprit(and learn what I am dealing with). I'm a few days into this and at a loss.

[edit: realized I'll need to run iperf from the server and clients. ]
[edit #2: it just dawned on me that firewalls may be part of the problem. I need to make sure they are turned off, just to be sure]

Thanks

 

[VictorR]

I thought that maybe anti-virus and/or Mac firewall might be causing the problems. But, all 3 Mac Pros have their firewall turned off. I turned off AV and ran test, got the same results. On my MacBook Pro, I turned firewall and AV off, same results.

I've run a few iperf tests from my MacBook Pro to the NAS, and vice versa

From MB Pro to NAS

From NAS to MB Pro

Bi-directional, initiated from the MB Pro

 

[SweetAndLow]

Try only changing one variable at a time. So use the same cable and freenas nic but change the client. Do you get similar results or different? If different than the client has something strange going on. If similar use the same client and nic but switch out the cables.

You should also try nfs and smb if you don't need afp. Nfs will probably give you the best performance. But deal with that after you figure out the varying performance numbers.

 

[VictorR]

Try only changing one variable at a time. So use the same cable and freenas nic but change the client. Do you get similar results or different? If different than the client has something strange going on. If similar use the same client and nic but switch out the cables.

Sorry if my explanation above wasn't clear. I have tried the same NIC port, CAT6 cable, Sonnet 10G converter, Thunderbolt cable moved between all 4 Macs - same results. I then switched CAT6 cable, and repeated. Next, I tried a different ethernet, Thunderbolt, Sonnet, and same NIC port. Then, that new setup on a different NIC card port. Finally, 3 different NIC ports, Sonnets, cables and Macs connected at the same time.

You should also try nfs and smb if you don't need afp. Nfs will probably give you the best performance. But deal with that after you figure out the varying performance numbers.

I'm waiting to hear back from 45 Drives and Sonnet's tech support. But, I will delete this volume and try those next.
I wanted to try AFP because this is an all Mac shop (excluding the NAS). But hey, Apple is pretty much letting AFP die on the vine(defaults to SMB now), so whatever it takes to get things running smoothly.

Thanks for the suggestion. I hope to try that out tomorrow

 

[VictorR]

Ok, after much frustration, 45 Drives tech reminded me to set interface Frame Rate to 9000 on both Mac/Sonnet and Q30 NAS
The embarrassing part is that I knew to do this. But, somehow forgot this simple detail juggling different assignments
It worked - to a certain degree...(20-25% improvement)

Also, he pointed out that the 0 MB/sec read time on Mac-4 was likely due to OS X 10.9.5 and a driver/compatibility issue. He had created the issue in their lab a few times in the past. So, I'll upgrade to 10.11.2 as soon as possible.

"Mac-1" (above) was connecting at ~580 MB/sec read/write. After the MTU Rate change to 9000, it connects at:
MB Pro to NAS

iperf -c 10.0.3.1 -P 1 -i 1 -p 5001 -f M -t 15 -T 1
------------------------------------------------------------
Client connecting to 10.0.3.1, TCP port 5001
TCP window size: 0.13 MByte (default)
------------------------------------------------------------
[ 4] local 10.0.3.2 port 49371 connected with 10.0.3.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 730 MBytes 730 MBytes/sec
[ 4] 1.0- 2.0 sec 730 MBytes 730 MBytes/sec
[ 4] 2.0- 3.0 sec 729 MBytes 729 MBytes/sec
[ 4] 3.0- 4.0 sec 725 MBytes 725 MBytes/sec
[ 4] 4.0- 5.0 sec 731 MBytes 731 MBytes/sec
[ 4] 5.0- 6.0 sec 734 MBytes 734 MBytes/sec
[ 4] 6.0- 7.0 sec 728 MBytes 728 MBytes/sec
[ 4] 7.0- 8.0 sec 730 MBytes 730 MBytes/sec
[ 4] 8.0- 9.0 sec 730 MBytes 730 MBytes/sec
[ 4] 9.0-10.0 sec 726 MBytes 726 MBytes/sec
[ 4] 10.0-11.0 sec 722 MBytes 722 MBytes/sec
[ 4] 11.0-12.0 sec 724 MBytes 724 MBytes/sec
[ 4] 12.0-13.0 sec 729 MBytes 729 MBytes/sec
[ 4] 13.0-14.0 sec 733 MBytes 733 MBytes/sec
[ 4] 0.0-15.0 sec 10928 MBytes 729 MBytes/sec
Done.

NAS to MB Pro

iperf -s -P 0 -i 1 -p 5001 -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[ 4] local 10.0.3.2 port 5001 connected with 10.0.3.1 port 62761
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 604 MBytes 604 MBytes/sec
[ 4] 1.0- 2.0 sec 697 MBytes 697 MBytes/sec
[ 4] 2.0- 3.0 sec 697 MBytes 697 MBytes/sec
[ 4] 3.0- 4.0 sec 697 MBytes 697 MBytes/sec
[ 4] 4.0- 5.0 sec 697 MBytes 697 MBytes/sec
[ 4] 5.0- 6.0 sec 697 MBytes 697 MBytes/sec
[ 4] 6.0- 7.0 sec 697 MBytes 697 MBytes/sec
[ 4] 7.0- 8.0 sec 696 MBytes 696 MBytes/sec
[ 4] 8.0- 9.0 sec 696 MBytes 696 MBytes/sec
[ 4] 9.0-10.0 sec 697 MBytes 697 MBytes/sec
[ 4] 0.0-10.0 sec 6876 MBytes 687 MBytes/sec

BlackMagic Disk Test

Testing one of the Mac Pros

For our particular uses, ultra-high def film editing, this is pretty good. Getting closer to the ~825MB/sec the manufacturer got. Tomorrow, I'll try iSCSI to see if it can hit the 1GB/sec level

 

[cyberjock]

I worked with someone that had one of those thunderbolt to ethernet adapters (and used the blackmagicdesign) benchmarking tool because it was included (from what I remember). Anyway, the bottom line turned out to be two different issues:

1. That benchmark tool is useless. We'd get low speeds that the end-user was really unhappy about, but eventually I asked him to move real-world data, and he nearly saturated the 10Gb LAN. So to be frank, any benchmark number from that tool means nothing to me and as far as I'm concerned, doesn't even exist.
2. Those thunderbolt converters seem vary wildly in their ability to provide great throughput. They seems to vary wildly from second to second, even with real-world tests. We never could identify if the driver for the hardware sucks, if thunderbolt just can't handle the kind of traffic necessary, or what the problem actually was/is.

Ultimately the end-user simply accepted the speeds that he got with that hardware and the configuration he had. "At least its better than 1 Gb LAN for throughput" and he accepted it.

Later I talked to a Mac authority (name withheld to protect the source) said that there are some good reasons that Macs really don't have a good 10Gb LAN card that is "relatively" inexpensive. Ultimately my source basically said that this rather unusual design using thunderbolt was to get around certain problems, but there are definite limitations that can significantly affect performance as a result.

So I would simply use it and be happy that you aren't limited to the 1Gb throughput. I know, not much consolation, but apparently thems-the-breaks when using Mac (I don't use a Mac so I am not an expert on Macs).

On an unrelated note, I would never recommend that Rocket HBA with FreeNAS. I truly hope you never have a problem with it, and considering the money you spent on the rest of the hardware I'm not sure why you didn't stick with stuff that is "tried and true" like the LSI controllers. The cost of the tried and true LSI stuff isn't outrageous, and probably cheaper than going with the Rocket HBA you are using. If you bought the system from some vendor with that controller, you should be cursing them out right now and demanding satisfaction for their blatant disregard for well-supported hardware. iXsystems won't use Highpoint HBA for good reasons. But I wish you luck!

 

[VictorR]

1. That benchmark tool is useless. We'd get low speeds that the end-user was really unhappy about, but eventually I asked him to move real-world data, and he nearly saturated the 10Gb LAN. So to be frank, any benchmark number from that tool means nothing to me and as far as I'm concerned, doesn't even exist.

Funny, I came to that same conclusion after a couple of weeks of testing. iPerf, iozone, and DD are far more useful. Blackmagic has it's place (like a nice GUI to show others) when its numbers match those other tests

2. Those thunderbolt converters seem vary wildly in their ability to provide great throughput. They seems to vary wildly from second to second, even with real-world tests. We never could identify if the driver for the hardware sucks, if thunderbolt just can't handle the kind of traffic necessary, or what the problem actually was/is.

Agreed. There is something going on at the driver/hardware level. Sonnet is the affordable player in this market, Atto's cost twice as much. And from reading posts by those that have owned both, the price reflects performance and stability

On an unrelated note, I would never recommend that Rocket HBA with FreeNAS. I truly hope you never have a problem with it, and considering the money you spent on the rest of the hardware I'm not sure why you didn't stick with stuff that is "tried and true" like the LSI controllers. The cost of the tried and true LSI stuff isn't outrageous, and probably cheaper than going with the Rocket HBA you are using. If you bought the system from some vendor with that controller, you should be cursing them out right now and demanding satisfaction for their blatant disregard for well-supported hardware. iXsystems won't use Highpoint HBA for good reasons. But I wish you luck!

They offer the Rocket 750 because it can handle 40 SATA drives on a single card. The other option was dual LSI 9201 cards.

I was finally able to isolate the intermittent problems with a disappearing disc to a faulty SATA cable. Three identical brand new WD Re 4TB drives report failure after a few days/week on the same slot. Switching the SATA cable to a different ports on the Rocket card, and the same drive slot/channel repeatedly fails. I'm guessing this cable problem has been a big factor in some of the wildly varying day to day tests results. It finally went to permanent failure yesterday on that single channel, sending up a degraded pool alert. A replacement card and cable are on the way, then I can start all over again running tests at 2x14 vdev RAID10(w/ 2 hot spares) , 3x10 RAID10, 5x6 Z2, 3x10 Z2

As frustrating as the last two weeks have been, it has forced me to learn a lot in a short period of time.

 

[cyberjock]

In all honesty, the dual 9201 would have been the only choice I would have even considered. Read around here on the Rocket 750s and they aren't as trustworth as 'ol reliable LSI'. If the 750s do start creating problems, how are you going to prove it? Will you even know how to prove it? Will it be worth spending another two weeks trying to troubleshoot? Are you expecting the logs to say "Rocket 750 error"?

They're rhetorical questions, and even if you gave me your system, trying to prove it was the 750 without simply doing a hardware replacement seeing the issue go away is not trivial. Lots of lost time will result. Even when I tried to help someone narrow down a problem, it got to the point where I told them "it's gotta be your controller.. nothing else makes sense.... sorry but I can't prove it.. but I bet if you get an LSI the problem will go away". He bought an LSI and the problem went away.

 

[VictorR]

I actually was thinking through the same rhetorical questions to myself while writing my last response.

Not so much the interaction between FreeNAS and the Rocket, just the defective card angle. Your advice adds a whole 'nother layer to it. And, as you point out, I don't have anymore time to spend on problems. This thing is supposed to be online the 1st - that's obviously not gonna happen. But, my client will not tolerate another two weeks of issues.

I actually have a conference call tomorrow morning with one of the guys at 45 Drives who does a lot of their lab testing. They are great about support, so switching to the LSI's won't be a problem.

Thanks for all the help

 

[cyberjock]

I wish one of us mods had one of those 45 drive servers. There's so many questions, so many people have said they used them but had problems, etc. I'd really like to see a FreeNAS pro use one that really can say whether this stuff is worth the money or not. I wonder if they'd give me a diskless chassis for testing... I have my own hard drives... It would probably help their business. :P

 

[Ericloewe]

I actually was thinking through the same rhetorical questions to myself while writing my last response.

Not so much the interaction between FreeNAS and the Rocket, just the defective card angle. Your advice adds a whole 'nother layer to it. And, as you point out, I don't have anymore time to spend on problems. This thing is supposed to be online the 1st - that's obviously not gonna happen. But, my client will not tolerate another two weeks of issues.

I actually have a conference call tomorrow morning with one of the guys at 45 Drives who does a lot of their lab testing. They are great about support, so switching to the LSI's won't be a problem.

Thanks for all the help

An LSI SAS 9211 or 9207 plus two expanders might also be worth considering, since it frees up a PCI-e slot,

I wish one of us mods had one of those 45 drive servers. There's so many questions, so many people have said they used them but had problems, etc. I'd really like to see a FreeNAS pro use one that really can say whether this stuff is worth the money or not. I wonder if they'd give me a diskless chassis for testing... I have my own hard drives... It would probably help their business. :p

I'm not exactly in the market for those things, but I have my doubts about their drive "bays". They look as if the drives are just resting on their connectors...

 

[VictorR]

I wish one of us mods had one of those 45 drive servers. There's so many questions, so many people have said they used them but had problems, etc. I'd really like to see a FreeNAS pro use one that really can say whether this stuff is worth the money or not. I wonder if they'd give me a diskless chassis for testing... I have my own hard drives... It would probably help their business. :p

I can put you in touch with one of their main testing/design guys(writes most of the performance blog posts), if you are interested. They do send out tester units for online review. PM me if you are interested. Can't guarantee anything, but they are fairly open to those kind of things

I'm not exactly in the market for those things, but I have my doubts about their drive "bays". They look as if the drives are just resting on their connectors...

The design for the drive slots works well. They lock into their connector tightly, then a dampened top plate tightly locks down over all the drives in the row and connects to the chassis at 4 points.