Poor performance for AFP/CIFS Share on Mac OS 10.9 Mavericks

[Smitterson]

I have a relatively new FreeNAS box setup (about 2 weeks old):

FreeNAS 9.2.0 x64
Intel i3 4130
16GB ECC DDR3 1600
6x 3TB WD Red
ASRock C226 Mini ITX
Drives configured in RaidZ2
No special configurations—only have Plex plugin running (transmission is installed, but disabled).

I have an AFP and CIFS share pointing to the same dataset. I get terrible performance on the AFP (or CIFS) when accessing from Mac OS 10.9 Mavericks. The CIFS performance under windows is normal (ethernet is maxed out at 100+ MB/s, wireless is as good as it can be in my area).

Under Mac OS, I am seeing around 40 MB/s reads (using CIFS or AFP) and a dismal 1 to 2 MB/s writes. This is on ethernet.

I had similar issues with a now retired Synology Disk Station after updating to Mac OS 10.9 Mavericks this fall. Synology issued an update sometime in November that fixed the issue (i think there were updates/config changes to both Samba and netatalk. Unfortunately, I do not have a non-Mavericks Mac to test with to see if the problem exists prior to Mavericks.

Has anyone else experienced this same poor performance with AFP/CIFS when accessing from a Mac?

Let me know if I need to provide more details on my setup.


Thanks!

 

[cyberjock]

Not a Mac user, but several Mac users have complained with bad CIFS performance relative to AFP. In your case, you are complaining about both. So maybe your Mac isn't up to the task of performing adequately?

 

[Smitterson]

It is a new retina Macbook Pro from October 2013 (quad core cpu, 16gb ram, etc), so it should be fine :)

I'll see if I can gather any more details/do more troubleshooting. I shall report back with my findings.

 

[walshlink]

I have a ReadyNAS Pro 6 I threw 9.2 on over the weekend:

E5300 2.6Ghz CPU
4GB RAM
4x 2TB Hitachi SATA 3.0
RAIDZ2 with encryption

OS X 10.9.2 (BETA)

I was writing at around 37 MBps.

Maybe the BETA fixed the SMB issues...

 

[Dennis.kulmosen]

I can confirm that the FreeNAS 9.2.1 BETA should have solved the performance issue. I am getting very flat-line 110 MB/s on 1GBe on both AFP and the new SMB. With only standard settings and auto tune on. Plus the best part is that you now can use the new SMB2 without issues.

FreeNAS in question is FreeNAS-9.2.1-BETA-b9c6b76-x64

The only setbacks is that the mDNSResponder reports some strange errors in the log (error 50) but i will investigate that before i report it as a bug. And if your connection to the network on the client is somehow lost, the unmounting of the share is very slow. But again i suspect that to be Bonjour (mDNSResponder) related.

My system is a SuperMicro 36-bay chassis, with 20 x 4TB drives installed in a 4 x 5 drive RaidZ, dual 6-core Xeon E5 and 128GB Ram. 4 x 1GBe in LACP and a Dual 10GBe SFP+ card.

Which should be able to saturate a lot of network connections. ;-)

 

[Smitterson]

Thank you for your feedback.

Based on the bug fixes listed in 9.2.1, it sounds like there were some changes to SMB and AFP. I also see that Mac OS 10.9.2 lists SMB as having been updated/patched.

I'll give these a shot when they're officially released--I can't run on betas or RCs.

I've done some additional troubleshooting on my end, but haven't found anything. Where would I find error/warning messages about AFP on FreeNAS? /var/log/messages?

Thanks again

 

[Smitterson]

9.2.1 seems to have addressed my AFP/CIFS share speeds from Mac OS 10.9.1 while on ethernet. :) Now seeing 100+MB/sec

I'm still getting slow speeds via wireless. Doing some troubleshooting now to make sure my wireless signal is not to blame.

 

[NeilSondhi]

I have the same problem, average 48-50 MB/sec on a Gigabit network..it is so frustrating..

I am using Mac OS 10.9.1 - Mac Mini i7, 8 GB RAM

FreeNAS (FreeNAS-9.2.1-RELEASE-x64 (bd35c86) )on Dell Precision 690, 2xXEON processors dual core and 16GB ECC RAM.
RAID configuration: 6x3 TB Seagate 7.2k rpm drive - RAIDZ2

 

[Pete]

I have had similar experiences.

Since FreeNAS v9.2.1 AFP speeds have improved a lot again, but directory listing is still pretty slow, around 4 KB/s sometimes, typically around 20-40 KB/s, never above 100 KB/s. Same when copying a bunch of smaller files, anything smaller than about 1 MB is awefully slow with speeds around 2-6 MB/s (could the harddrives blocksize play a significant role here? I thought I would have enough RAM to avoid any substantial effect here should it even matter).

Typical continuous transfer speeds of large files are fine and around 85-95 MB/s with peaks at 105-110 MB/s.

Naturally I am only seeing speeds of around 10-40 MB/s when doing Timemachine backups over AFP.


In my case this might be related to my hardware or configuration as I was aiming for energy efficiency with my setup, but I fail to see where.
I understand my CPU is kinda slow but it usually does not get close to 100% and I cannot find any other bottleneck (drive io is fine, iperf io is fine (again, since v9.2.1)). I doubt more RAM would improve much as I see the same issues right after booting up when there is lots of free RAM.

Server: FreeNAS 9.2.1 on a 16 GB USB2 drive, ASUS E35M1-I w/ AMD E-350, 16GB RAM, 6x 3TB WD Red in RAIDZ2, Intel NIC.
Client: OS X 10.9.1 on a MacBook Air i7 @1.7 GHz via GBit Ethernet (WLAN is much worse ofcourse).

The onboard NIC on the ASUS E35M1-I used to be fine with FreeNAS v8 btw. but had to be replaced since performance got really bad with FreeNAS v9, so I just gave in and got an IntelNIC as everyone advised to.

I might be able to test different OS X versions soon to see if this is related to just 10.9.x.


Pete

 

[NeilSondhi]

I made some tests on the network, using disk speed test - I got 340 MB/s read and 320 MB/s consistent speed - so I think it not an issue with AFP, but using the TimeMachine I get around 11 MB/s.. it can take weeks to finish my initial backup....

Server HW: FreeNAS (FreeNAS-9.2.1.1-RELEASE-x64 (0da7233)on Dell Precision 690, 2xXEON processors dual core and 16GB ECC RAM.
RAID configuration: 6x3 TB Seagate 7.2k rpm drive (red) - RAIDZ2

Tp-Link Gigabit network

 

[ser_rhaegar]

Time Machine does not just dump the data. It compares data and writes a lot of information about what is being backed up in addition to the data itself. It is a slow process. You'll never match just flat out file transfer speeds.

More details: http://pondini.org/TM/29.html

 

[NeilSondhi]

Thanks for the reply, it seems earlier I used to get 200 MB/s speed on TimeMachine last year - I don't exactly remember but probably that time I was using Mac OS 10.4 and FreeNAS 8..

Now I am scratching my head to get the same performance.. My HW is the same so there must be a SW change..

 

[cyberjock]

Thanks for the reply, it seems earlier I used to get 200 MB/s speed on TimeMachine last year - I don't exactly remember but probably that time I was using Mac OS 10.4 and FreeNAS 8..

Now I am scratching my head to get the same performance.. My HW is the same so there must be a SW change..

I can assure you with all the money in my retirement accounts that you most definitely did NOT transfer at 200MB/sec. I know this because the theoretical limit for Gigabit LAN is about 133MB/sec. So unless you've found a way to break the laws of physics, you are definitely in error.

 

[NeilSondhi]

Hey CyberJock - you are right, i meant throughout Mbit/s and not MByte (I thought it was trivial as when talking about TCP/IP network traffic its always in Mbits).. apologies for the confusion. even though the speed ~10-11 Mbit/s makes the whole need for a NAS meaningless in terms of TimeMachine.

Any bump USB 3.0 attached drive can do better job than that...

 

[cyberjock]

Have you actually let Time Machine do a backup? I think you are unfairly judging performance based on that speed. For example,if it's doing file compare by simple MD5sums from a table, it doesn't take much of a transfer rate to include a table of sums to compare. Time Machine may work differently when using a local disk than a remote disk.

Personally, I think you are putting too much emphasis on the speed you see. I think you should actually let it do a backup and see what happens. The first backup may take longer than subsequent backups, but something tells me it won't take the weeks you have projected based on the speeds you saw.

As for the USB3 thing, I really can't help you much. Many people use Time Machine here successfully. To be frank, as a Windows and Linux user I am a little jealous of how amazing Time Machine really is. I'd say that if you get good network speeds from your server that Time Machine should run just fine if you give it a chance.

 

[NeilSondhi]

In a nutshell - I have a 3 TB USB drive and it took around ~22 hours to finish the initial backup (~280 GB), to have 1+1 backup I wanted to use my NAS so that I do not have to connect my USB drive each time and this would automatically incrementally backup via the network (AFP). but the issue is since last 4 days its still struggling to finish the first initial backup and has done only 80GB so far.

In the past I used to do the same and never experienced this issue, (Mac OS 10.4 and FreeNAS 8.0) then I ran out of space on my FreeNAS so I stopped using it and relied on my humble USB drive for TimeMachine. Now that I have upgraded my FreeNAS to 21TB, I wanted to get back to my old backup strategy but it seems like I will have to keep my Laptop connected for another week - my flexibility and mobility is at danger now.

 

[joelmusicman]

As for the USB3 thing, I really can't help you much. Many people use Time Machine here successfully. To be frank, as a Windows and Linux user I am a little jealous of how amazing Time Machine really is. I'd say that if you get good network speeds from your server that Time Machine should run just fine if you give it a chance.

As a first time Mac owner I dropped my jaw when my first TM backup completed and I opened the app for the first time!


Frankly though I don't understand why people care so much about how long its taking. My initial backup took about a day and a half on wifi-N, but now it's just replication and totally transparent. The PowerNap function means that it can run TimeMachine even while the laptop is asleep (and plugged in to shore power).

 

[KHD]

My experience with AFP...

Mobo ASUS M4A785TD-M EVO 16 GB /AMD Athlon II initial setup SATA 3 x 4 TB WD ST4000DM000 (ripped out of the desktop enclosures). NIC onboard ASUS Realtek (FWIW - I tried an INTEL NIC and it made ZERO difference in my setup)
I am moving data over than growing to a 5 x 4 TB SATA
I wanted to weigh in with some painful lessons learned using AFP on FreeNAS. I struggled for almost a week trying to figure out why I had such awful performance. I believe that the developers need to do some work on figuring out how to make the CNID database more scale and USEABLE. Sure, I'm a dork for being so married to Apple and AFP but that's not the point. The point is that I have YEARS of backups, sparse bundles, iPhone, iPad, and every other of device not to mention the Terabytes of data in images from iPhoto libraries. You see my point. I may *never* use some of this stuff ever again, but if I want or need it, I would like to be able to get to it quickly. I pretty much have all my digital data going back 25 years and I am intent on keeping it. Until recently, I would just mirror my drives, use Google Drive, use box, use dropbox, etc. It was getting to the point that I needed something bigger. Google Drive has a file limit of 10GB, Dropbox and Box are 2GB. Again, not the point. But I have a single Tme Machine backup that has grown to almost 3TB. The "cloud" providers suck for massive amounts of data *AND* large file sizes. My next project is to get rsync working. I will tackle that later. so here are my lessons learned:

1. If you have oodles of files in directories, good luck - you're performance on FreeNAS is going to suck balls.. I don't care how fast a processor, quad Intel NIC, and screaming fast SATA drives, it's still dog slow. Solution? Create sparse bundles, then dmgs. Sparse bundles are definitely a step in the right direction but DMGs keep the dog slow CNID database out of the equation. Someone should figure out how to make it faster. My Time Machine 3TB had NO issues with performance so it's something in the FreeNAS code that needs to be looked at.

2. Don't mix shares with NFS or CIFS shares (duh, but really, don't do it - I battled this for an entire day and had to chown back A LOT).

3. Follow the setup guide VERBATIM. (again, duh, but it seems to make a difference that you don't start the service before the share is created). Make SURE you use Windows/MAC file permissions. Don't flip flop between them. I still don't think the "set permissions recursively works as claimed and if so, what does it do to the CNID database? On that note, don't be afraid to rebuild the CNID database but plan on it taking FOREVER - I'm still not sure it did much good except clear some errors. The performance boost I saw was mostly from reducing file structures.

4. Break out your AFP share carefully. Don't put too many files /directories in a single share. This makes the biggest difference in a SCREAMING system versus a dog slow POS. I was lucky to hit 300 Mb/sec. I frequently get 800Mb/sec + on EXACTLY the same HW. I think the reason for this is that the CNID database is not optimized in Memory or on disk.

5. From #4, STORE the "database path in the share to the root of the share itself. in other words, if your AFP share is something like /mnt/NAS/AFP/iPhoto/ then make your database path the same. The documentation is unclear on where the hell it puts it. It says in the root of the share. Doesn't hurt to ensure it's there IMO.

6. Keep Time Machine backups SEPARATE per MACHINE being backed up - this means created a UNIQUE share for each Machine. See point #1 and #4 if you want to know why.

7. If you have have large spare bundles open and the share "disappears" (i.e. you created a new one which seems to restart the whole AFP service (which I get)) 0- get ready for a world of the dreaded "Beach Balls". Go ahead and force quit finder and disk Utility, it may never recover properly.

8. by FAR and I mean WOW - what a difference - if you are moving/copying/deleting files, ssh in as root or even better, create a user that has the same *exact* name and Full Name and group membership as the file owner. Any large scale file moves, copies and ESPECIALLY deletes is best done from the command line. Pay particular attention to ownership. If you create a directory as root in the command line and copy a bunch of crap over as the user - well - get familiar with chown. Quite frankly, if you don't know command line either learn it or buy a NAS from BestBuy or Amazon...seriously.

anyways, if anyone has a helpful suggestion, chime in. I probably won't respond but wanted to share my experience since I learned so much about this great Open Source project and trolled all the helpful posts. I think scalability needs a second look or at least a way to tune or cache the database for AFP. Keep up the great work!

 

[cyberjock]

Nice write-up KHD. I'm not a Mac user, but it was an interesting read to see the stuff you did and how it came out.

When you say "I think scalability needs a second look or at least a way to tune or cache the database for AFP" do you mean on the FreeNAS side or on the Mac side? If on the FreeNAS side, do you have some recommendation? To be honest, I don't know if any of the developers have Macs. Very few users here have Macs either. AFP seems to be dying from what I've read. Apple seems to be in a position where they are slowly deprecating it and moving to CIFS. I think that's a wise choice too since nobody, and I mean nobody, uses AFP aside from Apple products. I'm not even sure how I'd access an AFP share from Windows, Linux, or my Android phone.

 

[xxup]

Yesterday afternoon I had to copy 867 jpg files (approx 3.62Gb) from my daughter's camera to the server (version 9.2.1.5). I would normally use the Windows PC, but my partner needed to use it for the afternoon. So I plugged the SD card into the MAC (MAC Air Intel i5 with 8Gb ram and SSD) and dragged the files from the SD folder to the drive on the freenas server (HP N40L 8Gb ram 2x2Tb drives mirrored in this volume). The transfer rates from any of our Windows computers over CIFS is around 80-100Mb/sec, but the MAC over CIFS (we don't have AFP configured) managed to peak at 40Kb/sec. I had other things to do so I just let it go overnight.

This morning I had a very long security report (I wish these things had time stamps) from the server. (attached). A quick visit to the server confirmed, what I suspected - the errors started when the file copy started and ended when the copying was completed. Here is the server messages:

May 7 17:32:19 bern kernel: pid 96108 (smbd), uid 1001 inumber 21698 on /var: filesystem full
May 7 17:32:50 bern last message repeated 31 times
May 7 17:34:51 bern last message repeated 121 times
May 7 17:44:52 bern last message repeated 600 times
May 7 17:54:53 bern last message repeated 599 times
May 7 18:00:00 bern last message repeated 298 times
May 7 18:00:32 bern last message repeated 30 times
May 7 18:05:21 bern last message repeated 8 times
May 7 18:18:07 bern kernel: pid 96108 (smbd), uid 1001 inumber 21686 on /var: filesystem full
May 7 18:18:38 bern last message repeated 30 times
May 7 18:20:39 bern last message repeated 118 times
May 7 18:30:40 bern last message repeated 581 times
May 7 18:40:41 bern last message repeated 581 times
May 7 18:50:42 bern last message repeated 583 times
May 7 18:55:21 bern last message repeated 272 times
May 7 19:07:56 bern kernel: pid 96108 (smbd), uid 1001 inumber 21698 on /var: filesystem full
May 7 19:08:27 bern last message repeated 30 times
May 7 19:10:28 bern last message repeated 120 times
May 7 19:20:29 bern last message repeated 590 times
May 7 19:30:30 bern last message repeated 596 times
May 7 19:40:31 bern last message repeated 586 times
May 7 19:45:22 bern last message repeated 283 times
May 7 20:13:11 bern kernel: pid 96108 (smbd), uid 1001 inumber 21698 on /var: filesystem full
May 7 20:16:24 bern last message repeated 186 times
May 7 20:16:24 bern kernel: pid 1473 (dhclient), uid 65 inumber 31367 on /var: filesystem full
May 7 20:16:25 bern kernel: pid 96108 (smbd), uid 1001 inumber 21698 on /var: filesystem full
May 7 20:16:56 bern last message repeated 29 times
May 7 20:18:58 bern last message repeated 118 times
May 7 20:28:59 bern last message repeated 580 times
May 7 20:39:00 bern last message repeated 579 times
May 7 20:42:22 bern last message repeated 196 times
May 7 20:42:23 bern kernel: pid 2674 (smbd), uid 0 inumber 21686 on /var: filesystem full
May 7 20:50:22 bern last message repeated 14 times
May 7 21:03:23 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:03:54 bern last message repeated 30 times
May 7 21:05:54 bern last message repeated 117 times
May 7 21:10:59 bern last message repeated 296 times
May 7 21:11:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:11:01 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:11:34 bern last message repeated 30 times
May 7 21:13:35 bern last message repeated 118 times
May 7 21:15:00 bern last message repeated 83 times
May 7 21:15:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:15:01 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:15:32 bern last message repeated 31 times
May 7 21:17:33 bern last message repeated 119 times
May 7 21:20:00 bern last message repeated 141 times
May 7 21:20:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:20:01 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:20:33 bern last message repeated 30 times
May 7 21:22:00 bern last message repeated 84 times
May 7 21:22:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:22:01 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:22:32 bern last message repeated 30 times
May 7 21:24:33 bern last message repeated 118 times
May 7 21:24:59 bern last message repeated 25 times
May 7 21:25:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:25:00 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:25:21 bern last message repeated 21 times
May 7 21:25:23 bern kernel: pid 2674 (smbd), uid 0 inumber 21698 on /var: filesystem full
May 7 21:25:24 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:25:53 bern last message repeated 29 times
May 7 21:26:25 bern last message repeated 31 times
May 7 21:26:26 bern kernel: pid 3712 (winbindd), uid 0 inumber 21678 on /var: filesystem full
May 7 21:26:27 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:26:58 bern last message repeated 30 times
May 7 21:28:59 bern last message repeated 118 times
May 7 21:29:59 bern last message repeated 59 times
May 7 21:30:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:30:00 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:30:31 bern last message repeated 30 times
May 7 21:32:32 bern last message repeated 119 times
May 7 21:33:00 bern last message repeated 26 times
May 7 21:33:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:33:01 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:33:32 bern last message repeated 31 times
May 7 21:34:59 bern last message repeated 82 times
May 7 21:35:00 bern kernel: pid 2172 (syslogd), uid 0 inumber 21686 on /var: filesystem full
May 7 21:35:00 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:35:15 bern last message repeated 15 times
May 7 21:35:16 bern kernel: pid 91071 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:35:17 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:35:49 bern last message repeated 31 times
May 7 21:37:22 bern last message repeated 90 times
May 7 21:37:23 bern kernel: pid 91071 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:37:25 bern kernel: pid 96108 (smbd), uid 1001 inumber 21689 on /var: filesystem full
May 7 21:37:54 bern last message repeated 29 times
May 7 21:39:54 bern last message repeated 115 times
May 7 21:40:00 bern last message repeated 6 times
May 7 21:40:23 bern last message repeated 21 times
May 7 23:20:23 bern last message repeated 20 times

I can confirm that none of the disks are full. Any ideas? Does this help to identify the problem for Mac users?