afp: cnid_dbd hangs with high cpu load

[giox069]

We are running FreeNAS-8.3.0-RELEASE-p1-x64 (r12825) on VmWare ESXi 5.1. The VM has 16GB of RAM. FreeNAS is used as afp fileserver, with 4 afp shares. Authentication is done with active directory.

Every 2 days, at a random time, one of the four cnid_dbd processes hangs with a 100% cpu load.

Here is the output of ps:

Code:

[root@mynas] /mnt/vol1/scripts# ps -ax -o %cpu,pid,command | fgrep cnid
 0.0 40847 /usr/local/sbin/cnid_metad
100.0 48154 /usr/local/sbin/cnid_dbd /mnt/vol1/data1 8 4 default log_note
 0.0 99164 /usr/local/sbin/cnid_dbd /mnt/vol1/dati2/Scansioni 7 4 default log_note
 0.0 99165 /usr/local/sbin/cnid_dbd /mnt/vol1/dati2/MMCC 8 4 default log_note
 0.0 99416 /usr/local/sbin/cnid_dbd /mnt/vol1/dati2/FORMATS 11 4 default log_note
 0.0 17510 fgrep cnid

The share data1 is unavailable to the users: they get an error when trying to access the share.

This happens randomly, during the day with normal afp server usage, but also during the night without user access to afp.

As a temporary solution we "kill -9" the cnid_dbd process, which is automatically restarted and clients can then access the share. But once every 3-5 kills, the cnid2.db file gets corrupted, so we must stop afp, kill cnid_dbd, delete cnid2.db and restart afp.

There are no error logged in .AppleDB/db_errlog at the time of the problem happened.

Any idea ?
Thank you
Giovanni

 

[petr]

I am having exactly the same problem on FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty). Very annoying - all afp clients cannot access the share and only solution is to kill the process.

giox, did you ever solve your problem?

also, I am not aware of any unclean shutdowns on my system, running on a UPS and using NFS extensively without any trouble.

 

[giox069]

Unfortunately the problem is still here. We usually have 4 cnid_dbd hangs per week. And that is not acceptable in a production environment. So we are analyzing and planning a migration to a Windows 2008 R2 server and access the files via smb://
In the meanwhile I developed a script which every night does the following:

1. stop netatalk via /usr/local/etc/rc.d/netatalk stop
2. kill -9 all remaining cnid_dbd
3. rebuild all CNID DB of our volumes with two commands: dbd -re vol_path and dbd -rf vol_path
4. restart netatalk

Every time it is run (it takes 2 hours in our fileserver!), the script finds some inconsistencies with extended attribute files, and sometimes with the CNID dbd. But stopping and starting netatalk and cnid_dbd allows us to live (survive :) ) without cnid_dbd hangs for the following day.

If you need this script, please post a personal message with your email address to me in this forum and I will send you a copy.

Giovanni

 

[cyberjock]

giox069- so why not just get rid of AFP and use SMB on FreeNAS now? Then you wouldn't need to buy a license of server 2008 R2....

 

[petr]

Unfortunately the problem is still here. We usually have 4 cnid_dbd hangs per week. And that is not acceptable in a production environment. So we are analyzing and planning a migration to a Windows 2008 R2 server and access the files via smb://
In the meanwhile I developed a script which every night does the following:

1. stop netatalk via /usr/local/etc/rc.d/netatalk stop
2. kill -9 all remaining cnid_dbd
3. rebuild all CNID DB of our volumes with two commands: dbd -re vol_path and dbd -rf vol_path
4. restart netatalk

Every time it is run (it takes 2 hours in our fileserver!), the script finds some inconsistencies with extended attribute files, and sometimes with the CNID dbd. But stopping and starting netatalk and cnid_dbd allows us to live (survive :)) without cnid_dbd hangs for the following day.

Thank you very much! I think the re-build would take a while, there are around 400k files on the AFP shares.. just out of curiosity, how many are on your server?

Also, what effect does the netatalk restart have on your clients? If I go to the freenas UI and restart AFP, the shares need to be re-connected on clients, which is quite annoying. TM backup shares re-connect themselves but those mounted manually through Finder need to be mounted again (and finder shows message that the server has closed down).

giox069- so why not just get rid of AFP and use SMB on FreeNAS now? Then you wouldn't need to buy a license of server 2008 R2....

I think this would be the route I will go eventually. The reason I chose AFP over NFS for mac shares was that I could get higher transfer speed (with large files). I did not investigate in detail, this was simply by dragging large files to/from the share. AFP could saturate the NIC while NFS was 30-40% slower.

 

[cyberjock]

AFP could saturate the NIC while NFS was 30-40% slower.

Odd. I use CIFS and I can saturate Gb LAN. I also tested NFS and was able to do the same. I don't have any apple devices so I couldn't test AFP. It's quite odd that NFS is slower for you by that much.

 

[petr]

Odd. I use CIFS and I can saturate Gb LAN. I also tested NFS and was able to do the same. I don't have any apple devices so I couldn't test AFP. It's quite odd that NFS is slower for you by that much.

Maybe the Mac OS nfs implementation is somehow limited? Also, I am doing the mounting through finder's facilities, maybe more low-level would be better.

 

[giox069]

@cyberjock: we are considering many possible solutions here, included FreeBSD9 + netatalk 3 (we must compile it by hand).
NTFS on win 2008R2 will fit best on our infrastructure, since we already have licenses for both server and clients (CAL). And we already use powerful tool for backup and recovery: VEEAM for VMWare ESXi is far longer better than rsync scripts. We still haven't found the final solution yet. Useful things that we would like to have are:
- A filesystem and protocol which keeps extended attributes attached to the file, and not on extra files as netatalk2 does. So when you move a file via SMB, their xattrs (forks) are moved with it.
- Snapsots (ZFS) or windows VSS
- Possible spotlight search compatibility (netatalk 3 on freebsd ?)
- Backup tools
So we haven't yet decided where to go now, we are just testing various possible paths. In the meanwhile, the cnid_dbd killer/rebuilder script is helping us to survive with freenas.

@petr: 900K files on 2.8TB on the biggest afp share. Then there are other smaller shares (< 50GB).
Restarting netatalk disconnects all clients, and we have exactly the same side effect as stopping it from the UI: users get the annoying message and all clients must be manually reconnected. But we run the script during the night, when users PC are powered off.
Thank you for the info on NFS. Next week i hope we will have the time to investigate further on netatalk2 alternatives.

 

[petr]

@petr: 900K files on 2.8TB on the biggest afp share. Then there are other smaller shares (< 50GB).
Restarting netatalk disconnects all clients, and we have exactly the same side effect as stopping it from the UI: users get the annoying message and all clients must be manually reconnected. But we run the script during the night, when users PC are powered off.
Thank you for the info on NFS. Next week i hope we will have the time to investigate further on netatalk2 alternatives.

Thanks for your info - I guess I will just give up on the extra potential bit of speed - I do not need it day-to-day, I only chose AFP to start with. I guess I will go with CIFS/SMB testing now, to see how well it works for my purposes. I am storing a few files I then serve on my network on mac mini home server and this disconnect means that I need to go in and re-connect the share to regain access to my music (I guess I could have yet another script to do that..).

So in conclusion - I will probably migrate away from AFP.

On that note, are the .AppleDouble files internal to the netatalk service, or if I share the files ofer CIFS, are they going to be used by the Mac clients? I do not really need the extended attributes to be preserved.

 

[giox069]

Some extra info for @cyberjock (FreeNAS+CIFS vs Win2008R2): I did some small tests with OSX clients and Freenas8+CIFS vs Win2008R2.

1. In FreenNAS8+CIFS the MAC clients stores the files extended attributes in separate files (starting with "._"). In NTFS the whole file (data fork + EA) are stored in a single file. This is very important when you plan to share files to normal windows PC also: when you move a MAC file with extended attibutes using a windows PC, your EA will be lost with FreeNAS. So, in my opinion, Windows2008R2 behaves better, because a file is a single object seen by both PC and MAC.
   I also tried to enable "EA" on the CIFS configuration page, but newer files on freenNAS are still separated from the "._" files. Is this a SAMBA problem ? I don't know... any help or idea would be appreciated.
2. Windows2008R2 file indexing service seems to be compatible with finder search: user can type the file name they want on the finder, and the indexing service on 2008R2 will answer in few seconds. This is not possible with Freenas8, both via CIFS or AFP (netatalk2).

 

[giox069]

Some extra info for @cyberjock (FreeNAS+CIFS vs Win2008R2): I did some small tests with OSX clients and Freenas8+CIFS vs Win2008R2.

1. In FreenNAS8+CIFS the MAC clients stores the files extended attributes in separate files (starting with "._"). In NTFS the whole file (data fork + EA) are stored in a single file. This is very important when you plan to share files to normal windows PC also: when you move a MAC file with extended attibutes using a windows PC, your EA will be lost with FreeNAS. So, in my opinion, Windows2008R2 behaves better, because a file is a single object seen by both PC and MAC.
   I also tried to enable "EA" on the CIFS configuration page, but newer files on freenNAS are still separated from the "._" files. Is this a SAMBA problem ? I don't know... any help or idea would be appreciated.

This problems cannot be solved on Freenas8 on a clean way. We should be able to load "vfs objects = streams_xattr" in smb.conf, but a bug makes it quite complex to add extra vfs objects. And then other modifications to the samba config are needed.

 

[cyberjock]

I agree with giox069s assessment. Apple has some software that does really weird things in certain circumstances. FreeNAS8 has no "easy" way to clean up the "mess" from these kinds of applications. The issue is strictly related to Apple products(both hardware running MACOSX as well as iTunes in some situations). You wanted an OS that does what it wants without control, so you got it. :)

I don't have any apple devices so I can't really do any troubleshooting with FreeNAS9 to tell you if there is a new better way to solve your problem. But if anyone lives in IL, has this issue, and wants to let me try some things I'm definitely open to trying. I guess I could try a Teamviewer session if you are remotely located and okay with letting me try some things. Shouldn't be anything "dangerous" to existing data so the risk is minimal.