AFP transfer hangs (freezes) when transferring .app and .pkg files containing alias (macOS) file

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
OS Version:
TrueNAS-12.0-U8.1

Model:
TRUENAS-MINI-3.0-XL+

Memory:
32 GiB​

8 bays filled with 4TB drives
Single ZFS pool in RAIDZ2

I just recently set up my first proper TrueNAS system (having dabbled with FreeNAS for a few years). I've been attempting to copy my data onto the new TrueNAS system but I keep running into the same problem. It seems to only happen with AFP. SMB seems to transfer without any hiccup but doesn't copy file metadata correctly (such as creation date) which is why I'm using AFP instead.

When transferring data from a Mac, either using Finder or even a copy command in the terminal, the transfer will hang with no network traffic on specific files. I have narrowed down the issue to occur specifically when the transfer attempts to copy an alias file contained within an application (.app) or package (.pkg). I have to disable the AFP service to stop the transfer once it hangs because attempting to cancel in Finder does nothing. If I don't stop the AFP service, it will hang for hours and give no error at all. Finder will freeze and not respond, even when forced to quit (necessitating a full reboot of the Mac). Stopping the AFP service causes the connection to fail, and the volume to be disconnected and the transfer errors out with the macOS error "The operation can't be completed because an unexpected error occurred (error code 100057)." Once the connection has been interrupted I have to use the TrueNAS shell to delete whatever it was that failed to transfer, otherwise I can't transfer anything when I reconnect. And even if it's one file that makes a whole transfer hang, I have to delete the entire directory the offending file was nested in.

I seem to be able to create alias files and transfer them without issue, but for some reason .app and .pkg files containing at least one alias file will cause this problem. I can also package up an application into a zip or image (dmg) and transfer it with no trouble. But that's a big pain I'd rather not do just to archive my files.

Is this an known issue? Has anyone else run into this problem? Is there a solution?
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Is this an known issue? Has anyone else run into this problem? Is there a solution?
This is known issue in 12.0-U8.1. Solution is to upgrade to 13.0-U1 (we should have all of edge cases for regression from upstream security release covered there).
 
Joined
Jun 2, 2019
Messages
591
@EMP-Tea

Welcome.

You do know that Apple depreciated AFP in favor of SMB? You should be using multi protocol SMB shares.
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
This is known issue in 12.0-U8.1. Solution is to upgrade to 13.0-U1 (we should have all of edge cases for regression from upstream security release covered there).
Thank you. I've been searching for two days to find an answer. I will look into upgrading to 13.0-U1.

Welcome.

You do know that Apple depreciated AFP in favor of SMB? You should be using multi protocol SMB shares.
Thanks for the welcome. I do know that Apple deprecated AFP (much to my chagrin) but I've found AFP still works faster than SMB on my network, maintains the metadata and tags, and is generally more compatible with my legacy Macs (the newest being a 2012 Mac Pro). If it ain't broke, don't fix it, as the saying goes.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Thank you. I've been searching for two days to find an answer. I will look into upgrading to 13.0-U1.


Thanks for the welcome. I do know that Apple deprecated AFP (much to my chagrin) but I've found AFP still works faster than SMB on my network, maintains the metadata and tags, and is generally more compatible with my legacy Macs (the newest being a 2012 Mac Pro). If it ain't broke, don't fix it, as the saying goes.
AFP will be available for the rest of the TrueNAS 13 lifecycle.

It has already been removed from SCALE. There are two reasons for this: (1) Apple is deprecating AFP and (2) the Netatalk project itself appears to be fading away - this can happen with opensource projects.

If you encounter bugs for SMB clients on MacOS (in case of pure SMB share, not data written by AFP and attempting to be shared via SMB), please file jira tickets. This is the primary way that we fix issues (users report them through formal channels).
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
Okay, I upgraded to 13.0-U1 and the AFP seemed to work without hangs. And everything appeared to be intact, metadata and all. Although old alias files have been shrunk from a couple MB to a couple KB which is odd. They still work, though. I think It might be something to do with macOS Mojave because transferring via macOS El Capitan doesn't seem to have that issue.

However I ran into more problems. When transferring a few directories, macOS randomly rebooted I assume from a kernel panic. And when attempting to delete directories, random files would be "in use" preventing anything form being deleted. I could not identify what was "using" the random files it would complain about. I could, however, delete the file "in use", but I could not delete the directory it was in. And if it got fixated on one file, and I delete that file, then attempted to delete the directory, it would choose another random file that was "in use" and not allow deletion of the directory.

I did try using the SMB service again (on 13.0-U1), but it did the same thing as OS version 12.0-U8.1 and stripped a bunch of data off files and folders like ICNS data. It also randomly added or subtracted data from somewhere in multiple cases; the directory size was never exactly the same (sometimes larger, sometimes smaller). And we're talking significantly smaller or larger, sometimes by 10%. The permissions were also messed up no matter what I did. Had a similar problem deleting directories over SMB as with AFP. MacOS kept complaining that random files were "in use" but nothing I could identify was.

Sad to see AFP go when it works better than SMB. AFP is sometimes twice as fast as SMB. Of course they both seem to have an unreliability issue with TrueNAS.

I reverted back to 12.0-U8.1 and tried a different approach. I was reading this thread about using a sparse disk image mounted over smb and that got me thinking. I went ahead and tried a test and so far the sparse disk image is incredibly stable. It maintains all metadata, doesn't add or subtract data, and I haven't yet run into an issue with deleting directories.

However it has several caveats: It's incredibly slow (about 1/10th the speed of AFP) and you can't mount the same sparse disk image to multiple machines... not to mention I don't even know if Windows can even read a sparse disk image (I assume not). It's also not future-proof because Apple might decide to drop that standard too and then I would be really screwed.

Should I try using an older version of TrueNAS or even FreeNAS? Was there a version where AFP was much more stable/reliable? I supposed I could try reporting these SMB bugs for TrueNAS 13 and hope they get fixed within the year, but I need to migrate my data ASAP (currently running a Synology NAS in a degraded state).
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
I did try using the SMB service again (on 13.0-U1), but it did the same thing as OS version 12.0-U8.1 and stripped a bunch of data off files and folders like ICNS data.

There is a hotfix release upcoming for some SMB permissions issues. Are you transferring data over SMB and having issues or is this data that was already written via AFP and you switched protocols? In this case planning for switchover requires special SMB configuration otherwise MacOS metadata can be stripped (because netatalk and samba write metadata in different ways).
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Kernel panics on a MacOS client indicate a MacOS issue. If you can reproduce the AFP issues on 13.0-U1 then I will look at the debug (especially if a corefile was generated).
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
Are you transferring data over SMB and having issues or is this data that was already written via AFP and you switched protocols? In this case planning for switchover requires special SMB configuration otherwise MacOS metadata can be stripped (because netatalk and samba write metadata in different ways).
Yes, the data I'm transferring was previously written via AFP and I'm connected via AFP to the old NAS and SMB to the new NAS. I will try connecting via SMB for both and see what happens.

Kernel panics on a MacOS client indicate a MacOS issue. If you can reproduce the AFP issues on 13.0-U1 then I will look at the debug (especially if a corefile was generated).
It could have just been a fluke with Mojave, but I will try and reproduce it if I can.
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
So for both 12.0-U8.1 and 13.0-U1 transferring via SMB only on both servers resulted in some stripped metadata. The same goes for data transferred from a Mac via SMB to TrueNAS (i.e. not from NAS to NAS). I get the same results with Mojave and El Capitan (which are the only two I have to test).

I also discovered the "file in use" issue when deleting a directory with lots of subfolders and files only happens with Mojave over SMB. El Capitan has no trouble deleting over SMB. For reference, I read this thread on the issue so it seems to be a known problem. No idea of it's a MacOS issue or not.

Doing a stress test tonight on 13.0-U1 via AFP to see if I can get a MacOS kernel panic again (using Mojave).
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
What exact metadata is stripped?
As far as I can tell the only metadata being stripped on 13.0-U1 via SMB are ICNS image data on folders. Created dates appear intact. Alias files appear intact. The weird thing is, the ICNS data is consistently being stripped from the same folders. Only about 10% seem to make it through the transfer intact, but it is consistent and repeatable.

Top is the destination. Bottom is the source.
Steps taken: transferred the folder "ICNS folders" from the desktop to the SMB share via the SMB connection.

folders.jpg


About 90% of the folders with ICNS data applied to them end up like "Capsule AirPort" with the data stripped when sent via SMB. But it's consistently the same folders so there must be something different about them that's confusing SMB.


Can you zip up a sample file for me to use to investigate this?
Yes, I have prepared a zip file with the two folders seen above. See attached file. I only included two folders to save on file size but I have more examples if you need them.
 

Attachments

  • ICNS folders.zip
    3.6 MB · Views: 86

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
As far as I can tell the only metadata being stripped on 13.0-U1 via SMB are ICNS image data on folders. Created dates appear intact. Alias files appear intact. The weird thing is, the ICNS data is consistently being stripped from the same folders. Only about 10% seem to make it through the transfer intact, but it is consistent and repeatable.

Top is the destination. Bottom is the source.
Steps taken: transferred the folder "ICNS folders" from the desktop to the SMB share via the SMB connection.

View attachment 57122

About 90% of the folders with ICNS data applied to them end up like "Capsule AirPort" with the data stripped when sent via SMB. But it's consistently the same folders so there must be something different about them that's confusing SMB.



Yes, I have prepared a zip file with the two folders seen above. See attached file. I only included two folders to save on file size but I have more examples if you need them.
Oh. That's fun:

Code:
[2022/07/26 06:55:39.996948,  0] ../../source3/modules/vfs_streams_xattr.c:980(streams_xattr_pwrite)
  streams_xattr_pwrite: Write to xattr [user.DosStream.AFP_Resource:$DATA] on file [ICNS folders/Capsule AirPort/Icon] exceeds maximum supported extended attribute size. Depending on filesystem type and operating system (OS) specifics, this value may be increased using the value of the parameter: smbd max xattr size = <bytes>. Consult OS and filesystem manpages prior to increasing this limit.


You actually hit a parameter that I added to samba a while back :)

In TrueNAS we write resource forks as extended attributes. The APIs for xattrs are rather unlike forks in some ways. For instance, there is no ability to pwrite() to an xattr. It's all or nothing, which means you need to allocate a buffer large enough to fit the entire fork... which means you need to set an upper limit otherwise you have a potential avenue to DOS a server or cause excessively large memory allocations. Current max is 2 MiB, but you can set an auxiliary parameter on Core / Enterprise to bump this up. Unfortunately, for SCALE this is a compile-time constant for the kernel. This means it's not so configurable. We have some flexibility in how large we make this. Will require some consideration.

Unfortunately, Finder has quite bad error handling in this area and will silently truncate resource forks and not present the error from the SMB server to the user.
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Code:
root@homenas[/mnt/dozer/DEV/linux]# getextattr -qq user 'DosStream.AFP_Resource:$DATA' /mnt/dozer/SCHOOL/ICNS\ folders/Capsule\ AirPort/Icon$'\357\200\215' | wc -c
 2143133

Looks like you lost the blackjack game on this. Our upper bound for xattr size is 2097152 bytes. The resource fork is 2143133. That said, this is a rather large fork to have on a file. How did you generate it?
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
It would be helpful if you can provide any information about what generated the files with the abnormally large resource forks. Setting a reasonable upper-bound on this has been a problem in the past.
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
You actually hit a parameter that I added to samba a while back :)

In TrueNAS we write resource forks as extended attributes. The APIs for xattrs are rather unlike forks in some ways. For instance, there is no ability to pwrite() to an xattr. It's all or nothing, which means you need to allocate a buffer large enough to fit the entire fork... which means you need to set an upper limit otherwise you have a potential avenue to DOS a server or cause excessively large memory allocations. Current max is 2 MiB, but you can set an auxiliary parameter on Core / Enterprise to bump this up. Unfortunately, for SCALE this is a compile-time constant for the kernel. This means it's not so configurable. We have some flexibility in how large we make this. Will require some consideration.

Unfortunately, MacOS / Finder has quite bad error handling in this area and will silently truncate resource forks and not present the error from the SMB server to the user.
Fascinating. Glad I'm talking to someone who understands all this. :D

FYI, try setting smbd max xattr size = 4194304 as an auxiliary parameter on the SMB share.
You'll have to forgive my ignorance. I assume this is where the parameter is entered: Services > SMB > Configure > Advanced Options
auxpara.jpg

I didn't have any results from just restarting the service. However restarting TrueNAS did get a result. All ICNS data seems to have been transferred intact! And It's consistently working with multiple attempts. At first I thought there was a syntax I was missing in the auxiliary parameters, but a reboot seems to have done the trick.

It would be helpful if you can provide any information about what generated the files with the abnormally large resource forks. Setting a reasonable upper-bound on this has been a problem in the past.
Unfortunately I don't know who made these or how they applied the ICNS data to the folders. It's possible they used an app to apply the data. My method for applying ICNS data has always been to copy/paste through the Get Info panel. I did find if I copy/pasted the ICNS data to a fresh folder then the directory copies just fine and the ICNS data stays intact. So maybe the folder is corrupted somehow? Maybe it was an older version of MacOS that somehow made it different enough to be a problem?

In any case, that auxiliary parameter seems to have done the trick. I will do some more testing and report back if the issue crops up again. Thanks for the help!

One more question: I'm running into an error when deleting folders with large directory trees. It says:
in_use.jpg

If I attempt again, it gives the same error. If I find that "in use" file or folder, and delete it, on a second attempt it sometimes allows me to delete the parent directory I was trying to delete or it gives the same error but with a different file or folder. There appears to be a repeating pattern, though. It's the same folders causing the issue. I saw a thread on the topic but I don't see a conclusion. I tried applying another ACL group entry but that doesn't seem to solve the issue. The permissions look identical for both the parent and child directories. Sometimes reapplying the ACL recursively allows for the deletion, but not with the first attempt.

Is this related to the incoming permissions hotfix for SMB?
 

EMP-Tea

Dabbler
Joined
Jul 20, 2022
Messages
13
Okay, update to this situation.

For some reason putting the auxiliary parameter into the SMB service configuration only worked for one particular share after reboot. The other shares did not take on the new upper bound for xattr size. No idea why.

However, after re-reading your post I realized I put the parameter in the wrong place. So I added the parameter to an actual SMB share instead of the SMB service config. The result was more of the metadata was transferred than before but not all. So I increase the xattr size again and this time it worked for all metadata.

And after applying smbd max xattr size = 8388608 to each share individually, this appears to have worked. Fingers crossed that it will keep working. I will post if anything changes.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Okay, update to this situation.

For some reason putting the auxiliary parameter into the SMB service configuration only worked for one particular share after reboot. The other shares did not take on the new upper bound for xattr size. No idea why.

However, after re-reading your post I realized I put the parameter in the wrong place. So I added the parameter to an actual SMB share instead of the SMB service config. The result was more of the metadata was transferred than before but not all. So I increase the xattr size again and this time it worked for all metadata.

And after applying smbd max xattr size = 8388608 to each share individually, this appears to have worked. Fingers crossed that it will keep working. I will post if anything changes.
Our default takes precedence over the global parameter. You have to do that per-share.
 
Top