Mounting and using APFS sparse bundles over SMB

[Arctic Moose]

My apologies if I have missed something, but I do believe I have performed an exhaustive search on the forums and have not found any information relating to my specific issue. If there is, I would appreciate a link to it. Thanks!

I have been building, configuring and testing a TrueNAS server (based on i7-2600, 8 GB RAM, a SSD system disk and four SATA data drives) over the last few months, and believe I have something reasonably stable now, although I have not quite finished burn-in testing. The configuration is about as vanilla as can be, with only S.M.A.R.T, SMB, ssh and UPS enabled.

I intend to use it primarily for serving media, temporary data storage and various types of backups. Perhaps I will move Pi-hole and Home Assistant to it in the future.

I have set up SMB using the default share parameters for testing. This has worked reasonably well, except for one remaining issue concerning APFS sparse bundles.

Server: TrueNAS 12.2-RELEASE-p3
Client: Intel macOS 11.2.2

I have tried creating the image using:

hdiutil create -size 3000g -type SPARSEBUNDLE -volname "u4volume" -fs APFS -imagekey sparse-band-size=262144 -verbose -encryption -stdinpass u4test.sparsebundle

I tried creating it directly on the share, but get the following error:

Formatting… tx_flush:955: rdisk4s2 xid 1 sync/barrier failed: 45 tx_mgr_free_tx:207: rdisk4s2 Trash unfinished pending tx, xid range = 0x1 - 0x1 newfs_apfs: unable to format /dev/disk4s2: Operation not supported Finishing… DIHLDiskImageCreate() returned 73 hdiutil: create: returning 73 hdiutil: create failed - RPC version wrong

If I create it locally and then copy it to the share I can mount it, but when copying to it I get the following error sooner or later:

The operation can’t be completed because it isn’t supported.

All test activities to the volume are successful when it is created and mounted locally.

The tests complete successfully fairly consistently (although not 100%) if I instead of APFS use HFS:

hdiutil create -size 3000g -type SPARSEBUNDLE -volname "u4volume" -fs HFS+J -imagekey sparse-band-size=262144 -verbose -encryption -stdinpass u4test.sparsebundle

(Sometimes the volume will unmount for no apparent reason, and I cannot figure out where to look in the logs for errors relating to this.)

I know I have messed around with my SMB settings on the client in the past for other issues, but I cannot remember what I have done. Here's my output from smbutil:

================================================================================================== SHARE ATTRIBUTE TYPE VALUE ================================================================================================== dada4 SERVER_NAME 10.0.0.20 USER_ID 502 SMB_NEGOTIATE SMBV_NEG_SMB1_ENABLED SMB_NEGOTIATE SMBV_NEG_SMB2_ENABLED SMB_NEGOTIATE SMBV_NEG_SMB3_ENABLED SMB_VERSION SMB_3.02 SMB_SHARE_TYPE DISK SIGNING_SUPPORTED TRUE EXTENDED_SECURITY_SUPPORTED TRUE UNIX_SUPPORT TRUE LARGE_FILE_SUPPORTED TRUE OS_X_SERVER TRUE FILE_IDS_SUPPORTED TRUE DFS_SUPPORTED TRUE FILE_LEASING_SUPPORTED TRUE MULTI_CREDIT_SUPPORTED TRUE ENCRYPTION_SUPPORTED TRUE --------------------------------------------------------------------------------------------------

My two major questions are:

1. Is there an obvious fix for APFS sparse bundles, assuming I am doing something wrong?
2. Where can I find error logs relating to SMB, mounting, volumes and disk images? (I guess primarily on the client, but also on the server if relevant.)

Thanks!

 

[anodos]

This is interesting. Can you send me a pcap of it through Private Message? Easiest way to generate one might be to run smbdiagnose in Terminal app, then reproduce issue, then kill the smbdiagnose with ^c (control+c).

 

[Arctic Moose]

This is interesting. Can you send me a pcap of it through Private Message? Easiest way to generate one might be to run smbdiagnose in Terminal app, then reproduce issue, then kill the smbdiagnose with ^c (control+c).

Thanks, I'll do that!

 

[anodos]

I see same error when trying to create the apfs sparsebundle on Windows. Have you successfully performed this operation against an SMB server?

 

[Arctic Moose]

I see same error when trying to create the apfs sparsebundle on Windows. Have you successfully performed this operation against an SMB server?

I created a fresh APFS sparse bundle (as per above) on a clean-installed macOS 11.2 laptop.

I connected to my normal client iMac (macOS 11.2.2, but updates have been installed on top of older installations since 2017, and there are lots of tweaks for different reasons) through SMB and uploaded a copy of the bundle to the iMac. I mounted the bundle from the laptop, and set it as the Time Machine destination.

sudo tmutil setdestination /Volumes/u4test2

The initial backup completed without any issues.

I then copied the very same (empty) APFS sparse bundle to TrueNAS. It mounts, and I can copy a varying amount of data to it, but after a while it always fails after some amount of time with the same error message as earlier:

The operation can’t be completed because it isn’t supported.

After rebooting the laptop, reconnecting to TrueNAS through SMB and remounting the sparse bundle, the files that were supposedly successfully copied are not there.

Of course, I tried the Time Machine experiment as well, which also failed.

Test results, for clarity:

• APFS sparse bundle hosted on macOS works as desired
• APFS sparse bundle hosted on TrueNAS does not work at all

 

[anodos]

I created a fresh APFS sparse bundle (as per above) on a clean-installed macOS 11.2 laptop.

I connected to my normal client iMac (macOS 11.2.2, but updates have been installed on top of older installations since 2017, and there are lots of tweaks for different reasons) through SMB and uploaded a copy of the bundle to the iMac. I mounted the bundle from the laptop, and set it as the Time Machine destination.

sudo tmutil setdestination /Volumes/u4test2

The initial backup completed without any issues.

I then copied the very same (empty) APFS sparse bundle to TrueNAS. It mounts, and I can copy a varying amount of data to it, but after a while it always fails after some amount of time with the same error message as earlier:

The operation can’t be completed because it isn’t supported.

After rebooting the laptop, reconnecting to TrueNAS through SMB and remounting the sparse bundle, the files that were supposedly successfully copied are not there.

Of course, I tried the Time Machine experiment as well, which also failed.

Test results, for clarity:

• APFS sparse bundle hosted on macOS works as desired
• APFS sparse bundle hosted on TrueNAS does not work at all

Right. The SMB protocol allows a client to potentially send OS-specific commands to the server. I'm trying to determine whether we have a bug or if it's simply a matter of a MacOS-specific IOCTL not being implemented. The error seems to imply the latter, but I'm not 100% sure. I will try to test myself, but may not be able to get to it over the next few days.

Does the same error occur if you try the operation on an SMB share on a Windows server?

 

[Arctic Moose]

Right. The SMB protocol allows a client to potentially send OS-specific commands to the server. I'm trying to determine whether we have a bug or if it's simply a matter of a MacOS-specific IOCTL not being implemented. The error seems to imply the latter, but I'm not 100% sure. I will try to test myself, but may not be able to get to it over the next few days.

Thanks! I am just grateful for any assistance I receive, regardless of how long it takes!

Does the same error occur if you try the operation on an SMB share on a Windows server?

I do not currently have any Windows machines set up, will see what I can do over the weekend.

 

[anodos]

Thanks! I am just grateful for any assistance I receive, regardless of how long it takes!



I do not currently have any Windows machines set up, will see what I can do over the weekend.

Okay. Setting up a basic VM is sufficient. Microsoft provides free 180 day trials of Windows Server https://www.microsoft.com/en-us/windows-server/trial
Deploying in a VM is dead simple. You can user server manager to add the file-sharing feature if it isn't added already. I find Windows Server a vastly more enjoyable user experience than Windows 10 (no in-your-face candy-crush stuff).

 

[Arctic Moose]

Does the same error occur if you try the operation on an SMB share on a Windows server?

I have now tried it on both Kali Linux and Windows 10.

In both cases, I get the same error when trying to copy to the APFS sparse bundle.

If I replace "APFS" in the command to create the sparse bundle with "HFS+J", everything works as expected.

So, it seems, this is not a TrueNAS issue, but perhaps an APFS issue or a non-Apple SMB issue. (I cannot recall if I ever got a chance to try it on the failed Drobo that led me down this path, but I do believe I got it to work.)

Regardless, it is a dealbreaker for what I hoped would be a TrueNAS future for me, as I need the APFS snapshot Time Machine features available on the network. If I cannot get it to work I'll need to get a Mac mini and a box of disks instead, which would mean a waste of perfectly fine hardware.

Now that I know it is not TrueNAS-specific, maybe I can tweak my searches and hopefully find a solution.

Of course, any further tips or suggestions of workarounds I could try would be welcome! I would much prefer getting this working on TrueNAS.

 

[anodos]

I have now tried it on both Kali Linux and Windows 10.

In both cases, I get the same error when trying to copy to the APFS sparse bundle.

If I replace "APFS" in the command to create the sparse bundle with "HFS+J", everything works as expected.

So, it seems, this is not a TrueNAS issue, but perhaps an APFS issue or a non-Apple SMB issue. (I cannot recall if I ever got a chance to try it on the failed Drobo that led me down this path, but I do believe I got it to work.)

Regardless, it is a dealbreaker for what I hoped would be a TrueNAS future for me, as I need the APFS snapshot Time Machine features available on the network. If I cannot get it to work I'll need to get a Mac mini and a box of disks instead, which would mean a waste of perfectly fine hardware.

Now that I know it is not TrueNAS-specific, maybe I can tweak my searches and hopefully find a solution.

Of course, any further tips or suggestions of workarounds I could try would be welcome! I would much prefer getting this working on TrueNAS.

We have a feature if you enable "Enhanced time machine" preset for SMB share where a snapshot will be taken for the user's dataset (unique datasets will be created based on username used to authenticate to the share) on successful SMB2 Tree Disconnect. This will generally equate to a successful time machine backup.

 

[Arctic Moose]

We have a feature if you enable "Enhanced time machine" preset for SMB share where a snapshot will be taken for the user's dataset (unique datasets will be created based on username used to authenticate to the share) on successful SMB2 Tree Disconnect. This will generally equate to a successful time machine backup.

I am not quite sure I fully understand, but as far as I can tell this would mean server-end snapshots of Time Machine HFS+J sparse bundles, correct?

The primary reason I want APFS instead of HFS+ sparse bundles (future-proofing aside) is that backups to APFS are so much faster, the time required for the "preparing Time Machine backup" phase goes from excruciating to practically instantaneous.

(And now, of course, I am curious about what is going on, and want to solve this just for the sake of solving it.)

 

[anodos]

tx_flush:955: rdisk4s2 xid 1 sync/barrier failed: 45

Didn't catch this earlier. It might be failing because of lack of support for F_BARRIERFSYNC. If you can get me a packet capture of same on MacOS SMB server (maybe running smbdiagnose on the MacOS server when copying empty aapl-based sparsebundle volume to the server, then opening and writing a file to it -- keep file small), I'll take a look when I have time and see about expanding our advertised flags to include F_BARRIERFSYNC (for testing purposes).

 

[Arctic Moose]

If you can get me a packet capture of same on MacOS SMB server (maybe running smbdiagnose on the MacOS server when copying empty aapl-based sparsebundle volume to the server, then opening and writing a file to it -- keep file small), I'll take a look when I have time and see about expanding our advertised flags to include F_BARRIERFSYNC (for testing purposes).

Thanks!

I started smbdiagnose on both the client and the server, mounted the share on the client and created an APFS sparse bundle directly on the share:

hdiutil create -size 1g -type SPARSEBUNDLE -volname "smbtest" -fs APFS -imagekey sparse-band-size=262144 -verbose -encryption -stdinpass /Volumes/smbt/smbt.sparsebundle

This was successful.

Enter disk image passphrase: Initializing… Creating… DIDiskImageCreatorProbe: interface 1, score 1000, CSparseBundleDiskImage DIDiskImageCreatorProbe: interface 2, score -1000, CSparseDiskImage DIDiskImageCreatorProbe: interface 3, score -1000, CRawDiskImage DIDiskImageCreatorProbe: interface 4, score -1000, CWOUDIFDiskImage DIDiskImageCreateWithCFURL: CSparseBundleDiskImage CBSDBackingStore::createProbe directory, not a valid image file. DIBackingStoreCreatorProbe: interface 0, score -1000, CBSDBackingStore DIBackingStoreCreatorProbe: interface 1, score 1000, CBundleBackingStore DIBackingStoreCreatorProbe: interface 2, score 0, CRAMBackingStore DIBackingStoreCreatorProbe: interface 4, score -100, CCURLBackingStore DIBackingStoreCreateWithCFURL: CBundleBackingStore tesDIFileEncodingCreatorProbe: interface 0, score 1000, CEncryptedEncoding DIFileEncodingCreateWithCFURL: CEncryptedEncoding DIFileEncodingCreatorProbe: interface 0, score -1000, CEncryptedEncoding DIBackingStoreCreatorProbe: interface 0, score 100, CBSDBackingStore DIBackingStoreCreatorProbe: interface 1, score -1000, CBundleBackingStore DIBackingStoreCreatorProbe: interface 2, score 0, CRAMBackingStore DIBackingStoreCreatorProbe: interface 4, score -100, CCURLBackingStore DIBackingStoreCreateWithCFURL: CBSDBackingStore DIBackingStoreCreateWithCFURL: creator returned 0 DIFileEncodingCreateWithCFURL: creator returned 0 DIBackingStoreCreateWithCFURL: creator returned 0 DIDiskImageCreateWithCFURL: creator returned 0 DI_kextWaitQuiet: about to call IOServiceWaitQuiet... DI_kextWaitQuiet: IOServiceWaitQuiet took 0.000003 seconds Formatting… Finishing… created: /Volumes/smbt/smbt.sparsebundle hdiutil: create: returning 0

I then mounted the sparse bundle from the share, copied a small text file to it, and then unmounted the sparse bundle and then the share.

I'll pm you the location of the pcap files from both the server and the client.

 

[Arctic Moose]

Short update for anyone with this issue who finds this thread.

After updating to TrueNAS-12.0-U6.1 and getting a new M1 Mac with macOS Monterey 12.0.1 I now have everything working the way I want.

I can create a new 8 TB sparse bundle locally:

Code:

hdiutil create -size 8000g -type SPARSEBUNDLE -volname "TM" -fs APFS -imagekey sparse-band-size=262144 -verbose -encryption -stdinpass tm.sparsebundle

I believe these args, especially band size, remain from years ago when I wanted to use home-rolled sparse bundles to add TM quotas to a Time Capsule. I am not sure if they are still required or relevant.

For each user, I copied this to a TrueNAS SMB share. (This is a standard "No presets" share with the "Use as Home Share" option checked. Each TM user has an account, and mounts their own home folder.)

I believe I tried creating the sparse bundle directly on the share, which, if I remember correctly, did not work.

After mounting the share and the sparse bundle I can set it as the TM target on each Mac:

Code:

sudo tmutil setdestination /Volumes/TM

I have successfully backed up over 6 TB to this sparse bundle daily for a week, and it all appears to work as expected.

I don't know if the new Mac, macOS 12 or the update to U6.1, or some combination thereof, made the difference. I am just happy to finally be up and running. I'll report back when more Macs have completed their TM backups.

 

[flotpg]

Same here with Samba on Debian + ZFS.
The workaround with creating the sparse bundle image locally works fine for me. Thanks!