Copying files with Alternative Data Stream : Avi Files.

[ludamus]

Hi all,

I have a very strange issue that I have been working on for weeks to the point that its driving me crazy , after scouring the internet and searching through numerous forums including this one I have yet to find someone who had experienced the same issue.

Problem: The issue first exhibited itself when I was attempting to mirror my local photos and video folder (from a windows 7 machine) to a newly created share (a mapped drive using matched credentials on both freenas and my local machine) using a program called beyond compare, certain files had failed to copy over. Of the 100's of avi files only a handful had failed .

only when I manually dragged and dropped the file to the share did I then notice odd behaviour, the file would copy over and I would see the file on the remote folder however at the end of the copy it would delete and copy over again and then at the end of that copy would say that "you'll need to provide administrator permission top copy this folder", odd.

initial thoughts that it looked like a permissions issue, however after comparing to other similar files (and larger) all attributes were the same, i had no problems with the other 900Gb of data that copied over fine.

To see if this was freenas related I tried copying to other PCs around the house, another windows 7 pc , which copied fine , I then tried to copy this to another samba share running on Ubuntu when it gave me an error saying "Are you sure you want to copy this file without its properties?, the file has properties that cant be copied to the new location". I clicked yes and then it copied and also played fine. I then copied the file back from my ubuntu share to my main PC and then copied the file to the freenas share , guess what ? it copied over without any issues!!

searching around on embedded proprieties led me down a line of enquiry looking at issues with files with ADS (alternative data streams) , using a tool called "LADS" I compared the file that I copied to Ubuntu (now renamed file2) and the the file with the issue (File1), LADS showed that the file did have an an alternative data stream.

So there is something strange the way the latest iteration of samba handles these kind of files especially ones that are specific to an NTFS file system :(

I hope someone can help me with this really strange issue before I go insane...Please help !!


LADS - Freeware version 4.10
(C) Copyright 1998-2007 Frank Heyne Software (http://www.heysoft.de)
This program lists files with alternate data streams (ADS)
Use LADS on your own risk!

Scanning directory C:\Users\user1\Desktop\Test folder\

size ADS in file
---------- ---------------------------------
165656 C:\Users\user1\Desktop\Test folder\File 1.avi:TOC.WMV


//START OF COPY

[2014/10/17 21:17:01.665781, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=Yes write=Yes (numopen=4)
[2014/10/17 21:17:02.487748, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=5)
[2014/10/17 21:17:02.488372, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=4) NT_STATUS_OK
[2014/10/17 21:17:02.631438, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=5)
[2014/10/17 21:17:18.463705, 2] ../source3/smbd/server.c:419(remove_child_pid)
Could not find child 8346 -- ignoring
[2014/10/17 21:17:31.547153, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=4) NT_STATUS_OK
[2014/10/17 21:17:31.549456, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=5)
[2014/10/17 21:17:31.549701, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi:TOC.WMV read=No write=Yes (numopen=6)
[2014/10/17 21:17:31.552541, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 1149780789, file FN-testfolder/File 1.avi:TOC.WMV, length=65536 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:17:31.553085, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 1149780789, file FN-testfolder/File 1.avi:TOC.WMV, length=65536 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:17:31.553361, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 1149780789, file FN-testfolder/File 1.avi:TOC.WMV, length=34584 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:17:31.554009, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi:TOC.WMV (numopen=5) NT_STATUS_OK
[2014/10/17 21:17:31.554083, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=4) NT_STATUS_OK
[2014/10/17 21:17:31.580414, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=3) NT_STATUS_OK
[2014/10/17 21:17:31.581128, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=4)
[2014/10/17 21:17:31.582394, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=3) NT_STATUS_OK
[2014/10/17 21:17:31.586238, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=Yes write=Yes (numopen=5)
[2014/10/17 21:17:32.427761, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=6)
[2014/10/17 21:17:32.428426, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=5) NT_STATUS_OK
[2014/10/17 21:17:32.456111, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=6)


/[2014/10/17 21:17:35.005998, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=7)
[2014/10/17 21:17:35.006567, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=6) NT_STATUS_OK
/[2014/10/17 21:17:35.140927, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=7)
[2014/10/17 21:17:35.159940, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=6) NT_STATUS_OK
[2014/10/17 21:17:35.160408, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=5) NT_STATUS_OK
[2014/10/17 21:18:01.380319, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=6)
[2014/10/17 21:18:01.380749, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi:TOC.WMV read=No write=Yes (numopen=7)
[2014/10/17 21:18:01.383518, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 428049813, file FN-testfolder/File 1.avi:TOC.WMV, length=65536 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:18:01.384066, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 428049813, file FN-testfolder/File 1.avi:TOC.WMV, length=65536 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:18:01.384348, 2] ../source3/smbd/smb2_write.c:187(smb2_write_complete_internal)
smb2_write failed: fnum 428049813, file FN-testfolder/File 1.avi:TOC.WMV, length=34584 offset=0 nwritten=-1: NT_STATUS_ACCESS_DENIED
[2014/10/17 21:18:18.473703, 2] ../source3/smbd/server.c:419(remove_child_pid)
Could not find child 8388 -- ignoring
[2014/10/17 21:18:48.420897, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi:TOC.WMV (numopen=6) NT_STATUS_OK
[2014/10/17 21:18:48.421044, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=5) NT_STATUS_OK
[2014/10/17 21:18:48.447579, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=4) NT_STATUS_OK
[2014/10/17 21:18:48.448211, 2] ../source3/smbd/open.c:975(open_file)
user1 opened file FN-testfolder/File 1.avi read=No write=No (numopen=5)
[2014/10/17 21:18:48.449166, 2] ../source3/smbd/close.c:780(close_normal_file)
user1 closed file FN-testfolder/File 1.avi (numopen=4) NT_STATUS_OK



ls -l
drwxrwxr-x+ 2 1001 familygroup 2 Oct 17 21:13 Testfolder/


[root@freenas] /mnt/homenas_zfs_volume/Backup# getfacl FN-testfolder/
# file: FN-testfolder/
# owner: user1
# group: familygroup
owner@:rwxpDdaARWcCos:fd----:allow
group@:rwxpDdaARWcCos:fd----:allow
everyone@:r-x---a-R-c---:fd----:allow

 

[anodos]

NTFS alternate data streams were a feature to improve compatibility with Mac filesystems. This functionality is replicated in samba via the streams_xattr vfs module, which is enabled by default in freenas.

 

[anodos]

To clarify: streams_xattr stores ads in an xattr. Filesystems tend to limit the size of xattrs and it appears this does not always fail gracefully. You can disable the vfs module by adding the following auxiliary parameter to your share:
vfs objects = shadow_copy2 zfsacl

 

[philiplu]

To clarify a bit more - samba supports NTFS alternate data streams, but the length of those is limited to 64K (cumulative over all streams for a specific file, I think). I hit a couple instances of too-large alt streams when I first set up my NAS and tried using the built-in Win 7 backup. Those were old files produced by, IIRC, a movie-maker app. The fix was to get rid of the streams on just those couple files. I did that using the SysInternals streams.exe tool, from www.sysinternals.com. Normally, it should be fine to get rid of the alternate streams, since they shouldn't be relied upon. Because they're supported by NTFS but not FAT32 or ExFAT, you'll easily lose them when copying to a default-formatted USB stick. So most apps that use them don't rely on them for correctness, just for optimization of some kind.

To see if this was freenas related I tried copying to other PCs around the house, another windows 7 pc , which copied fine , I then tried to copy this to another samba share running on Ubuntu when it gave me an error saying "Are you sure you want to copy this file without its properties?, the file has properties that cant be copied to the new location". I clicked yes and then it copied and also played fine. I then copied the file back from my ubuntu share to my main PC and then copied the file to the freenas share , guess what ? it copied over without any issues!!

When you clicked yes to the error dialog, you were telling the OS to go ahead and strip off the alternate data streams. So when you copied it back to your main PC and overwrote the original, you got a copy without those streams, and the file could be copied to FreeNAS without a problem.

 

[ludamus]

Philiplu, Using the streams tool (like LADS) showed the alternative stream, and used it to remove, It copied over fine as expected.
Come to think of it those files I had issues with were files that were probably converted using movie maker a looooong time ago when when transferring from the old camcorder via fire-wire.

C:\Users\user1\Desktop\TEST FOLDer>streams.exe "File 1 - Copy.avi"

Streams v1.56 - Enumerate alternate NTFS data streams
Copyright (C) 1999-2007 Mark Russinovich
Sysinternals - www.sysinternals.com

C:\Users\potts\Desktop\TEST FOLDer\File 1 - Copy.avi:
:TOC.WMV:$DATA 165656 <--- Larger than 64K


Anodos, tried vfs objects = shadow_copy2 zfsacl under auxiliary but didn't help :(

In all removing the ADS on problem files is the quickest fix but its not the solution , ideally I just want to copy any file without worrying if samba will support it.

The odd thing is that I uploaded the file via FTP , copied it over to the share internally back to my computer and it worked ?

Thanks for all your help so far.

 

[PWinger]

Wow...I've been having this same problem for 3 weeks and it was driving me crazy as well.

Many thanks to ludamus for documenting the problem (and solution), and anados & philiplu for clarifying situation with FreeNAS.

My situation involved only certain .ts files that were captured with a Hauppauge HDPVR2. Running LADS indicated >64k of Timing Info in the ADS, but running QuickStream Fix on the file seems to remove the ADS while repairing the video/audio streams. Most of my time was spent looking at possible file corruption in the capture process or incorrrect SAMBA configuration in FreeNAS.

It would be nice if FreeNAS removed the 64k limit, and/or MS acknowledged the ADS in the Explorer interface & copy process. But hey, my problem has been identified and resolved....for that I'm extremely grateful.