Good Morning..
We are attempting to migrating away from Windows Storage Spaces to FreeNAS as our Milestone XProtect video archive server but have ran into some performance issues that may or may not have to do with SAMBA or CIFS configuration or general interoperability between Microsoft Windows and FreeNAS. I'll do my best to make this short. Any assistance is appreciated.
We have two 45drive.com Storinator chassis, both identical in hardware. One is Windows 2012 R2 with Storage Spaces, the other is FreeNAS 9.10
FreeNAS: 9.10 Stable
Platform: Intel(R) Core(TM) i3-3240 CPU @ 3.40GHz
Memory: 32711MB ECC
Controllers: 2 x Rocket 750 sata controllers
Disks: 45 x 4TB WD RED drives - Nine raidz groups, containing 5 disks each for a total of 121TB storage.
Network: 2 x Intel X540 dual port 10gb nics / 40gb LACP lagg0
Reference links:
http://www.45drives.com/products/
https://www.milestonesys.com/our-products/video-management-software/xprotect-expert/
Required Workload:
Reads: 1600 to 1800 open files reads at any given time.
Writes: 1.5TB to 2.2TB data written to disk ever 24 hours
Video Files 30%: between 1MB and 16MB in size
Supporting files 70%: 1KB to 50KB in size
Current Files count: 10 million'ish
30 day churn rate: 100%
Growth expectations: 20% in the next 180 days.
I have installed most recent version of FreeNAS on one of our Storinators. Validated the system in a test environment for about 30 days but discovered very quickly after the device brought into production there was a very impactfull issue with the way the Milestone XProtect software and FreeNAS communicate. Milestone R&D came back with some very good information on how their software moves data during the archive process:
"The performance/behavior observed between Milestone and FreeNAS is related to the way in which Milestone internally handle threading/locking. A normally brief lock is taken on the source database bank during the table moving process, and it seems this lock takes several seconds due to slow IO on the FreeNAS shares resulting in images buffering to memory until the end of that lock."
I have sense moved the archive server back to our windows storage space server and collected various SMB captures with Wireshark between our Milestone recording servers, Windows Storage Spaces and FreeNAS to better understand the problem. One thing stands out above the rest. All SMB transmissions to the FreeNAS server have a max TCP windows size of 1024 and eventually drop to 500'ish across all data block transfers regardless of size. I am also observing large percentage of Zero Window size packets when communicating with FreeNAS due to the smaller scope of TCP windows.
It is possible that I am not looking at this from the right angle, please let me know if you would like additional information.
We are attempting to migrating away from Windows Storage Spaces to FreeNAS as our Milestone XProtect video archive server but have ran into some performance issues that may or may not have to do with SAMBA or CIFS configuration or general interoperability between Microsoft Windows and FreeNAS. I'll do my best to make this short. Any assistance is appreciated.
We have two 45drive.com Storinator chassis, both identical in hardware. One is Windows 2012 R2 with Storage Spaces, the other is FreeNAS 9.10
FreeNAS: 9.10 Stable
Platform: Intel(R) Core(TM) i3-3240 CPU @ 3.40GHz
Memory: 32711MB ECC
Controllers: 2 x Rocket 750 sata controllers
Disks: 45 x 4TB WD RED drives - Nine raidz groups, containing 5 disks each for a total of 121TB storage.
Network: 2 x Intel X540 dual port 10gb nics / 40gb LACP lagg0
Reference links:
http://www.45drives.com/products/
https://www.milestonesys.com/our-products/video-management-software/xprotect-expert/
Required Workload:
Reads: 1600 to 1800 open files reads at any given time.
Writes: 1.5TB to 2.2TB data written to disk ever 24 hours
Video Files 30%: between 1MB and 16MB in size
Supporting files 70%: 1KB to 50KB in size
Current Files count: 10 million'ish
30 day churn rate: 100%
Growth expectations: 20% in the next 180 days.
I have installed most recent version of FreeNAS on one of our Storinators. Validated the system in a test environment for about 30 days but discovered very quickly after the device brought into production there was a very impactfull issue with the way the Milestone XProtect software and FreeNAS communicate. Milestone R&D came back with some very good information on how their software moves data during the archive process:
"The performance/behavior observed between Milestone and FreeNAS is related to the way in which Milestone internally handle threading/locking. A normally brief lock is taken on the source database bank during the table moving process, and it seems this lock takes several seconds due to slow IO on the FreeNAS shares resulting in images buffering to memory until the end of that lock."
I have sense moved the archive server back to our windows storage space server and collected various SMB captures with Wireshark between our Milestone recording servers, Windows Storage Spaces and FreeNAS to better understand the problem. One thing stands out above the rest. All SMB transmissions to the FreeNAS server have a max TCP windows size of 1024 and eventually drop to 500'ish across all data block transfers regardless of size. I am also observing large percentage of Zero Window size packets when communicating with FreeNAS due to the smaller scope of TCP windows.
It is possible that I am not looking at this from the right angle, please let me know if you would like additional information.
Last edited: