Should be in your inbox.
-
Important Announcement for The TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Performance 9.3 vs 9.10 (folder browsing) + Linux
- Thread starter pernils
- Start date
- Status
Should be in your inbox.
- Joined
- Mar 6, 2014
- Messages
- 9,554
A few comments:
- You have "null passwords = yes" in your smb.conf. This parameter has been deprecated. It looks like your smb.conf file keeps trying to reload, which can have performance implications.
- Disable hostname lookups.
- There are entries in /var/log/messages complaining about permissions issues in your login.conf. Perhaps consider testing with a fresh FreeNAS install.
- It doesn't look like you have SMART tests configured (not relevant in this case, but thought it was worth pointing out).
Last edited by a moderator:
- Joined
- Mar 6, 2014
- Messages
- 9,554
I think you're getting overly fixed on a single test that you created.
A few quick questions:
- What CAD software are you using?
- Were the CAD files originally saved on other server that has since been taken offline?
- Does the CAD software use VSS?
- Have you tried other OSes on the hardware you have? (for instance temporarily loading a Linux Install with ZOL)?
- Have you tried killing your symlink / snapshots trickery and seeing if it improves performance?
- Some CAD software allows you to store different types of files in different directories. Perhaps you can look carefully at your CAD software settings and try to break your data up differently.
Last edited:
- CAD software used is Solid Edge
- 2.5 years ago they where moved from a windows server due out of space. I have been reediting the file links to point at the right server.
- No the cad software don't use VSS. Only us engineers want a easy way to revert changes.
- No I have not tested other OSes on the asrock mobo C2750D4I. I do some planning for it but lacks time to do it.
- My snapshot trickery .. Yes I have removed some of it. Here is also something. The more snapshot you have the worse it gets.
As we are constant using the file tree many times during the day this is becoming really annoying.So where do you go ? Just buy new hardware and hope you have luck?
I may be wrong but from what I understand ZFS is caching up alot of the metadata so this file list should more or less be served out from memory. So my impression was that when file count increased it will beat ntfs but sadly thats' not the case, not today at least.
Right now I'm reverting back to 9.3. The small test by browsing in and out into folders right now seem to be more snappy. It's midnight here so tomorrow I will tell how much they differ.
- Joined
- Mar 6, 2014
- Messages
- 9,554
You can look at your ARC composition.
arc_summary.py. If you're hitting a wall on metadata, you can try increasing the value of vfs.zfs.arc_meta_limit. It defaults to 1/4th the size of your pool. Try increasing to 1/2. It will take a little while for it to warm up. See if it helps the situation. If it does, you might be able to increase it a bit more, or perhaps consider adding a SSD L2ARC as a metadata-only cache device. This is covered under #3 here: https://forums.freenas.org/index.php?threads/cifs-directory-browsing-slow-try-this.27751/It is somewhat less expensive than trying to get a new system.
That said, you should look into testing ZoL on a test zpool with sa-based xattrs enabled. It would be useful to see how much performance difference there is under Linux.
This folder response in samba is indeed client depended. For example win10 seems to be a fraction faster then win7 (experience from one client only) . Okay .. not the same hardware and the the diff was only 0.4 sec for 50k files, but my private Ubuntu Mate install with Caja as file explorer will just hang forever while trying to open it.
My old acer notepad win7 is much slower then my virtualbox win7 install witch I'm going to use in this test.
Just saying if you can't fix the server you maybe can reduce it by upgrade the client.
My test server hardware:
Asrock C2750D41 (freenas mini)
32 GB mem
500 GB ZFS raid1
4 Tb ZFS raid1
60 Gb Intel SSD
Both ZFS raids was created from a earlier Freenas install.
My test client :
Win7 in Virtualbox on HP Z600
No antivirus is used.
File count is 50k.
For timing I will use my script and also enter with file browser and silent count until I see all then files. This will be repeated multiple times to average the result.Both result, script and manual will be printed.
First I installed Win2008R2 on the 60 Gb SSD.
No antivirus is used.
WIN2008 R2 (accessed on the server it self via GUI )
script 7.2 - 8.2 sec (the higher value was more common)
manual 1+ sec (weary odd )
WIN2008 R2 (accessed from VM client)
script 1 sec
manual 1+ sec
Installed FreeNAS on a USB and used the SSD for a zpool instead. Hereafter the virtualbox win7 client will be used for the measurements.
FreeNAS 11 (zpool on SSD, samba "go fast" parameter is used)
script 3.7 sec
manual 3.5 - 4 sec
Then I installed Proxmox 4.4 on the SSD for be able to use the ZoL in a comfortable manner. The Debian was installed as LXC container.
Testing with the FreeNAS "go fast" samba parameter didn't affect the outcome on Debian install.
Debian 8 EXT4 SSD
script 2.3 sec
manual 2.5+ sec
Debian 8 ZoL raid1 500GB
script 2.5 sec
manual 3 sec
Then I installed FreeNAS 11 as a VM under proxmox and tested both scenarios. Proxmox serving ZoL (raid1 500GB) as a block device and also PCI passthru of the raid1 4 TB zpool.
Special SMB parameter is used in both cases.
FreeNAS 11 in VM, block device from ZoL raid1
script 4.9 sec
manual 5 sec
FreeNAS 11 in VM, 4tb raid1 zpool pci passthru
script 5 sec
manual 5.5 sec
Summary:
FreeBSD + zfs + samba is less responsive then its counterparts period. Even if you boost it with SSD you will be outrun by ZoL on regular disk. One thing is odd that qemu passthru of disk is slower then have it served as ZoL block device. This could be that it's not exactly the same disks.
But having ZoL and then FreeNAS ZFS + samba just clocking in the same as a normal passthru FreeNAS ZFS + samba must indicate that there are some serious overhead with FreeBSD + samba.
While testing on other hardware I noticed that under windows you will get more fluctuating timings. Even if you have 2 folders with the same file count (50k) it can differ from 1 sec to 7 sec. Disable antivirus didn't affect that really bizarre behavior. So today I leaning to recommend Linux instead of win due you will get a more solid experience for end user if you are going to have large folders. I have no experience of newer win servers beyond 2008 so maybe this have address also in the windows world.
My findings today is that you will with ZoL cut the response time in half compare to its BSD brother. The overhead of virtualizing FreeNAS is small compared to using FreeNAS itself.
For actual throughput I found FreeNAS as good as any store.
My old acer notepad win7 is much slower then my virtualbox win7 install witch I'm going to use in this test.
Just saying if you can't fix the server you maybe can reduce it by upgrade the client.
My test server hardware:
Asrock C2750D41 (freenas mini)
32 GB mem
500 GB ZFS raid1
4 Tb ZFS raid1
60 Gb Intel SSD
Both ZFS raids was created from a earlier Freenas install.
My test client :
Win7 in Virtualbox on HP Z600
No antivirus is used.
File count is 50k.
For timing I will use my script and also enter with file browser and silent count until I see all then files. This will be repeated multiple times to average the result.Both result, script and manual will be printed.
First I installed Win2008R2 on the 60 Gb SSD.
No antivirus is used.
WIN2008 R2 (accessed on the server it self via GUI )
script 7.2 - 8.2 sec (the higher value was more common)
manual 1+ sec (weary odd )
WIN2008 R2 (accessed from VM client)
script 1 sec
manual 1+ sec
Installed FreeNAS on a USB and used the SSD for a zpool instead. Hereafter the virtualbox win7 client will be used for the measurements.
FreeNAS 11 (zpool on SSD, samba "go fast" parameter is used)
script 3.7 sec
manual 3.5 - 4 sec
Then I installed Proxmox 4.4 on the SSD for be able to use the ZoL in a comfortable manner. The Debian was installed as LXC container.
Testing with the FreeNAS "go fast" samba parameter didn't affect the outcome on Debian install.
Debian 8 EXT4 SSD
script 2.3 sec
manual 2.5+ sec
Debian 8 ZoL raid1 500GB
script 2.5 sec
manual 3 sec
Then I installed FreeNAS 11 as a VM under proxmox and tested both scenarios. Proxmox serving ZoL (raid1 500GB) as a block device and also PCI passthru of the raid1 4 TB zpool.
Special SMB parameter is used in both cases.
FreeNAS 11 in VM, block device from ZoL raid1
script 4.9 sec
manual 5 sec
FreeNAS 11 in VM, 4tb raid1 zpool pci passthru
script 5 sec
manual 5.5 sec
Summary:
FreeBSD + zfs + samba is less responsive then its counterparts period. Even if you boost it with SSD you will be outrun by ZoL on regular disk. One thing is odd that qemu passthru of disk is slower then have it served as ZoL block device. This could be that it's not exactly the same disks.
But having ZoL and then FreeNAS ZFS + samba just clocking in the same as a normal passthru FreeNAS ZFS + samba must indicate that there are some serious overhead with FreeBSD + samba.
While testing on other hardware I noticed that under windows you will get more fluctuating timings. Even if you have 2 folders with the same file count (50k) it can differ from 1 sec to 7 sec. Disable antivirus didn't affect that really bizarre behavior. So today I leaning to recommend Linux instead of win due you will get a more solid experience for end user if you are going to have large folders. I have no experience of newer win servers beyond 2008 so maybe this have address also in the windows world.
My findings today is that you will with ZoL cut the response time in half compare to its BSD brother. The overhead of virtualizing FreeNAS is small compared to using FreeNAS itself.
For actual throughput I found FreeNAS as good as any store.
Last edited by a moderator:
- Joined
- Mar 6, 2014
- Messages
- 9,554
I think you've become overly fixed on a single metric, which appears to be all over the board with lots of OSes. I don't see a strong correlation between what's happening here and what you describe from within Solid Edge, which appears has very long delays. Ultimately, you want to get your fileserver to perform well with Solid Edge. Since the delays seem to be much longer browsing within Solid Edge than browsing through a File Explorer session, it's fairly clear that Solid Edge is doing something (or trying to do something). You should try to figure out what that "something" is. It may be a simple configuration change inside the application. One diagnostic step will be to reproduce the "slow" behavior with samba's logging set to "debug". Post the log here. Maybe look for logs in Solid Edge as well.
Perhaps compare Solid Edge performance between operating systems. If it performs noticeably better on one OS than another (for instance), use that OS. :)
Using SE build in browser is doable up to around 15k files. Therefore we have been using windows explorer to click and drag for several years now. So you can take SE quirks out of the equation. My focus is only on windows explorer. When I enter a folder with explorer I prefer to have it up and ready as fast as it can be.
And Solid Edge is windows only so no luck by changing OS either.
And here we land what I have been trying to show over and over again. BSD + zfs + samba is not up for big folders. Okay fare enought I have only tested this on iX's preferred asrock board and some "none supported" motherboard. So one can argue that I have the wrong hardware and end of story.
Me on the other hand is confident that it's bsd to blame. If I don't remember wrong Hubbard was also mention this in one of his videos also. Nor especially folder browse but that samba folks was poking him on meetings about slow performance (can if needed dig it up).
Last edited:
I can just add that d_off field of struct dirent, which as I told before is required for proper directory listing by Samba, was recently added to FreeBSD 12. It seems the work is still in progress and that field is not really used yet, but that is a critical first step.
As a long time engineer I have come to find that specially one types of question is more irritating then others. Common for those irritating question is the starting keyword "When" or have it's weight on it.
After looking at historical bsd releases it seems that the major ones is happen in interval of around 1.5 - 2 years. From what I hear some functionality is sometimes backported to lower releases but in this case when it will as you mention break binary compatibility, I asume it will not happen.
So against one of my rule .. Do you have a estimate of when this could be seen in real life ?
(From self hard earned experience a good approximation is if one takes ones estimate and double it.)
(Ukraine .. hmm I guess there is media war going about the eurovision.. "Did it left any revenue ?")
Last edited:
Finally I have moved the files back to a MS server (win2008) and what can I say ... I can't help it ...a ntfs server is sooo much snappier in this area. It's like day and night. It will be interesting to see what BSD12 bring to table for this problem. Yes I think this is a serious problem for bsd+zfs+samba.
Enough with the bashing ... one thing that zfs beat the heck out of ms server is the snapshots. VSS is joke.
A fresh experience with ransom-ware is that for rollback zfs snapshot with 150k files is a 10 sec job but for 70+ files with VSS it's 3 hours copy.
Enough with the bashing ... one thing that zfs beat the heck out of ms server is the snapshots. VSS is joke.
A fresh experience with ransom-ware is that for rollback zfs snapshot with 150k files is a 10 sec job but for 70+ files with VSS it's 3 hours copy.
- Status
