Sharethevibe
Dabbler
- Joined
- Aug 21, 2019
- Messages
- 21
Have in last few months (after some thorough investigation into ZFS and FreeNAS) been building and testing my first FreeNAS-unit.
The unit works, but is not 'running'... the speeds for transferring files and listing folders are way too slow.
So please let me have your raw remarks on the hardware chosen and settings done etc. Dont spare me to be polite... the essence is to get this thing working so I can get ahead with using the NAS.
I have 'newbie' status (wearing that status with pride ;)). My actual status is one of being well involved and on par with general computer building, having some (ancient) programming knowledge (but nothing close to C++). I have made it my task to get a fairly thorough understanding of ZFS and FreeNAS and also learned a lot when doing testing in last months. Yet I lack FreeNAS-experience plus need to be able to discuss (detailed) tech matters with guys that do have that experience, in order to be able to get this NAS of mine here changed where necessary and running.
Use-case:
The NAS will be a dedicated unit for storing, building (processing) and sharing a large musiccollection (~2 million tracks). The files are all fairly large files of either ~10MB or ~40MB. No small files. Just folders filled 50/50 with these 10 or 40MB files. This is relevant as this is approx 25x larger than average files; also the record/blocksize can be large (i.e. 1MB or 4MB; 30x larger than the standard 128kB); this I believe makes transfer-performance and also directory-listing easier as its a lot less IO-activity. Lots of MB per file and per block that is handled. (practise unfortunately does not show this... just read on..).
Performance-needs/lack:
The main performance demands are on the activities that take place between this NAS and a single WindowsPC from which the main processingwork takes place. This is done via a peer-to-peer 10Gb-connection. Next to that there is a normal 1Gb LAN-connection (think in the order of say max 10 web-clients simultanuously).
A SMB/Samba-share is used for exchanging the data with the WindowsPC; data is accessed via a networkconnection as made in Windows Explorer.
The relevant performance and activities where the NAS untill now is way too slow are:
So long story short: this is way too slow for practical work. It's a waiting game with each action that you start...
Alterations/tests done:
Below there is more info on the hardware set-up and all settings, and more on tests done.
In short I obviously have tackled the aspect of IOPS and therefore VDEVs by also testing with 2 and with 4 and even with 8 VDEV's but this did not give any better transfers-speeds. Also I have destroyed this diskpool and created a NVME-SSD-pool: also here no better speeds.
But you can also see from the write-speed (300MB/s), where data is coming from superfast RAMdisk, going via 10Gb=1200MB-NICs, writing to superfast ARC, that lots of that 1200MB/s is lost somewhere.
Similar to this I also did re-re-re-reads of same testbatch (ARC freed for file-cache, to make reads come from ARC) and still got no more than that same 300MB/s.
(again: RAM does ~6000MB/s; networkspeed tested via iperf on 1200MB/s; so again performance-loss from 1200 to 300MB/s).
So to me it looks like the diskpool is not the cause for the slow speeds (even when reading from RAM its slow).
(diskpool itself gives ~700-750MB/s in dd-testing by the way).
And the line-speed is also OK (1200MB/s).
So (assuming this all is OK... we need to doublecheck that somehow), where then does the slow speed come from?
It looks like the handling per file is taking too much time.
(if you transfer very large files, so have less files/sec, the speed goes up considerably) (very large files of say 250MB give 400MB/s read; 600MB/s write).
Even when just plain deleting files, it only does this at 100-150files/sec.
So the system is 'sniffing up and down' each file first and then only processing it (sorry for the vivid expression..).
So I dived into SMB/Samba and worked with several suggested settings. Also installed the new FreeNAS testversion 11.3 Beta1, as in this version newer versions of SMB are included and there's a default group of settings now in that, initiated when choosing for 'SMB-share' in the Dataset (on good advice of Anodos).
Unfortunately none of this made the speed better. On the contrary, only slower (only the deletespeeds got a bit better).
Situ now & my request to you:
So... I am still stuck with a NAS that is not even giving me the performance of an old harddisk under Windows...
Below there is the full data on hardware set-up and all settings, and more on tests done.
Apologies for the 'textwall' and 'datadump' but you cannot really give any good advice when not knowing what the specifics of the system are.
Just dig in and let me know your suggestions on what to change and/or how to test to isolate the speedproblem.
I am available for any questions or to do any tests.
Thanks in advance!
INFO:
The unit works, but is not 'running'... the speeds for transferring files and listing folders are way too slow.
So please let me have your raw remarks on the hardware chosen and settings done etc. Dont spare me to be polite... the essence is to get this thing working so I can get ahead with using the NAS.
I have 'newbie' status (wearing that status with pride ;)). My actual status is one of being well involved and on par with general computer building, having some (ancient) programming knowledge (but nothing close to C++). I have made it my task to get a fairly thorough understanding of ZFS and FreeNAS and also learned a lot when doing testing in last months. Yet I lack FreeNAS-experience plus need to be able to discuss (detailed) tech matters with guys that do have that experience, in order to be able to get this NAS of mine here changed where necessary and running.
Use-case:
The NAS will be a dedicated unit for storing, building (processing) and sharing a large musiccollection (~2 million tracks). The files are all fairly large files of either ~10MB or ~40MB. No small files. Just folders filled 50/50 with these 10 or 40MB files. This is relevant as this is approx 25x larger than average files; also the record/blocksize can be large (i.e. 1MB or 4MB; 30x larger than the standard 128kB); this I believe makes transfer-performance and also directory-listing easier as its a lot less IO-activity. Lots of MB per file and per block that is handled. (practise unfortunately does not show this... just read on..).
Performance-needs/lack:
The main performance demands are on the activities that take place between this NAS and a single WindowsPC from which the main processingwork takes place. This is done via a peer-to-peer 10Gb-connection. Next to that there is a normal 1Gb LAN-connection (think in the order of say max 10 web-clients simultanuously).
A SMB/Samba-share is used for exchanging the data with the WindowsPC; data is accessed via a networkconnection as made in Windows Explorer.
The relevant performance and activities where the NAS untill now is way too slow are:
- File-transfer: I'd be happy to get about 500MB/s (then also on-pool file-transfers (half that speed) are OKisch):
- actual reads: 150MB/s (copy-paste 20GB-batch of 10MB-files; file-cache nearly null (ARC claimed for Metadata); read from NAS, write to RAMdisk)
- actual writes: 300 MB/s (same but vise versa)
- actual reads: 150MB/s (copy-paste 20GB-batch of 10MB-files; file-cache nearly null (ARC claimed for Metadata); read from NAS, write to RAMdisk)
- Directory-listing: (opening a folder in Windows Explorer... waiting for the file-data to come on screen...)
- opening folder from FreeNAS-share: 25 files/sec (often also totally stopping/halting)
- opening same data-folder, from an old 1TB harddisk, under Windows: 50 files/sec (non-stop).
- opening folder from FreeNAS-share: 25 files/sec (often also totally stopping/halting)
- Deleting files: (right-clicking a folder in Windows Explorer and deleting it)
- from a folder from FreeNAS-share: 100files/sec
-
- same data-folder, from an old 1TB harddisk, under Windows: 1000 files/sec.
Alterations/tests done:
Below there is more info on the hardware set-up and all settings, and more on tests done.
In short I obviously have tackled the aspect of IOPS and therefore VDEVs by also testing with 2 and with 4 and even with 8 VDEV's but this did not give any better transfers-speeds. Also I have destroyed this diskpool and created a NVME-SSD-pool: also here no better speeds.
But you can also see from the write-speed (300MB/s), where data is coming from superfast RAMdisk, going via 10Gb=1200MB-NICs, writing to superfast ARC, that lots of that 1200MB/s is lost somewhere.
Similar to this I also did re-re-re-reads of same testbatch (ARC freed for file-cache, to make reads come from ARC) and still got no more than that same 300MB/s.
(again: RAM does ~6000MB/s; networkspeed tested via iperf on 1200MB/s; so again performance-loss from 1200 to 300MB/s).
So to me it looks like the diskpool is not the cause for the slow speeds (even when reading from RAM its slow).
(diskpool itself gives ~700-750MB/s in dd-testing by the way).
And the line-speed is also OK (1200MB/s).
So (assuming this all is OK... we need to doublecheck that somehow), where then does the slow speed come from?
It looks like the handling per file is taking too much time.
(if you transfer very large files, so have less files/sec, the speed goes up considerably) (very large files of say 250MB give 400MB/s read; 600MB/s write).
Even when just plain deleting files, it only does this at 100-150files/sec.
So the system is 'sniffing up and down' each file first and then only processing it (sorry for the vivid expression..).
So I dived into SMB/Samba and worked with several suggested settings. Also installed the new FreeNAS testversion 11.3 Beta1, as in this version newer versions of SMB are included and there's a default group of settings now in that, initiated when choosing for 'SMB-share' in the Dataset (on good advice of Anodos).
Unfortunately none of this made the speed better. On the contrary, only slower (only the deletespeeds got a bit better).
Situ now & my request to you:
So... I am still stuck with a NAS that is not even giving me the performance of an old harddisk under Windows...
Below there is the full data on hardware set-up and all settings, and more on tests done.
Apologies for the 'textwall' and 'datadump' but you cannot really give any good advice when not knowing what the specifics of the system are.
Just dig in and let me know your suggestions on what to change and/or how to test to isolate the speedproblem.
I am available for any questions or to do any tests.
Thanks in advance!
INFO:
- Use-case:
- Mass quantities (~2 million) of audiofiles; 10MB/40MB size (no small files; just folders filled 50/50 with these larger audio-files)
- Critical processes are only between 1 WindowsPC and FreeNAS (10Gb peer to peer)
- Relating to collection-building: large runs (say 100.000 files) in file-transfers/listings/deletes or tagdata-writes.
- Hardware set-up:
- NAS:
- FreeNAS v 11.3 Beta1
SuperMicro X8DTL-iF
Dual Intel Xeon X5670 CPU's (3Ghz - 6c/12t)
4x 8GB ECC-RAM
(L2ARC: Transcend 220s 500GB NVME-SSD (not in use yet)
IT-flashed Dell H310 controller
Pool of 8x 8TB WD-Red hard-disks (Z2-VDev)
NIC: 10Gb Emulex dual-port SFP+
- FreeNAS v 11.3 Beta1
- (1-on-1 connected via 10Gb direct-attach SFP+cable with 1 Windows10-PC (workstation from which most of the collection-processing is done)
- Windows10-PC: Mobo Foxconn Intel C602 X79 / dual Xeons E5-2660 / 32 GB RAM / mobo-based 10Gb FSP+ NIC (Intel 82599)
- NAS:
- Settings:
- Interfaces (NAS/WindowsPC):
- In the NAS/Interface-setting:
- added the 'MTU 9000' (to use the large segment size)
- Furthermore for the NAS-NIC: added a series of settings in the Tunables (please see the screenshots ‘Tunables (1)(2)(3)’ in the attachments
- For the WindowsPC-NIC: (settings on that adapter):
On jumbo-frames 9000mtu
No. of receive buffers 4096
No. of transmit buffers 16384
All offloading disabled
RSS enabled; 16x no of RSS
Interupt moderations disabled
File/printer sharing disabled
Flow control disabled
- In the NAS/Interface-setting:
- Dataset: sync disabled, compress on, 1MB record/blocksize (also tried 4MB); case Insensitive; Windows-share
- SMB/Samba-share:
- (1)(under previous FreeNAS 11.2-U5 install):
- General SMB-settings: tested this series of settings (added to the SMB-services, aux section) (deleted them again as speed got worse in stead of better…)(indicated best speeds therefore are without these general settings):
- socket options = iptos_lowdelay so_rcvbuf=131072 so_sndbuf=131072
min receivefile size = 16384
use sendfile = yes
aio read size = 16384
aio write size = 16384
server signing = no
strict allocate = yes
client signing = no
client min protocol = smb2
client max protocol = smb3_11
server min protocol = smb2
server max protocol = smb3_11
log level = 1
- socket options = iptos_lowdelay so_rcvbuf=131072 so_sndbuf=131072
- Settings on Samba-Share:
Deleted VFS-objects zfsacl and zfs_space ; added Ixnas
Added in aux parameters:
case sensitive = true
mangled names = illegal
aio write size = 16384
- General SMB-settings: tested this series of settings (added to the SMB-services, aux section) (deleted them again as speed got worse in stead of better…)(indicated best speeds therefore are without these general settings):
- (2)(under current FreeNAS 11.3 Beta1 install:)(nice new fresh version..;-)
- chosen for share-type 'SMB' in the Dataset; this now default gives some optimized settings (not all previous; don't know why)
- these 'optimised SMB-settings' under 11.3 Beta1 did not improve the speeds.. got slower instead... (just the deletes got some better).
- (1)(under previous FreeNAS 11.2-U5 install):
- Interfaces (NAS/WindowsPC):
- Tests & results: (below you can see the screenshots):
- Iperf-test gives 1200MB/s (and I also managed to get about 900MB/s accross when doing real copy-paste of very large files, using a NVME-SSD as pool), so it looks like the (10Gb) NIC to NIC speed is OK
- dd-test gives 700MB/s write and 750MB/s read; so diskpool-speed is also OK (although the pool is 8 disks and each disk has a seq speed of 200; so 1600 reads is the potential of the diskpool; looks like the RaidZ2 results in say loosing half the bandwidth)
- real transfer-speeds when doing copy-paste of a 20GBbatch of files, being 10MB in size: (file-cache limited by claiming almost all ARC for metadata-cache)
- reading from NAS, writing to RAMdisk on WindowsPC: 150MB/s
- writing to NAS, reading from RAMdisk on WindowsPC: 300MB/s.
- .
- especially the 150MB/s reads is troublesome, too slow for practical work
- (I work with a SMB/Samba-share (entered on the WindowsPC via a networklocation, made in Windows Explorer)
- Directory-listing (opening folders with lots of files and waiting for all tag-data to list):
- 25files/sec (vs 50 files/sec for an old, simple harddisk, under Windows direct)
- Deleting folders/lists of files:
- 100files/sec (vs 1000 files/sec for an old, simple harddisk, under Windows direct).
- My conclusion so far:
- It is clear that FreeNAS, using the hardware set-up used here, and going via SMB/Samba, is able to perform at least as good/bad as an old, single harddisk under Windows. And also it is clear that just having 150MB/s read-speed is not normal.
- It looks like there is something wrong in hardware or settings that is making the handling of files go slow.
(if you transfer really big files (say 250MBfiles)(so very few files per sec), speed is OK (400MB/s read); but if you transfer 10MBfiles (still fairly big!) speed drops quite a lot to that 150MB/s) (even plain deleting of files goes slow; 10x slower than an old harddisk).
- What helps if you can please let me know:
- What suggestions you have or ideas on where the problem can sit
- How I could do tests in isolating where the problem is.
Attachments
-
Jperf network results (Windows=Client, NAS=Server)(~1200MBps).jpg283 KB · Views: 299
-
network speed test (writing to NAS, using NVME-SSD pool (1GB-size files) (~900MBps).PNG1.3 MB · Views: 311
-
dd test WRITE to Diskpool (100GB-file) (1 resp 4MB bs) (~700MBps).JPG38.5 KB · Views: 297
-
dd test READ from diskpool (100GB-file) (~750MBps).JPG62 KB · Views: 304
-
transferspeed reading from NAS (20GB-batch of 10MBfiles).JPG30 KB · Views: 297
-
transferspeed writing to NAS (20GB-batch of 10MBfiles).JPG31.9 KB · Views: 302
-
directory-listing speeds compared.jpg1.4 MB · Views: 305
Last edited: