slushieken
Dabbler
- Joined
- May 28, 2014
- Messages
- 24
winbindd[6530]: failed to munlock memory: Cannot allocate memory (12)
Also errors of failed to mlock memory.
This is occurring on my NAS constantly now. Right after reboot as well.
Research shows that this is a problem not with the erroring service but with the ability of the erroring service to get the 'wire' or wire the memory for its work. I don't know what wired memory is yet so forgive my explanation.
This seems to be blocking activity. It seems to disconnect/log off my user from AFP (Apple Filing Protocol) Time Machine shares. This leaves my time machine backup files in an open state on the server, and the backup client then reports the backup files are already in use and therefore it cannot access them. I reset AFP or reboot to clear that state, rinse/repeat.
I am not sure what to provide or how much detail. I am guessing this is a bug and I need to submit it but I have not done that before (ever) so asking here to 1. be sure that is what I have and 2. hoping someone can help fix it.
The basics:
BuildFreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
PlatformAMD Athlon(tm) II X3 400e Processor
Memory16324MB (ECC memory - passes all checks)
The following was set via autotune. These are my only non vanilla settings:
vfs.zfs.arc_max 10584279705
vm.kmem_size 11760310784
vm.kmem_size_max 14700388480
5x500gb disks in raidz1 array - total size 2TB, 1.7TB available
https://bugs.freenas.org/issues/3842 is my problem. Adjusting memory settings did not help there. Says it was fixed in earlier release without any details given.
arc_summary.py shows all memory and performance within norms.
I have very slowly built this system at home step by step working out problems as I find them. So at this point I have not configured anything more than the Time Machine AFP share and backups. I use a simple windows 2012 server with 2 users Active Directory Domain for authentication. That part is working well. I did notice that I did not see my user accounts when setting initial permissions when I created the share in the GUI. So I set those via command line instead, which worked fine. I have seen indications that the 2 probs are related.
This is my chance to also say Thanks to all of you for the help you give and your time. Off and on over my life I have helped out in forums and etc answering questions; but the way some people stay around helping others for months and years on a project they are not even paid for just amazes me. You are truly a different breed IMHO. It is people like you that make open source possible and your dedication is an inspiration to me.
Also errors of failed to mlock memory.
This is occurring on my NAS constantly now. Right after reboot as well.
Research shows that this is a problem not with the erroring service but with the ability of the erroring service to get the 'wire' or wire the memory for its work. I don't know what wired memory is yet so forgive my explanation.
This seems to be blocking activity. It seems to disconnect/log off my user from AFP (Apple Filing Protocol) Time Machine shares. This leaves my time machine backup files in an open state on the server, and the backup client then reports the backup files are already in use and therefore it cannot access them. I reset AFP or reboot to clear that state, rinse/repeat.
I am not sure what to provide or how much detail. I am guessing this is a bug and I need to submit it but I have not done that before (ever) so asking here to 1. be sure that is what I have and 2. hoping someone can help fix it.
The basics:
BuildFreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
PlatformAMD Athlon(tm) II X3 400e Processor
Memory16324MB (ECC memory - passes all checks)
The following was set via autotune. These are my only non vanilla settings:
vfs.zfs.arc_max 10584279705
vm.kmem_size 11760310784
vm.kmem_size_max 14700388480
5x500gb disks in raidz1 array - total size 2TB, 1.7TB available
https://bugs.freenas.org/issues/3842 is my problem. Adjusting memory settings did not help there. Says it was fixed in earlier release without any details given.
arc_summary.py shows all memory and performance within norms.
Code:
[root@vault] ~# arc_summary.py
System Memory:
1.76% 278.83 MiB Active, 2.53% 400.20 MiB Inact
56.49% 8.72 GiB Wired, 0.02% 3.58 MiB Cache
39.18% 6.05 GiB Free, 0.00% 768.00 KiB Gap
Real Installed: 16.00 GiB
Real Available: 99.64% 15.94 GiB
Real Managed: 96.87% 15.44 GiB
Logical Total: 16.00 GiB
Logical Used: 59.72% 9.55 GiB
Logical Free: 40.28% 6.45 GiB
Kernel Memory: 193.50 MiB
Data: 88.37% 170.99 MiB
Text: 11.63% 22.51 MiB
Kernel Memory Map: 10.91 GiB
Size: 75.07% 8.19 GiB
Free: 24.93% 2.72 GiB
Page: 1
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Storage pool Version: 5000
Filesystem Version: 5
Memory Throttle Count: 0
ARC Misc:
Deleted: 352.34k
Recycle Misses: 3.52k
Mutex Misses: 0
Evict Skips: 0
ARC Size: 82.42% 8.12 GiB
Target Size: (Adaptive) 82.42% 8.12 GiB
Min Size (Hard Limit): 12.50% 1.23 GiB
Max Size (High Water): 8:1 9.86 GiB
ARC Size Breakdown:
Recently Used Cache Size: 93.81% 7.62 GiB
Frequently Used Cache Size: 6.19% 515.08 MiB
ARC Hash Breakdown:
Elements Max: 105.75k
Elements Current: 100.00% 105.75k
Collisions: 173.30k
Chain Max: 6
Chains: 16.36k
Page: 2
------------------------------------------------------------------------
ARC Total accesses: 600.39k
Cache Hit Ratio: 97.50% 585.39k
Cache Miss Ratio: 2.50% 15.00k
Actual Hit Ratio: 67.61% 405.90k
Data Demand Efficiency: 96.93% 193.96k
Data Prefetch Efficiency: 18.76% 4.63k
CACHE HITS BY CACHE LIST:
Anonymously Used: 30.10% 176.18k
Most Recently Used: 19.22% 112.52k
Most Frequently Used: 50.12% 293.38k
Most Recently Used Ghost: 0.42% 2.46k
Most Frequently Used Ghost: 0.15% 858
CACHE HITS BY DATA TYPE:
Demand Data: 32.12% 188.00k
Prefetch Data: 0.15% 869
Demand Metadata: 37.22% 217.89k
Prefetch Metadata: 30.51% 178.62k
CACHE MISSES BY DATA TYPE:
Demand Data: 39.71% 5.96k
Prefetch Data: 25.08% 3.76k
Demand Metadata: 12.20% 1.83k
Prefetch Metadata: 23.01% 3.45k
Page: 3
------------------------------------------------------------------------
Page: 4
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)DMU Efficiency: 3.78m
Hit Ratio: 95.81% 3.62m
Miss Ratio: 4.19% 158.34k
Colinear: 158.34k
Hit Ratio: 0.03% 45
Miss Ratio: 99.97% 158.29k
Stride: 3.62m
Hit Ratio: 99.99% 3.62m
Miss Ratio: 0.01% 248
DMU Misc:
Reclaim: 158.29k
Successes: 2.55% 4.03k
Failures: 97.45% 154.26k
Streams: 7.48k
+Resets: 0.71% 53
-Resets: 99.29% 7.42k
Bogus: 0
Page: 5
------------------------------------------------------------------------
Page: 6
------------------------------------------------------------------------
ZFS Tunable (sysctl):
kern.maxusers 384
vm.kmem_size 11760310784
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 14700388480
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 1297345024
vfs.zfs.mfu_ghost_metadata_lsize 698368
vfs.zfs.mfu_ghost_size 1298043392
vfs.zfs.mfu_data_lsize 441823744
vfs.zfs.mfu_metadata_lsize 2333184
vfs.zfs.mfu_size 444617728
vfs.zfs.mru_ghost_data_lsize 420366848
vfs.zfs.mru_ghost_metadata_lsize 90061312
vfs.zfs.mru_ghost_size 510428160
vfs.zfs.mru_data_lsize 8126392320
vfs.zfs.mru_metadata_lsize 40626688
vfs.zfs.mru_size 8204063232
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 16384
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 2646069926
vfs.zfs.arc_meta_used 155423568
vfs.zfs.arc_min 1323034963
vfs.zfs.arc_max 10584279705
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.nopwrite_enabled 1
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.no_scrub_prefetch 0
vfs.zfs.no_scrub_io 0
vfs.zfs.resilver_min_time_ms 3000
vfs.zfs.free_min_time_ms 1000
vfs.zfs.scan_min_time_ms 1000
vfs.zfs.scan_idle 50
vfs.zfs.scrub_delay 4
vfs.zfs.resilver_delay 2
vfs.zfs.top_maxinflight 32
vfs.zfs.write_to_degraded 0
vfs.zfs.mg_noalloc_threshold 0
vfs.zfs.mg_alloc_failures 8
vfs.zfs.condense_pct 200
vfs.zfs.metaslab.weight_factor_enable 0
vfs.zfs.metaslab.preload_enabled 1
vfs.zfs.metaslab.preload_limit 3
vfs.zfs.metaslab.unload_delay 8
vfs.zfs.metaslab.load_pct 50
vfs.zfs.metaslab.min_alloc_size 10485760
vfs.zfs.metaslab.df_free_pct 4
vfs.zfs.metaslab.df_alloc_threshold 131072
vfs.zfs.metaslab.debug_unload 0
vfs.zfs.metaslab.debug_load 0
vfs.zfs.metaslab.gang_bang 131073
vfs.zfs.ccw_retry_interval 300
vfs.zfs.check_hostid 1
vfs.zfs.deadman_enabled 1
vfs.zfs.deadman_checktime_ms 5000
vfs.zfs.deadman_synctime_ms 1000000
vfs.zfs.recover 0
vfs.zfs.txg.timeout 5
vfs.zfs.max_auto_ashift 13
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.trim_on_init 1
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.mirror.non_rotating_inc 0
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.rotating_seek_inc 5
vfs.zfs.vdev.mirror.rotating_inc 0
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.scrub_max_active 2
vfs.zfs.vdev.scrub_min_active 1
vfs.zfs.vdev.async_write_max_active 10
vfs.zfs.vdev.async_write_min_active 1
vfs.zfs.vdev.async_read_max_active 3
vfs.zfs.vdev.async_read_min_active 1
vfs.zfs.vdev.sync_write_max_active 10
vfs.zfs.vdev.sync_write_min_active 10
vfs.zfs.vdev.sync_read_max_active 10
vfs.zfs.vdev.sync_read_min_active 10
vfs.zfs.vdev.max_active 1000
vfs.zfs.vdev.larger_ashift_minimal 0
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.vdev.trim_max_pending 64
vfs.zfs.vdev.trim_max_bytes 2147483648
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.sync_pass_rewrite 2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.zio.use_uma 1
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.version.ioctl 3
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 5000
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
vfs.zfs.trim.enabled 1
vfs.zfs.trim.max_interval 1
vfs.zfs.trim.timeout 30
vfs.zfs.trim.txg_delay 32
Page: 7
------------------------------------------------------------------------
[root@vault] ~#
I have very slowly built this system at home step by step working out problems as I find them. So at this point I have not configured anything more than the Time Machine AFP share and backups. I use a simple windows 2012 server with 2 users Active Directory Domain for authentication. That part is working well. I did notice that I did not see my user accounts when setting initial permissions when I created the share in the GUI. So I set those via command line instead, which worked fine. I have seen indications that the 2 probs are related.
This is my chance to also say Thanks to all of you for the help you give and your time. Off and on over my life I have helped out in forums and etc answering questions; but the way some people stay around helping others for months and years on a project they are not even paid for just amazes me. You are truly a different breed IMHO. It is people like you that make open source possible and your dedication is an inspiration to me.
