wired memory

[kr55]

Hi,

I know this has been mentioned in other posts when i've googled, but i'm still unsure about zfs and wired memory. When the wired memory figure in 'reporting' is constantly high and not dropping, some posts seem to suggest it's not a concern "if performance is ok". In my case, a server with 24gb memory, the wired memory is around 17gb used after a couple of backup jobs from another server and the wired memory just seems to increase rather than drop each time a job is run.

If the wired memory keeps getting consumed (by zfs?), will the machine eventually run out of free memory to handle my backup jobs? In short, I'm not sure what's happening with the physical memory on my system!

Any advice would be great.

 

[William Grzybowski]

ZFS will use as much as memory it has disponible, for cache and to retain the metadata tables. ZFS runs in kernel space, so the memory counts as wired and it wont release it.
Be default it is capped by about 80% of RAM.

If you wish to reduce it (not recommended), google about vfs.zfs.arc_max and vm.kmem_size loader/tunables.

 

[JaimieV]

Physical memory is there to be used. If it wasn't, that would be a waste of resources.

 

[kr55]

Physical memory is there to be used. If it wasn't, that would be a waste of resources.

as long as it's being used productively! :p

I guess with ZFS the ratio is more memory you have = the more wired and "free" memory you'll get via the 80% ratio. Obviously if the 20% is sitting there doing nothing then it's a waste, so i take your point, but....

I just want to gauge whether i'd need to increase the memory in that 20% for my "external to freenas" backup jobs or whether the current wired memory will absorb the required memory resource? Is there a way of measuring a memory bottleneck?

 

[JaimieV]

A plain ZFS install would like to have 1gig per terabyte, as a rule of thumb. It'll happily take advantage of more, and will run with much less but you'll lose a lot of speed when multiple clients are asking for data.

Dedupe would like the moon on a (RAM) stick, but we'll not count that!

I don't know any bottleneck measuring tools, and would be interested too. But the backup jobs will run mostly from the info in the ZFS memory caches, they'll only use a bit more to handle temp storage of filename lists etc. If they needed more RAM, the kernel would force ZFS to hand some back.

 

[kr55]

thanks for the fast reply! i'm within the 1gb per tb rule of thumb. It's a bit trial and error for me at the moment!

I'm not sure whether my system config may be giving me some troubles..i have hardware raid 5 with a single disk visible in freenas - which is not raidz'd. From what i've read on the forum, I should probably have jbod mode instead of hardware raid, but i can't see how i can do that on my system. With that in mind, I tried to set up with UFS on the h/w raid instead, but i get permissions problems in windows for some reason. ZFS works fine which is why i've gone back to that! I'm still a bit unclear as to whether you shouldn't use ZFS with hardware raid or just don't use Raidz on top of hardware raid though.

Additionally my Raid controller had a fault and was replaced last week so that may have been my main performance problem before, all of my RAM seemed to be used up then! Now it's looking like the 80% cap is working, so i was just trying to understand what was going on. I'm retesting stuff again now the controller has been replaced.

 

[JaimieV]

Probably worth letting the system settle for a while and check it over in a week or three.

You're ~okay~ as is with a single ZFS striped volume across the hardware RAID, although you will have to worry about how the RAID system might interact with ZFS during failure modes, and your ZFS set isn't portable to another machine because it's reliant on the hardware RAID card. Which sucks.

Putting a RAIDZ across a hardware RAID leads to insanity, so good choice on not doing that.

What you do still get while using a ZFS volume on top of a hardware RAID is data scrubbing which is worth the price of entry alone. And ZFS compression which may or may not be appropriate for your data loadout.

What is the RAID controller? Others might know how to knock it into JBOD mode in case you want to do things fully ZFS in future.

 

[kr55]

The raid controller is a dell perc h700. I've got another post on the forum asking if anyone's set something similar up. It's looking after 6 x 3tb sas drives. I read 'somewhere on some forum' that people had raid0'd all of their disks to make 'like' jbod, but i'm not sure if that's the way to go.

btw thanks for your advice!

 

[petr]

ZFS will use as much as memory it has disponible, for cache and to retain the metadata tables. ZFS runs in kernel space, so the memory counts as wired and it wont release it.
Be default it is capped by about 80% of RAM.

If you wish to reduce it (not recommended), google about vfs.zfs.arc_max and vm.kmem_size loader/tunables.

Hi, sorry to resurrect old thread! However, I could use a bit of help. I have a 32GB machine, which I upgraded memory to run a few VMs, which means I need that memory free.

I can see the roughly 80% consumption being allocated but I am unable to find corresponding numbers for tunables:

Code:

$ sysctl -a |grep vm.kmem
vm.kmem_map_free: 19019390976
vm.kmem_map_size: 14256087040
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 33275490304

Seems to me that all of the number refer to the actual amount of memory, not to the 80% value. Which value shall I change to make it effectively a 24GB RAM system (19.2 GB cache) for caching purposes?

 

[solarisguy]

https://www.freebsd.org/doc/handbook/zfs-advanced.html

sysctl -a | grep vfs.zfs.arc_max

 

[petr]

https://www.freebsd.org/doc/handbook/zfs-advanced.html

sysctl -a | grep vfs.zfs.arc_max

I've had that already set to approx 8GB - but the wired memory of the kernel was close to the 80% of my RAM. I've now changed the vm.kmem_size, let's see if it works!

 

[solarisguy]

It makes no sense to me. You can adjust vfs.zfs.arc_max only upon a reboot.

 

[petr]

It makes no sense to me. You can adjust vfs.zfs.arc_max only upon a reboot.

That is correct - the vfs.zfs.arc_max was set to 8GB from my previous machine's autotune script.

 

[no_connection]

Have you actually experienced an event where ZFS didn't back off when other processes needed RAM?

From what I read you should not *need* to limit it manually.

 

[petr]

Have you actually experienced an event where ZFS didn't back off when other processes needed RAM?

From what I read you should not *need* to limit it manually.

No, but I read in some of the threads that one should be careful when using VMs that use fair amount of memory to not interfere with the ZFS's memory usage.. so I just wanted to be careful! Also, VirtualBox can be fiddly enough on its own, let alone testing if competes for memory etc.

 

[no_connection]

I asked CJ in the VB thread and ZFS should automatically adapt. Contending for resources was my concern as well at the time.

Limiting it more than necessary does not seem beneficial. (correct me if I'm wrong)

VB has been a disappointment for me though and it is far to unstable to be of much use in production or even home use.
Bought a second server for ESXi instead.

<rant>
On a related note I have had quite some problem during lab with VB and 2k8 server (host being win7 this time). DC server simply got worse and worse and would stop communicating properly. GPO for example never updated, two DC servers would not communicate, replication broke, DHCP and DNS would never authenticate. And so on.
Managed to get it running by only using one NIC. (did try every type of nic before that)
Some other groups never got their to work once it started to deteriorate with same symptom as mine.
And of course some didn't have problems at all.

I would blame 2k8 for the problems, but if these problems ever crept up it would be patched long ago.
Although the new network stack is as unreliable as anything I've seen, and win7 have a nasty problem of dropping network connection once i a while for a few sec (happens on two physical and one VM for me alone) and that was never patched.
</rant>

 

[cyberjock]

Honestly, I think that some of the VB problems are that people are doing things like what I'm doing.. a single 10 disk vdev RAIDZ2 with VB. If you want to do virtual machines and get good performance you are basically implying >32GB of RAM, L2ARC, etc. Otherwise you start getting I/O timeouts (which means lost data) and things go downhill quickly. If you want to run one or two VMs that are lightly loaded and don't use hardly any disk resources you might be okay. But every time I've done a teamviewer with someone using VB they've got a situation where they have 4+ VMs using up 1/2 the system's RAM (or more), the ARC is like 8GB or so in size, and the pool is maxed out with I/O and can't keep up.

I'm not saying that's everyone's problem, but I have yet to see someone show me an unreliable server that was done in what I would call "a reasonable way".

VB isn't my favorite Type-2 hypervisor. If VMWare Workstation existed in FreeBSD I'd have a jail for that in a heartbeat.

 

[petr]

Honestly, I think that some of the VB problems are that people are doing things like what I'm doing.. a single 10 disk vdev RAIDZ2 with VB. If you want to do virtual machines and get good performance you are basically implying >32GB of RAM, L2ARC, etc. Otherwise you start getting I/O timeouts (which means lost data) and things go downhill quickly. If you want to run one or two VMs that are lightly loaded and don't use hardly any disk resources you might be okay. But every time I've done a teamviewer with someone using VB they've got a situation where they have 4+ VMs using up 1/2 the system's RAM (or more), the ARC is like 8GB or so in size, and the pool is maxed out with I/O and can't keep up.

I'm not saying that's everyone's problem, but I have yet to see someone show me an unreliable server that was done in what I would call "a reasonable way".

VB isn't my favorite Type-2 hypervisor. If VMWare Workstation existed in FreeBSD I'd have a jail for that in a heartbeat.

I really treat the VB as a nice feature to have but I understand its limits. I have been hitting huge number of bugs myself - for example, changing the VNC port crashes the VM, multiple occasions of VMs locking up and not being able to perform anything via the UI/command line of VB, had to manually kill the process.

At this moment, I've got a VM that's using approx 0.75% of a single core, 2GB of RAM and that has been running flawlessly (security camera motion detection + event creation).

For more complex tasks, I've created a VM with 6GB of RAM, and that one seems to be having fair amount of trouble - mainly lock-ups. It is hardly doing anything IO intense - at this moment, it's only uploading CrashPlan content via my slow uplink (with plans to put Dropbox and Google Drive Insync there as well). To exercise it out a bit, I also run stress -c 2 to make it work out, and it simply freezes at random intervals (hours).

I've also been connected via SSH running htop, and the screen it froze on reported load number of 8 - which does not make sense, as I was not loading it in any sense.

There is absolutely no reason for IO starvation - the 2 VMs are running off 2xRAIDZ2 in a single pool.

Lastly - Why would IO starvation cause loss of data? During my experimentation with VMWare on my Mac (VMs stored on the FreeNAS), it simply waited - I once even accidentally unplugged the network cable, only to find VMware "hanging there" and resuming after I plugged it back in.

Also, I've got more than enough RAM to support running of 2+6gb machines with 32GB total, and 24TB raw storage on the pools with little use (mainly media archive / backups) - am I wrong here?


I am hardly a demanding user - I only run VB to run Dropbox / other sync apps that do not exist in a jail, thus a linux box is needed (I could probably make some of the tools work on linux jail but I do not have the expertise with those).

 

[cyberjock]

Windows (in a VM) and other OSes will only let a queued I/O last so long. If you take too long it simply discards it because it assumes that storage medium is gone/faulted. So if you have enough I/O to hit that threshold, you'll start losing data. Some will be written when the queue is worked down, and some will be lost when the queue is too long. This is a particularly nasty scenario because people will get the errors but things may look okay. But eventually you lose some key system files (or reboot and need to access data that doesn't exist) or your file system becomes inconsistent in the VM and you've got a non-functioning VM. :P

RAM isn't going to save the I/O problem. Only a sufficiently fast pool (and by fast I mean high I/O, which means multiple vdevs aka mirrors and possible L2ARC).

I ran 2 VB VMs on my server and I was having problems just getting the second VM to boot without problems, that were directly attributable to I/O because I have a single vdev, 32GB of RAM, and no L2ARC.

I hear the stories, and some of them I understand (like changing the port number), but when it comes to everything else I don't see anyone saying "I have 64GB of RAM, an L2ARC and 4 mirrored vdevs running my VMs". It's always a single RAIDZ2 vdev, 32GB of RAM or less, and no L2ARC. That's just not going to give good performance. Even when I did iSCSI to my ESXi server starting up one VM about made my FreeNAS box throw up because of the high I/O from booting. :P

 

[petr]

Another freeze to day - this time both the host and the guest were IDLE for all intents and purposes - there is absolutely NO chance of IO starvation.

I am seeing this in the log:

Code:

22:13:17.752113 AIOMgr: I/O manager 0x00000808e12420 encountered a critical error (rc=VERR_FILE_AIO_NO_REQUEST) during operation. Falling back to failsafe mode. Expect reduced performance
22:13:17.767509 AIOMgr: Error happened in /wrkdirs/usr/ports/emulators/virtualbox-ose/work/VirtualBox-4.3.12/src/VBox/VMM/VMMR3/PDMAsyncCompletionFileNormal.cpp:(1664){int pdmacFileAioMgrNormal(RTTHREADINT*, void*)}
22:13:17.767523 AIOMgr: Please contact the product vendor
22:13:48.120506 AHCI#0: Port 0 reset
22:13:48.120560 AHCI#0P0: Cancelled task 0
22:13:48.120586 AHCI#0P0: Cancelled task 1
22:13:48.120597 AHCI#0P0: Cancelled task 2
22:13:48.120605 AHCI#0P0: Cancelled task 3
...

I've found a mention on the forums that somebody had this problem with SATA controller without host cache - so I've enabled the host cache, let's see!

I've also just checked what is my so far solid VM using - it was imported from VMWare and it's using emulated SCSI controller.

Here is a link to the ticket I've found: https://www.virtualbox.org/ticket/12648

Could the issue be with disabling host cache with a drive hosted on a ZFS volume? From my understanding, it stops the host from caching the file IO - however, how does one even do this on ZFS?


Also, I am getting a bit confused with the IO requirements here. My previous setup involved lower-spec server, NFS share and a Mac mini with VMWare. I was running similar VM setup on the Mini (Mac OS as host, Vmware Fusion as virtualisation software) and I had absolutely no trouble over the course of a year. It seems odd to me that after moving the VMs directly onto more powerful server, the IO cannot keep up?