kern.maxswzone - increase needed (severe performance issues?)

[purduephotog]

I had previously tuned/posted my details on my 8x 2tb WD green Raid-Z2 with 8gb of ram (now 12gb). I was sustaining 50-66 mb/sec on transfer.

I stood up a new server- 9x3tb red Raid Z3, 16gb ram, and set up an rsync task from the green to the red.

In the process I began backing up data to the green again... only to find my transfers in the sub 2mb/sec rate.

Recently I've lost the web interface; I have to reboot the computer to bring them back up.

A dmesg right before I do so shows the following:

Code:

mfi0: 5620 (431834483s/0x0008/info) - Battery charge complete
swap zone exhausted, increase kern.maxswzone
pid 2649 (smbd), uid 1001, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 2028 (python), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 2583 (python), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 2157 (collectd), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 1979 (nmbd), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 1982 (smbd), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 1987 (smbd), uid 0, was killed: out of swap space
swap zone exhausted, increase kern.maxswzone
pid 2968 (nginx), uid 80, was killed: out of swap space

Could really use a suggestion here, besides bugger off that is ;)

Motherboard is a Celeron dual core 1.1ghz, 12gb DDR3 ram, Perc 6i, a 2gb USB drive (will move to 4gb later).

 

[gpsguy]

Duplicate thread - you never answered jgreco's reply as to what "swapinfo" reported.

 

[purduephotog]

My apologies; I didn't think they were related (other than the gui crashing)- and when it rebooted I figured the information was lost.

Code:

@freenas ~]$ swapinfo
Device          1K-blocks    Used    Avail Capacity
/dev/mfid0p1.eli  2097152        0  2097152    0%
/dev/mfid1p1.eli  2097152        0  2097152    0%
/dev/mfid2p1.eli  2097152        0  2097152    0%
/dev/mfid3p1.eli  2097152        0  2097152    0%
/dev/mfid4p1.eli  2097152        0  2097152    0%
/dev/mfid5p1.eli  2097152        0  2097152    0%
/dev/mfid6p1.eli  2097152        0  2097152    0%
/dev/mfid7p1.eli  2097152        0  2097152    0%
Total            16777216        0 16777216    0%

 

[purduephotog]

And looking in loader.conf in /boot/defaults/

Code:

#kern.maxproc=""                # Set the maximum # of processes
#kern.maxssiz=""                # Set the max stack size
#kern.maxswzone=""              # Set the max swmeta KVA storage
#kern.maxtsiz=""                # Set the max text size
#kern.maxusers="32"            # Set size of various static tables
#kern.msgbufsize="65536"        # Set size of kernel message buffer
#kern.nbuf=""                  # Set the number of buffer headers
#kern.ncallout=""              # Set the maximum # of timer events
#kern.ngroups="1023"            # Set the maximum # of supplemental groups
#kern.sgrowsiz=""              # Set the amount to grow stack
#kern.cam.boot_delay="10000"    # Delay (in ms) of root mount for CAM bus
                                # registration, useful for USB sticks as root
#

 

[Dusan]

and when it rebooted I figured the information was lost.

There is a swap graph at the reporting screen. The stats are saved every hour, so you should be able to see swap usage levels before the reboot.

 

[purduephotog]

Thanks Dusan-

Unfortunately, the gui is being shut down when python crashes due to memory issues- so while the data might be somewhere, it isn't recorded or logically written even though the system is running.

That means I can't see anything. I have seen the memory spike to 10gb on the 16gb system during runs, so I will see if I can drop another 8gb chip into it to get it up to 16gb and see whether or not it crashes.

 

[cyberjock]

There was a discussion about swap around here a few weeks ago. The short is that if you are in a position where you are using swap, you should have more RAM than what you currently have. Swap space for FreeNAS is like a spare tire. If things get messy it'll get you to your destination, but it'll be slow, and you shouldn't rely on it any more than you have to. Honestly, I'm surprised you are using only 12GB of RAM with a pool that size.

If both are ZFS(which I think they are from your post) then I'd do ZFS replication before rsync. Rsync is incredibly slow comparatively while ZFS replication is like a rocket.

 

[purduephotog]

Thanks for the hints. I intend to set up replication between the two boxes, and the Rsync was mostly to test pushing down the data from the windows box.

I just wish I knew what was causing the crash- if I knew it was lack of ram, then that's the fix. Proving a negative, however, is extremely difficult to do.

 

[cyberjock]

Well, if you are running out of swap space you will start having stability issues. Depending on your problem, more RAM may not fix it. If you have a runaway program or something that will consume infinite amounts of RAM then you should work in fixing that problem before anything else. In all seriousness, I'd get another stick because more RAM will make things faster anyway, and you can't really have "too much". In your situation I'd have tried for at least 16GB of RAM to start, 24 would be better.

I will tell you that when i setup my server with 12GB of RAM I had no issues at all. Then suddenly, and out of the blue, performance was horrible. I was unable to stream a single movie with no other loading on the server, transfers were single digit MB/sec, and copying anything to/from the server was eye-gougingly slow. I upgraded from 12 to 20GB of RAM, and poof, instant Gb saturation again.

 

[purduephotog]

I think you've hit my problem- I assume I'm running out of swap space, due to the messages received in dmesg, and it would logically follow that it's a memory shortage... but how do you prove it? Just because the GUI goes down?

*sigh* I guess I can keep upping the ram, at least until 16gb, then I'll have to deploy either a new mobo or a whole new interface. One of the supermicro boards and a low end pentium 4th gen sounds decent.

 

[purduephotog]

Well, I did pull this out of the last crash:

 

[gpsguy]

So, the graph clearly shows that you were using 13+Gb of swap and you've got 16Gb in total. As cyberjock said, start with a RAM upgrade. In fact, were you a noob, asking about setting up a new system, we probably would have recommended 16Gb to start with. It sounds like you started with 8Gb.