Update FreeNAS-11.1-U5 to FreeNAS-11.2 beta2 success ; then freezes

[Wouter]

I did a successful update from FreeNAS-11.1-U5 to FreeNAS-11.2 beta2. Also bhyve VM's were working. Then, at night, suddenly my system freezes. I had to power down and start the machine again. I looked at (remote stored/saved) syslog messages, nothing strange.

then, a day later, again. System freezes, nothing real strange on the monitor or syslog. Now I'm back at FreeNAS-11.1-U5 and no troubles at all.

Anyone that can give me hint to do trouble shooting? Yes, I know, this is beta. Just asking for info to help the project, see what's going on and maybe file a bug report.

 

[kdragon75]

Could it be a memory starvation issue? Do you have autotune enabled or any other tunables? What are your specs and how are your VMs configured?

 

[Wouter]

Could it be a memory starvation issue? Do you have autotune enabled or any other tunables? What are your specs and how are your VMs configured?

Some more output logging just before the freeze would be nice :). I just set the syslog level to "Debug".

About the memory starvation issue: the VM's are small, don't do a lot. I added munin picture below. The box is still running fine, now with 11.1u5. What I can do, is disable the autostart of the VM's, then boot into 11.2beta2 again and see what happens. If it freezes again, it shouldnt have anything to do with the VM's (right?). No tunables set (never did this). Autotune is disabled.

Good idea, to disable the autostart of the two VM's and reboot into beta2, see what happens?

 

[kdragon75]

At this point we need you system parts list. In the graph I see 44GB of memory usage but I have no clue if you have 128GB or 48GB of ram. Also what are the specs for the VMs. We like technical details!

 

[Wouter]

At this point we need you system parts list. In the graph I see 44GB of memory usage but I have no clue if you have 128GB or 48GB of ram. Also what are the specs for the VMs. We like technical details!

/me too!

The VM's are both Ubuntu 17.10, bhyve, uefi, RAM 8GB and 1.5GB big, running via an SSD , the SystemDataSetPool. Both have 2 virtual CPU's.

from dmesg and others:

CPU: Intel(R) Core(TM) i5-6600T CPU @ 2.70GHz (2712.11-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
real memory = 34359738368 (32768 MB)
avail memory = 32940371968 (31414 MB)
FreeBSD 11.1-STABLE #0 r321665+e0c4ca60dfc(freenas/11.1-stable): Wed May 30 14:18:20 EDT 2018


root@box:~ # lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1b.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #19 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1c.2 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
04:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller


root@box:~ # pciconf -l
hostb0@pci0:0:0:0: class=0x060000 card=0x191f1849 chip=0x191f8086 rev=0x07 hdr=0x00
vgapci0@pci0:0:2:0: class=0x030000 card=0x19121849 chip=0x19128086 rev=0x06 hdr=0x00
xhci0@pci0:0:20:0: class=0x0c0330 card=0xa12f1849 chip=0xa12f8086 rev=0x31 hdr=0x00
none0@pci0:0:20:2: class=0x118000 card=0xa1311849 chip=0xa1318086 rev=0x31 hdr=0x00
none1@pci0:0:22:0: class=0x078000 card=0xa13a1849 chip=0xa13a8086 rev=0x31 hdr=0x00
ahci0@pci0:0:23:0: class=0x010601 card=0xa1021849 chip=0xa1028086 rev=0x31 hdr=0x00
pcib1@pci0:0:27:0: class=0x060400 card=0xa1671849 chip=0xa1678086 rev=0xf1 hdr=0x01
pcib2@pci0:0:27:2: class=0x060400 card=0xa1691849 chip=0xa1698086 rev=0xf1 hdr=0x01
pcib3@pci0:0:28:0: class=0x060400 card=0xa1101849 chip=0xa1108086 rev=0xf1 hdr=0x01
pcib4@pci0:0:28:2: class=0x060400 card=0xa1121849 chip=0xa1128086 rev=0xf1 hdr=0x01
pcib5@pci0:0:29:0: class=0x060400 card=0xa1181849 chip=0xa1188086 rev=0xf1 hdr=0x01
isab0@pci0:0:31:0: class=0x060100 card=0xa1451849 chip=0xa1458086 rev=0x31 hdr=0x00
none2@pci0:0:31:2: class=0x058000 card=0xa1211849 chip=0xa1218086 rev=0x31 hdr=0x00
none3@pci0:0:31:3: class=0x040300 card=0x11511849 chip=0xa1708086 rev=0x31 hdr=0x00
none4@pci0:0:31:4: class=0x0c0500 card=0xa1231849 chip=0xa1238086 rev=0x31 hdr=0x00
em0@pci0:0:31:6: class=0x020000 card=0x15b81849 chip=0x15b88086 rev=0x31 hdr=0x00
ahci1@pci0:2:0:0: class=0x010601 card=0x06121849 chip=0x06121b21 rev=0x02 hdr=0x00
xhci1@pci0:4:0:0: class=0x0c0330 card=0x12421849 chip=0x12421b21 rev=0x00 hdr=0x00


 

[kdragon75]

Well you have 32GB of RAM and according to that graph your using 44GB. You're swapping 12GB... you may want to try setting your zfs.arc.max to something around 12GB. Also if you can afford it, remove one of your vCPUs. You CPU seems a bit over subscribed and I don't mean CPU use percent but ready time could be exceedingly high. I doubt the CPU oversubscription would cause a crash but everything may be a bit more responsive with fewer vCPUs. Keep in mind your effectively trying to schedule 2 dual core systems and a quad core with only 4 cores and that quad core is doing a lot of things for ZFS...

 

[Wouter]

Well you have 32GB of RAM and according to that graph your using 44GB. You're swapping 12GB... you may want to try setting your zfs.arc.max to something around 12GB. Also if you can afford it, remove one of your vCPUs. You CPU seems a bit over subscribed and I don't mean CPU use percent but ready time could be exceedingly high. I doubt the CPU oversubscription would cause a crash but everything may be a bit more responsive with fewer vCPUs. Keep in mind your effectively trying to schedule 2 dual core systems and a quad core with only 4 cores and that quad core is doing a lot of things for ZFS...

Thank you for the advise. I will do this, when the 11.2 branch is real stable for me.

I ran 11.2 beta2 now for about 6 hours, with no VM's running and it still suddenly freezes. No logging whatsever on the monitor nor syslog. So, I rebooted into 11.1u5 again and wait for 11.2 beta3 or later. Sad face.. because I wanted 11.2 to reinstall the VM's: Ubuntu 17.10 is not supported anymore :(.

Anyway: thanks a lot for thinking with me on this matter!