Docker Containers crash (seemingly) randomly

[CookiesMonster]

• FreeNAS 11.2
• Motherboard make and model: Supermicro X9SRi-F
• CPU make and model: Intel Xeon E5-1650 3.5 GHz 6-Core
• RAM quantity: 128GB PC3-14900R
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
  • Array 1: Valhalla: 10x 8TB HDDs in a RAIDZ2
  • Array 2: Impulse: 2x 500GB SSDs in a RAID0
  • Array 3: Improbability: 8x 14TB HDDs in a RAIDZ2 in a NetAPP DS4246 Disk Shelf DAS
  • Array 4: Excelsior: 2x 1TB SSDs in a RAID0
  • Boot drives: 2x 16GB USB drives in a RAID1
• Hard disk controllers:
  • LSI SAS2008 9211-8i HBA
  • LSI SAS9207-8e HBA (This connects to the NetAPP Disk Shelf DAS)
  • LSI 9210-8i HBA
• Network cards: Only using the 2x onboard ethernet adapters

Hello!

I built my FreeNAS machine around a year ago and have been expanding its capabilities recently. I have RancherOS installed to run Docker containers. I've had various issues with my containers but most of it has been worked out. There are a few that still give me problems but they work most of the time.

Recently (for the last month or so), my containers will (seemingly) randomly crash. I have some that are launched via YML files inside the VM itself but most go through Portainer for convenience. At first I thought maybe the issue was Portainer but even the containers that don't go through Portainer crashed as well, so it seems like something at a higher level.

The issue is strange because, as I said before, there doesn't seem to be a catalyst for it but it also seems to fix itself after 20 minutes to a couple hours.

I'm not sure if this information is pertinent, but originally I was following a guide to set up RancherOS so I only gave it 2 virtual CPUs and 4GB of RAM. After getting my containers set up, I noticed everything was running slowly so I decided to try and increase the virtual CPUs/RAM for the VM. At first I tried to set it to 4 virtual CPUs and 16GB of RAM (I have 128GB so that shouldn't be an issue), but the VM wouldn't even launch. It gave me some over-allocation error. Next I tried to set it to 4 virtual CPUs and 8GB of RAM. When I first restarted it, the VM wouldn't work but after another restart it launched fine. I only mention this in case I did something incorrectly and messed up the VM.

I'm really not sure where to start troubleshooting this so any help would be appreciated!

 

[Samuel Tai]

The default Docker VM doesn't have a swap partition, so will crash when it runs out of memory. Try attaching another RAW file (it can be empty) to the VM as a 2nd virtual disk, and then inside RancherOS, use fdisk to create a partition table (MBR or GPT, doesn't matter), and then a swap partition labeled RANCHER_SWAP. Then, using the RancherOS manual (https://rancher.com/docs/os/v1.x/en/storage/additional-mounts/), create the swap mount via ros config set mounts '[["/dev/sdb1","","swap",""]]', which will take effect on the next start of the VM.

 

[CookiesMonster]

Thanks for your reply!

I noticed you said "Try attaching another RAW file".

I actually have my storage mapped into my VM (and my dockers) through 6 NFS shares that are mounted inside the VM. I'm not sure if that matters or not.

 

[Samuel Tai]

No, it's not possible to use NFS for swap. Sorry, I meant to type ros config set mounts '[["LABEL=RANCHER_SWAP","","swap",""]]'.

 

[CookiesMonster]

A few questions:

1) From your response, it seems you think the issue is that the VM is running out of memory and that is the reason for it crashing. How much RAM can I give RancherOS? As I said earlier, I have 128GB of RAM and the majority of that is rarely used. Also I mentioned how I tried to give it 16GB and it wouldn't even launch. Any idea why that happened?

2) "No, it's not possible to use NFS for swap. " Is there a better way to map my storage into my VM? When I initially set this up, the guide I was following used NFS mounts but I'm open to changing it to something else if it increases performance/reliability.

Thanks!

 

[Samuel Tai]

Even if you give the RancherOS VM an insane amount of memory, its memory management architecture pretty much requires a swap partition somewhere eventually.

NFS is generally the best method for adding external storage for Docker volumes; in this case, it's the OS itself that needs a local swap partition.

 

[CookiesMonster]

Are your earlier instructions for creating the local swap partition or do I need to use another method?

Thanks!

 

[Samuel Tai]

Yes, I provided instructions for creating a local swap partition. You can just copy the current RAW file, and give it priority 1004, or whatever's next in the device order.

 

[CookiesMonster]

I copied the RAW file I had already created (100GB) and then added it with the following settings:
Sector Size: Default
Mode: AHCI
Device Order: 1003
Raw filesize: 100GB

I'm not exactly sure what to do with fdisk at this point.

Can you elaborate?

Thanks!

 

[Samuel Tai]

1. fdisk /dev/sdb
2. Type m to bring up the fdisk command menu.
3. Type g to create a new GPT partition table.
4. Type n to create a new partition. In your case, you want 19, a Linux swap partition. It doesn't need to fill the entire RAW file. Recommended size is either the same as the virtual RAM allocated to the VM, or twice that.
5. Type w to save the new partition table and swap partition.
6. Now, use mkswap -L RANCHER_SWAP /dev/sdb1 to format the new swap partition, and assign it the correct label.

 

[CookiesMonster]

I don't have a sdb drive, but I do have sda and sda1.

Is one of them the one I'm looking for?

 

[Samuel Tai]

Did you restart the VM after attaching the new RAW file?

 

[CookiesMonster]

Yes. Still just sda and sda1

I did notice that when I look at "Details" for the new RAW file, boot is set to "false", whereas the main RAW file is set to "true".

Is that correct?

Also, I placed the new RAW file in the same location as the other one. I'm not sure if that was correct or not.

 

[Samuel Tai]

Yes, only your primary RAW file should have boot="true". What are the device orders of the other devices? There can't be a gap.

 

[CookiesMonster]

(See photo)

Device 3 and 4 were created when I set up the VM. Device 5 is the new one I added.

 

[Samuel Tai]

What version of RancherOS are you running? What does ros os list show as the running version?

 

[CookiesMonster]

Here is the output.

 

[Samuel Tai]

OK, try changing the device order. Shutdown the VM, and then make the boot RAW file 1000, the new RAW file 1001, and the NIC 1002.

 

[CookiesMonster]

1. fdisk /dev/sdb
2. Type m to bring up the fdisk command menu.
3. Type g to create a new GPT partition table.
4. Type n to create a new partition. In your case, you want 19, a Linux swap partition. It doesn't need to fill the entire RAW file. Recommended size is either the same as the virtual RAM allocated to the VM, or twice that.
5. Type w to save the new partition table and swap partition.
6. Now, use mkswap -L RANCHER_SWAP /dev/sdb1 to format the new swap partition, and assign it the correct label.

sdb and sdb1 are showing up now (see photo). I'm on Step 4 of your steps. It's asking for "First Sector". I'm not sure what that means. You mentioned the partition size but I just wanted to be sure.

 

[Samuel Tai]

These are the values I use for a 4G swap partition:

Start	End
2097152	10485759

This assumes 512-byte sectors. For an 8G swap partition, the end sector would be 14680064.