Unable to run apps on Baremetal SCALE

jhbriese · Sep 15, 2021

Hello,

I´m trying to run app on my Testserver (HP microserver gen8, Xeon E3-1220L V2, 16 GB ECC RAM, 4x2TB SATA HDD). Deployment stopping with followinfg Application Events:

Code:

2021-09-15 14:11:40
Scaled up replica set nextcloud-56f4755469 to 1
Successfully assigned ix-nextcloud/nextcloud-56f4755469-289xz to ix-truenas
2021-09-15 14:11:40
Scaled up replica set nextcloud-postgres-5bf49b87bc to 1
Successfully assigned ix-nextcloud/nextcloud-postgres-5bf49b87bc-vq9l4 to ix-truenas
2021-09-15 14:11:40
Created pod: nextcloud-postgres-5bf49b87bc-vq9l4
2021-09-15 14:11:40
Created pod: nextcloud-56f4755469-289xz
2021-09-15 14:12:41
Add eth0 [172.16.0.4/16] from ix-net
2021-09-15 14:12:42
Add eth0 [172.16.0.5/16] from ix-net
2021-09-15 14:15:54
Failed to pull image "postgres:13.1": rpc error: code = Unknown desc = context canceled
2021-09-15 14:15:54
Error: ErrImagePull
2021-09-15 14:15:54
Back-off pulling image "postgres:13.1"
2021-09-15 14:15:54
Error: ImagePullBackOff
2021-09-15 14:12:42
Pulling image "postgres:13.1"
2021-09-15 14:14:13
Failed to pull image "postgres:13.1": rpc error: code = Unknown desc = context canceled
2021-09-15 14:14:13
Error: ErrImagePull
2021-09-15 14:14:14
Back-off pulling image "postgres:13.1"
2021-09-15 14:14:14
Error: ImagePullBackOff
2021-09-15 14:12:41
Pulling image "postgres:13.1"
2021-09-15 14:18:24
Marking for deletion Pod ix-nextcloud/nextcloud-postgres-5bf49b87bc-vq9l4
2021-09-15 14:18:24
Created pod: nextcloud-postgres-5bf49b87bc-jjtwf
2021-09-15 14:18:24
Marking for deletion Pod ix-nextcloud/nextcloud-56f4755469-289xz
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
2021-09-15 14:18:24
Created pod: nextcloud-56f4755469-7tmdc
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
0/1 nodes are available: 1 node(s) had taint {ix-svc-stop: }, that the pod didn't tolerate.
Successfully assigned ix-nextcloud/nextcloud-postgres-5bf49b87bc-jjtwf to ix-truenas
Successfully assigned ix-nextcloud/nextcloud-56f4755469-7tmdc to ix-truenas
2021-09-15 14:19:48
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2021-09-15 14:19:48
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2021-09-15 14:21:09
Add eth0 [172.16.0.6/16] from ix-net
2021-09-15 14:21:09
Pulling image "postgres:13.1"
2021-09-15 14:21:38
Add eth0 [172.16.0.8/16] from ix-net
2021-09-15 14:21:38
Pulling image "postgres:13.1"

When I inspect k3 via Shell:
k3s kubectl describe nodes

Code:

Name:               ix-truenas
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ix-truenas
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    openebs.io/nodeid=ix-truenas
                    openebs.io/nodename=ix-truenas
Annotations:        k3s.io/node-args:
                      ["server","--flannel-backend","none","--disable","traefik,metrics-server,local-storage","--disable-kube-proxy","--disable-network-policy",...
                    k3s.io/node-config-hash: UJ34GDNYEMW3NSZ4LZKLRC7VM4FTVWSO5HSUJ2MHWN6ETIGVOSLA====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/pool0/ix-applications/k3s/data/b923ec15745249e9ba73deb8dc22f52cc5b7615239be700fbc14a9e6d80314aa"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 15 Sep 2021 13:56:26 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  ix-truenas
  AcquireTime:     <unset>
  RenewTime:       Wed, 15 Sep 2021 14:25:07 +0200
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 15 Sep 2021 14:24:57 +0200   Wed, 15 Sep 2021 13:56:24 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 15 Sep 2021 14:24:57 +0200   Wed, 15 Sep 2021 13:56:24 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 15 Sep 2021 14:24:57 +0200   Wed, 15 Sep 2021 13:56:24 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 15 Sep 2021 14:24:57 +0200   Wed, 15 Sep 2021 14:19:55 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.188.180
  Hostname:    ix-truenas
Capacity:
  cpu:                4
  ephemeral-storage:  96491008Ki
  hugepages-2Mi:      0
  memory:             16340904Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  93866452509
  hugepages-2Mi:      0
  memory:             16340904Ki
  pods:               110
System Info:
  Machine ID:                 004d150100b3470eae69e7fbb33f2da2
  System UUID:                31393138-3538-5a43-3136-323030305859
  Boot ID:                    bf5415d8-89b2-4030-94f2-4cbe38f75423
  Kernel Version:             5.10.42+truenas
  OS Image:                   Debian GNU/Linux 11 (bullseye)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.8
  Kubelet Version:            v1.21.0-k3s1
  Kube-Proxy Version:         v1.21.0-k3s1
PodCIDR:                      172.16.0.0/16
PodCIDRs:                     172.16.0.0/16
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                   ------------  ----------  ---------------  -------------  ---
  kube-system                 openebs-zfs-controller-0               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  kube-system                 openebs-zfs-node-sltqw                 0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m17s
  ix-nextcloud                nextcloud-postgres-5bf49b87bc-jjtwf    0 (0%)        0 (0%)      0 (0%)           0 (0%)         6m47s
  kube-system                 coredns-7448499f4d-8brwb               100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     6m47s
  ix-nextcloud                nextcloud-56f4755469-7tmdc             0 (0%)        0 (0%)      0 (0%)           0 (0%)         6m47s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             70Mi (0%)  170Mi (1%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type    Reason                   Age                From     Message
  ----    ------                   ----               ----     -------
  Normal  Starting                 28m                kubelet  Starting kubelet.
  Normal  NodeHasSufficientMemory  28m (x2 over 28m)  kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    28m (x2 over 28m)  kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     28m (x2 over 28m)  kubelet  Node ix-truenas status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  28m                kubelet  Updated Node Allocatable limit across pods
  Normal  NodeReady                28m                kubelet  Node ix-truenas status is now: NodeReady
  Normal  Starting                 5m30s              kubelet  Starting kubelet.
  Normal  NodeHasSufficientMemory  5m30s              kubelet  Node ix-truenas status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    5m30s              kubelet  Node ix-truenas status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     5m30s              kubelet  Node ix-truenas status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  5m30s              kubelet  Updated Node Allocatable limit across pods
  Normal  NodeNotReady             5m27s              kubelet  Node ix-truenas status is now: NodeNotReady
  Normal  NodeReady                5m16s              kubelet  Node ix-truenas status is now: NodeReady

Any Ideas?

Thanks

dayrain · Sep 15, 2021

docker pull postgres image is fail
"network is not ready" ,check network setting
try manual pull
docker pull postgres:13.1

jhbriese · Sep 16, 2021

docker finally pulled the Image. I Think there is an bigger issue with CNI:


network plugin is not ready: cni config uninitialized

Somehow kubernetes will not initialize the Network. By inspecting the k3s log i found:

Code:

I0915 13:56:24.008934   16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
I0915 13:56:24.014181   16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
I0915 13:56:24.014282   16787 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
I0915 13:56:24.014657   16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"

There is no configuration in /etc/cni/net.d
Maybe ther is an issue with multiple NIC's connected to Host (only 1 active)?

darkcloud784 · Feb 1, 2022

jhbriese said:
docker finally pulled the Image. I Think there is an bigger issue with CNI: network plugin is not ready: cni config uninitialized

Somehow kubernetes will not initialize the Network. By inspecting the k3s log i found:

Code:
I0915 13:56:24.008934 16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d" I0915 13:56:24.014181 16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d" I0915 13:56:24.014282 16787 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni" I0915 13:56:24.014657 16787 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"

There is no configuration in /etc/cni/net.d
Maybe ther is an issue with multiple NIC's connected to Host (only 1 active)?

I have the same issue using LACP connected to a bridge.

mcowart · Apr 2, 2022

Same deal here; I have a primary LACP bridge of 2 x 1Gb along with a single 10g card that is directly attached to my main PC. Every time I reboot kubernetes throws a fit and hangs forever until I completely reconfigure the networking for it and restart it.

Now that I've added the Truecharts catalog it's behaving even worse in that I've yet to get it to start cleanly after an upgrade reboot.

Saberwolf · May 6, 2022

edit you app config and review your network settings and ports that is where you issue lies

mcowart · May 18, 2022

Saberwolf said:
edit you app config and review your network settings and ports that is where you issue lies

This is the single most helpful reply I've ever received. Every one of us having this issue obviously have our networks completely misconfigured.

Saberwolf · May 18, 2022

mcowart said:
This is the single most helpful reply I've ever received. Every one of us having this issue obviously have our networks completely misconfigured.

review you network config in the app your problem lies there. as for having all the answers some times you just have to mess around with your config to find out what is broken try to make config simpler. sorry you are having a hard time getting nextcloud to run

Saberwolf · May 18, 2022

as stated above just installed and used default setting and it starts and works

Code:

2022-05-18 20:14:04 Pulling image "nextcloud:24"
2022-05-18 20:14:04 Started container nginx
2022-05-18 20:14:04 Created container nginx
2022-05-18 20:14:01 Successfully pulled image "nginx:1.21.3" in 11.835188567s
2022-05-18 20:13:50 Pulling image "nginx:1.21.3"
2022-05-18 20:13:43 Started container nextcloud-postgres
2022-05-18 20:13:42 Created container nextcloud-postgres
2022-05-18 20:13:41 Started container init-postgresdb
2022-05-18 20:13:41 Created container init-postgresdb
2022-05-18 20:13:39 Successfully pulled image "postgres:13.1" in 25.464728376s
2022-05-18 20:13:37 Successfully pulled image "postgres:13.1" in 24.232560759s
2022-05-18 20:13:13 Pulling image "postgres:13.1"
2022-05-18 20:13:13 Add eth0 [172.15.0.56/16] from ix-net
2022-05-18 20:13:13 Pulling image "postgres:13.1"
2022-05-18 20:13:13 Add eth0 [172.15.0.55/16] from ix-net Successfully assigne ix-next/next-nextcloud-postgres-fbf8b5b5d-kf7ff to ix-truenas
2022-05-18 20:13:08 Created pod: next-nextcloud-postgres-fbf8b5b5d-kf7ff Successfully assigned ix-next/next-nextcloud-54c985fdb9-4fdtn to ix-truenas
2022-05-18 20:13:08 Scaled up replica set next-nextcloud-postgres-fbf8b5b5d to 1
2022-05-18 20:13:08 Created pod: next-nextcloud-54c985fdb9-4fdtn
2022-05-18 20:13:08 Scaled up replica set next-nextcloud-54c985fdb9 to 1

now unless you have messed with the network config in some way the only thing i changed was to use freenas_default cert.

other than that you have made setting change or configured you network in a non standard way and you need to fix that before it will work

Saberwolf · May 18, 2022

oh and it is not a double nic issue as i have two installed and working 1000 gig and 10 GIGe

Saberwolf · May 18, 2022

mcowart said:
Same deal here; I have a primary LACP bridge of 2 x 1Gb along with a single 10g card that is directly attached to my main PC. Every time I reboot kubernetes throws a fit and hangs forever until I completely reconfigure the networking for it and restart it.

Now that I've added the Truecharts catalog it's behaving even worse in that I've yet to get it to start cleanly after an upgrade reboot.

simple solution set a static ip to the unused nic's and try again it can be in the active network or out side as stated before you config is your issue

mcowart · Jun 1, 2022

Saberwolf said:
simple solution set a static ip to the unused nic's and try again it can be in the active network or out side as stated before you config is your issue

My network is literally working 100%. LACP is up on my 1Gb team and failover works flawlessly. The 10Gb network connection is working fine as well. All my VMs have 0 issues; I even have a pfSense VM as my primary firewall/router in an HA pair with a physical device with seemless failover working.

Are you running an active/active LACP team? If not you do not have the same setup I do. I have 3 total NICs.

My workaround was to just install docker into one of my VMs (I prefer docker-compose anyway for ease of use).

Important Announcement for the TrueNAS Community.

Unable to run apps on Baremetal SCALE

jhbriese

Cadet

dayrain

Cadet

jhbriese

Cadet

darkcloud784

Dabbler

mcowart

Cadet

Saberwolf

Explorer

mcowart

Cadet

Saberwolf

Explorer

Saberwolf

Explorer

Saberwolf

Explorer

Saberwolf

Explorer

mcowart

Cadet

Similar threads