I am struggling with rancheros.
It seems to startup judging from the serial console (from VM / rancher / serial).
However usually would show a rancheros login screen and a prompt.
The login prompt does not appear not can I ssh into rancheros.
The container are not started and it seems that rancheros is stuck somewhere during startup.
This issue started when I changed the NFS. However I cannot see if this is causing the issue.
Can somebody help me find why I cannot ssh into rancheros nor access it via the serial console?
Rancheros startup log
Rancher 1.5.4, kernel version vmlinuz-4.14.138-rancher
Freenas 11.2u7, 64 GB RAM, Intel Xeon D-1521
It seems to startup judging from the serial console (from VM / rancher / serial).
However usually would show a rancheros login screen and a prompt.
The login prompt does not appear not can I ssh into rancheros.
The container are not started and it seems that rancheros is stuck somewhere during startup.
This issue started when I changed the NFS. However I cannot see if this is causing the issue.
Can somebody help me find why I cannot ssh into rancheros nor access it via the serial console?
Rancheros startup log
Code:
[ 1.890448] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.891717] ata1.00: ATA-9: BHYVE SATA DISK, 001, max UDMA/133 [ 1.892805] ata1.00: 41943040 sectors, multi 128: LBA48 NCQ (depth 31/32) [ 1.894248] ata1.00: configured for UDMA/133 [ 1.895262] scsi 0:0:0:0: Direct-Access ATA BHYVE SATA DISK 001 PQ: 0 ANSI: 5 [ 1.896973] sd 0:0:0:0: [sda] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB) [ 1.898581] sd 0:0:0:0: [sda] 131072-byte physical blocks [ 1.899777] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 1.900820] sd 0:0:0:0: [sda] Write Protect is off [ 1.901677] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.903804] sda: sda1 [ 1.904618] sd 0:0:0:0: [sda] Attached SCSI disk [ 2.417528] Freeing unused kernel memory: 1596K [ 2.433112] Write protecting the kernel read-only data: 16384k [ 2.435968] Freeing unused kernel memory: 2020K [ 2.437467] Freeing unused kernel memory: 228K [ ] init:info: [1/23] Starting set env WITH NIL cfg [ ] init:info: [2/23] Starting preparefs WITH NIL cfg [ ] init:info: [3/23] Starting save init cmdline WITH NIL cfg [ ] init:info: [4/23] Starting mount OEM [ ] init:info: [5/23] Starting debug save cfg [ 2.496768] tsc: Refined TSC clocksource calibration: 2398.245 MHz [ 2.497860] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2291bda8f41, max_idle_ns: 440795255703 ns [ ] init:info: [6/23] Starting load modules [ ] init:info: [7/23] Starting recovery console [ ] init:info: [8/23] Starting b2d env [ ] init:info: [9/23] Starting mount STATE and bootstrap [ ] init:info: Mounting state device /dev/sda1 to /state [ 2.579312] random: fast init done [ 3.524390] clocksource: Switched to clocksource tsc [ 5.610598] EXT4-fs (sda1): 14 orphan inodes deleted [ 5.611578] EXT4-fs (sda1): recovery complete [ 10.484990] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts:(null) [ ] init:info: [10/23] Starting cloud-init [ ] init:info: ros init: Detected Hypervisor: bhyve [ ] init:info: Configdrive was enabled but has no configdrive deviceor filesystem, ignore cloudinit [ ] init:info: [11/23] Starting read cfg and log files [ ] init:info: [12/23] Starting switchroot [ ] init:info: [13/23] Starting mount OEM2 [ ] init:info: [14/23] Starting mount BOOT [ ] init:info: [15/23] Starting write cfg and log files [ ] init:info: Swizzle: Wrote file to /var/lib/rancher/conf/cloud-config.d/init.yml [ ] init:info: Swizzle: Wrote file to /var/log/boot/init.log [ ] init:info: [16/23] Starting b2d Env [ ] init:info: [17/23] Starting hypervisor tools [ ] init:info: no hypervisor matched [ ] init:info: [18/23] Starting preparefs2 [ ] init:info: [19/23] Starting load modules2 [ ] init:info: [20/23] Starting set proxy env [ ] init:info: [21/23] Starting init SELinux SELinux: Could not load policy file /etc/selinux/ros/policy/policy.29: Read-only file system [ ] init:info: [22/23] Starting setupSharedRoot [ ] init:info: [23/23] Starting sysinit [ ] init:info: Launching System Docker [ 10.671842] random: system-dockerd: uninitialized urandom read (4096 bytes read) [ 10.739313] random: ros: uninitialized urandom read (4096 bytes read) [ ] ros-sysinit:info: Resolv.conf == [# Generated by dhcpcd from eth0.dhcp # /etc/resolv.conf.head can replace this line domain slewe.com nameserver 192.168.178.1 # /etc/resolv.conf.tail can replace this line ], <nil> [ ] ros-sysinit:info: Exec [/usr/bin/ros-sysinit] [ ] ros-sysinit:info: [1/4] Starting loadSystemImages [ ] ros-sysinit:info: Skipped loading system images because /var/lib/rancher/.sysimages_%s_loaded.done exists [ ] ros-sysinit:info: [2/4] Starting start project [ ] ros-sysinit:info: Project [os]: Starting project [ 16.983333] random: crng init done [ 139.780338] bridge: filtering via arp/ip/ip6tables is no longer available bydefault. Update your scripts to load br_netfilter if you need this. [ 139.784142] Bridge firewalling registered [ 139.794279] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [ 139.805322] ip_tables: (C) 2000-2006 Netfilter Core Team [ 139.982692] Initializing XFRM netlink socket [ 139.987640] Netfilter messages via NETLINK v0.30. [ 139.990524] ctnetlink v0.93: registering with nfnetlink. [ 140.148987] IPv6: ADDRCONF(NETDEV_UP): docker-sys: link is not ready [ ] ros-sysinit:info: [0/20] [command-volumes]: Starting [ ] ros-sysinit:info: [0/20] [system-volumes]: Starting [ ] ros-sysinit:info: [0/20] [container-data-volumes]: Starting [ ] ros-sysinit:info: [0/20] [media-volumes]: Starting [ ] ros-sysinit:info: [0/20] [user-volumes]: Starting [ ] ros-sysinit:info: [1/20] [system-volumes]: Started [ ] ros-sysinit:info: [2/20] [container-data-volumes]: Started [ ] ros-sysinit:info: [3/20] [command-volumes]: Started [ ] ros-sysinit:info: [3/20] [udev-cold]: Starting [ ] ros-sysinit:info: [3/20] [logrotate]: Starting [ ] ros-sysinit:info: [3/20] [syslog]: Starting [ ] ros-sysinit:info: [3/20] [acpid]: Starting [ ] ros-sysinit:info: [4/20] [user-volumes]: Started [ ] ros-sysinit:info: [5/20] [media-volumes]: Started [ ] ros-sysinit:info: [5/20] [all-volumes]: Starting [ ] ros-sysinit:info: [6/20] [logrotate]: Started [ ] ros-sysinit:info: [6/20] [system-cron]: Starting [ ] ros-sysinit:info: Rebuilding all-volumes [ 287.930195] cgroup: system-docker-r (427) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 287.932964] cgroup: "memory" requires setting use_hierarchy to 1 on the root [ ] ros-sysinit:info: [7/20] [syslog]: Started [ ] ros-sysinit:info: [8/20] [system-cron]: Started [ ] ros-sysinit:info: [9/20] [acpid]: Started acpid: starting up with netlink and the input layer acpid: 2 rules loaded acpid: waiting for events: event logging is off [ 312.163946] udevd[21]: starting version 3.2.5 [ 312.171915] udevd[22]: starting eudev-3.2.5 [ 314.502720] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/input/input3 [ 317.700535] AVX2 version of gcm_enc/dec engaged. [ 317.701499] AES CTR mode by8 optimization enabled [ 322.532760] EDAC sbridge: Ver: 1.1.2 [ ] ros-sysinit:info: [10/20] [all-volumes]: Started [ ] ros-sysinit:info: [11/20] [udev-cold]: Started [ ] ros-sysinit:info: [11/20] [udev]: Starting [ 352.401159] udevd[1]: starting version 3.2.5 [ 352.403477] udevd[1]: starting eudev-3.2.5 [ ] ros-sysinit:info: [12/20] [udev]: Started [ ] ros-sysinit:info: [12/20] [network]: Starting [ ] ros-sysinit:info: [13/20] [network]: Started [ ] ros-sysinit:info: [13/20] [ntp]: Starting [ 369.571416] overlayfs: upperdir is in-use by another mount, accessing files from both mounts will result in undefined behavior. [ 369.573353] overlayfs: workdir is in-use by another mount, accessing files from both mounts will result in undefined behavior. [ ] ros-sysinit:info: [14/20] [ntp]: Started [ ] ros-sysinit:info: [14/20] [cloud-init-execute]: Starting [ 391.372215] overlayfs: upperdir is in-use by another mount, accessing files from both mounts will result in undefined behavior. [ 391.375264] overlayfs: workdir is in-use by another mount, accessing files from both mounts will result in undefined behavior. [ ] ros-sysinit:info: [15/20] [cloud-init-execute]: Started [ ] ros-sysinit:info: [15/20] [console]: Starting [ ] ros-sysinit:info: Rebuilding console [ ] ros-sysinit:info: [16/20] [console]: Started [ ] ros-sysinit:info: [16/20] [subscriber]: Starting [ ] ros-sysinit:info: [16/20] [docker]: Starting [ ] ros-sysinit:info: [16/20] [preload-user-images]: Starting [ ] ros-sysinit:info: [17/20] [subscriber]: Started [ ] ros-sysinit:info: [18/20] [docker]: Started [ ] ros-sysinit:info: [19/20] [preload-user-images]: Started [ ] ros-sysinit:info: [19/20] [nfs]: Starting [ ] ros-sysinit:info: [20/20] [nfs]: Started [ ] ros-sysinit:info: Project [os]: Project started [ ] ros-sysinit:info: [3/4] Starting sync [ ] ros-sysinit:info: [4/4] Starting banner [ ] ros-sysinit:info: RancherOS v1.5.4 started
Rancher 1.5.4, kernel version vmlinuz-4.14.138-rancher
Freenas 11.2u7, 64 GB RAM, Intel Xeon D-1521
