Why systemd is not my friend

For a longer period of time I’m using OpenVZ for container virtualization and Xen for (para)virtualization. I wanted to try something new with KVM. I choosed Proxmox 4, which offers KVM and lxc as container solution. After playing around with KVM, I decided to migrate a live container from OpenVZ to Proxmox yesterday. The container has been migrated from one OpenVZ instance to another before and I upgraded from Debian Wheezy to Jessie. That means the container runs systemd before migrating from OpenVZ to Proxmox 4 / lxc. I needed to add three network interfaces to the container to add three IPv4 and three IPv6 adresses for the three services that run on this host. Later the container network died and the problems started.

Which problems did I see?

The container start required several minutes (2 minutes 20 seconds) before SSH or Proxmox console is accessible. top shows some processes with high CPU usage. I confirmed on the Proxmox host that the processes consuming the CPU are from the container.

top - 14:09:59 up 21 min, 2 users, load average: 2.41, 1.98, 1.74
Tasks: 27 total, 4 running, 23 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.2 us, 1.0 sy, 0.0 ni, 96.6 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 1048576 total, 1048532 used, 44 free, 0 buffers
KiB Swap: 262144 total, 56 used, 262088 free. 960492 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
796 message+ 20 0 43624 4808 2996 R 64.1 0.5 11:52.25 dbus-daemon
1 root 20 0 30100 6192 3028 R 52.4 0.6 11:36.41 systemd
232 root 20 0 254928 141956 141684 R 46.6 13.5 7:32.45 systemd-journal
749 root 20 0 19856 2532 2280 S 23.3 0.2 4:52.75 systemd-logind
833 root 20 0 258672 4080 2772 S 23.3 0.4 3:42.33 rsyslogd

The logs show the following:

journalctl -n 10
-- Logs begin at Tue 2016-05-10 13:48:46 CEST, end at Tue 2016-05-10 14:11:17 CEST. --
May 10 14:11:17 nl systemd[1]: Listening on udev Kernel Socket.
May 10 14:11:17 nl systemd[1]: Listening on udev Control Socket.
May 10 14:11:17 nl systemd[1]: Started udev Kernel Device Manager.
May 10 14:11:17 nl systemd[1]: Listening on udev Kernel Socket.
May 10 14:11:17 nl systemd[1]: Listening on udev Control Socket.
May 10 14:11:17 nl systemd[1]: Started udev Kernel Device Manager.
May 10 14:11:17 nl systemd[1]: Listening on udev Kernel Socket.
May 10 14:11:17 nl systemd[1]: Listening on udev Control Socket.
May 10 14:11:17 nl systemd[1]: Started udev Kernel Device Manager.
May 10 14:11:17 nl systemd[1]: Listening on udev Kernel Socket.

It appears to be a problem with systemd and/or udev. Wait. udev in a lxc container? Perhaps that’s not supposed to work. So I tried to remove it.

root@nl:~# apt-get remove udev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
sysvinit-core
The following packages will be REMOVED:
systemd systemd-sysv udev
The following NEW packages will be installed:
sysvinit-core
0 upgraded, 1 newly installed, 3 to remove and 0 not upgraded.
Need to get 0 B/132 kB of archives.
After this operation, 20.2 MB disk space will be freed.
Do you want to continue? [Y/n]
Preconfiguring packages ...
dpkg: systemd-sysv: dependency problems, but removing anyway as you requested:
init depends on systemd-sysv | sysvinit-core | upstart; however:
Package systemd-sysv is to be removed.
Package sysvinit-core is not installed.
Package upstart is not installed.

(Reading database ... 20454 files and directories currently installed.)
Removing systemd-sysv (215-17+deb8u4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Selecting previously unselected package sysvinit-core.
(Reading database ... 20438 files and directories currently installed.)
Preparing to unpack .../sysvinit-core_2.88dsf-59_amd64.deb ...
Unpacking sysvinit-core (2.88dsf-59) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up sysvinit-core (2.88dsf-59) ...
Not restarting sysvinit
(Reading database ... 20461 files and directories currently installed.)
Removing systemd (215-17+deb8u4) ...
systemd is the active init system, please switch to another before removing systemd.
dpkg: error processing package systemd (--remove):
subprocess installed pre-removal script returned error exit status 1
Failed to stop lib-init-rw.mount: Unit lib-init-rw.mount not loaded.
addgroup: The group `systemd-journal' already exists as a system group. Exiting.
dpkg: udev: dependency problems, but removing anyway as you requested:
systemd depends on udev (>= 208-8).

Removing udev (215-17+deb8u4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Errors were encountered while processing:
systemd
E: Sub-process /usr/bin/dpkg returned an error code (1)

I see, the system is broken now. Fix it.

root@nl:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
udev
The following NEW packages will be installed:
udev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/876 kB of archives.
After this operation, 6193 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Preconfiguring packages ...
Selecting previously unselected package udev.
(Reading database ... 20380 files and directories currently installed.)
Preparing to unpack .../udev_215-17+deb8u4_amd64.deb ...
Unpacking udev (215-17+deb8u4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (215-17+deb8u4) ...
Setting up udev (215-17+deb8u4) ...
addgroup: The group `input' already exists as a system group. Exiting.
udev does not support containers, not started.
update-initramfs: deferring update (trigger activated)
insserv: warning: script 'wide-dhcpv6-client' missing LSB tags and overrides
insserv: warning: script 'wide-dhcpv6-client' missing LSB tags and overrides

Okay. Load is good now. Now reboot and see what happens. The container starts fast, as I expected. I’ve now SysV init instead of systemd. The problem with the high load is gone. As of udev is still installed after this procedure, the problem seems to be related to systemd.

I tried some advices I found on Google, but nothing fixed the problem with systemd. What did not help? I read much about using lxc.autodev = 1 and lxc.kmsg = 0. This did not help when putting it in /etc/lxc/default.conf. I also found this way a bit obscure as I would assume Proxmox to handle this stuff.

I was also able to reproduce the problem by switching again to systemd. I made the experience that another container needed a apt-get upgrade which updated the installed systemd and solved a strange behaviour there.

This articel might get some updates later, if I can solve this systemd problem.