Background service doesn't reliably start at boot

[mats-olov_rustad]

My company have ported a logger application from a proprietary RTOS to Linux and Kunbus Revolution Pi Connect+. The development was initially done on 2 Raspberry Pi 4 Model B 2 GB and 4 GB. We added System V daemonization to the Linux application to be able to automatically start it in the background as a daemon at system boot. We can reliably start and stop the application from a shell prompt. The application doesn't start reliably at system start. Sometimes it starts but most often we have to manually start it from a shell prompt with the command sudo systemctl start loggerd.service . We have written the following definition file:

[Unit]
Description=Sarsys-ASFT data logger daemon
Requires=systemd-networkd-wait-online.service rsyslog.service time-sync.target
After=systemd-networkd-wait-online.service rsyslog.service time-sync.target
ConditionPathExists=!/etc/logger.d/loggerd_not_to_be_run

[Service]
User=pi
Group=pi
SupplementaryGroups=dialout
DynamicUser=no
RuntimeDirectory=logger.d
RuntimeDirectoryMode=0755
RuntimeDirectoryPreserve=yes
ConfigurationDirectory=logger.d
ConfigurationDirectoryMode=0755
NoNewPrivileges=yes
AmbientCapabilities=CAP_SYS_BOOT CAP_SYS_TIME
CapabilityBoundingSet=CAP_SYS_BOOT CAP_SYS_TIME
SecureBits=keep-caps
KillMode=process
Type=forking
RemainAfterExit=no
GuessMainPID=yes
PIDFile=/run/logger.d/loggerd.pid
ExecStart=/usr/local/bin/loggerd --group pi,dialout --user $USER --runtime /run/logger.d --config /etc/logger.d --daemon
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
RestartSec=5
TimeoutStartSec=10
TimeoutStopSec=30
WatchdogSec=0
Restart=on-failure
SuccessExitStatus=2 3 4 5 6 9 10
RestartPreventExitStatus=2 3 4 5 6 9 10 SIGABRT
NotifyAccess=main

[Install]
WantedBy=multi-user.target
Alias=loggerd.service

Our application is dependent upon correct system time and logging hence the requirement for rsyslog.service and time-sync.target after the keyword Requires. The application also opens two TCP server sockets and that is why systemd-networkd-wait-online.service also is specified after the Requires keyword. I used the command sudo systemctl enable systemd-networkd-wait-online.service systemd-networkd.service ebfore the first attempt to start with our logger service. I have googled this problem and most of the answers I have found says that the service needs to be enabled which it is according to systemctl status loggerd.service.

What are we doing wrong?

[mats-olov_rustad]

I found and fixed the problem. The unit definition file loggerd.service did not specify the keyword DefaultDependencies which means it assumed the systemd default value of yes. The explicit stated dependencies caused an implicit dependency on sockets.target and basic.target. Unfortunately sockets.target is started before basic.target while our service is started after both sockets.target and basic.target, This causes a start ordering loop which makes systemd to seemingly randomly delete the start of services such as our loggerd.service.

The fault was found with the command 'systemd-analyze verify default.target' which showed how the competing requirements of loggerd.service and sockets.target and it also show how sometimes thet start of loggerd.service was deleted and sometimes how the start of another service was deleted. This explains the seemingly random failures of loggerd.service to start automatically.

I watched the real dependencies of loggerd.service with the command 'systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After loggerd.service'. I compared it to the real dependencies of sockets.target with the command 'systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After sockets.target'.

I added the keyword DefaultDependencies with the value no and explicitly added the dependencies except for sockets.target. This fixed the start ordering loop according to systemd-analyze verify och the service have now always started on every system reboot.

With regards,