How to debug a failing service with systemd

Diagnose and fix a systemd service that fails to start using systemctl status, journalctl, and systemd-analyze.

How to debug a failing service with systemd

Diagnose and fix a systemd service that fails to start using systemctl status, journalctl, and systemd-analyze.

Prerequisites

  • Root or sudo access on a Linux system running systemd.
  • A service that fails to start or crashes after starting.

Step-by-Step: Debug a Failing systemd Service

  1. Check the service status with systemctl status. systemd displays the loaded state, active state, exit code, and the last few log lines:

    sudo systemctl status myapp.service

    Look for Active: failed and the Main PID exit status. The exit code often indicates the cause: status=203/EXEC means the binary was not found, status=217/USER means the specified user does not exist.

  2. Read the full journal output for the service. journalctl shows every log message the service produced, including startup errors:

    sudo journalctl -u myapp.service --no-pager -n 50

    The -n 50 flag shows the last 50 lines. Remove it to see the complete log.

  3. Verify the unit file syntax with systemd-analyze verify. This command checks for configuration errors without starting the service:

    sudo systemd-analyze verify /etc/systemd/system/myapp.service
  4. Test the ExecStart command manually. Run the exact command from the unit file as the specified user to see if it works outside systemd:

    sudo -u myapp /opt/myapp/bin/myapp --config /etc/myapp/config.yaml
  5. Check for dependency issues. systemd may fail a service if a required dependency is not available:

    systemctl list-dependencies myapp.service

How to Verify the Fix

After correcting the issue, reload and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service
sudo systemctl status myapp.service

The status output should show Active: active (running).