skip to content
‹ All posts

By Riya Madaan · Updated

systemd service won't start: a working checklist

A systemd service failed to start - the checklist that finds why: systemctl status, journalctl -xeu, exit code 203, ExecStart paths, deps, and limits.

#linux

Job for app.service failed because the control process exited with error code. Thanks, systemd. When a systemd service failed to start, the actual reason is always recorded - but it's spread across three places (unit status, the journal, and the unit file), and the error you see first is frequently the least informative of the three. Here's the checklist, ordered by how often each step ends the investigation.

1. systemctl status - read all of it

systemctl status app.service

Don't just check the red/green. Read three specific fields:

  • Active: - failed (Result: exit-code) vs failed (Result: timeout) vs failed (Result: start-limit-hit) are three different investigations.
  • Process: - the exit status. status=1 is the app failing on its own terms; status=203/EXEC and friends are systemd failing to even launch it (see the table below).
  • The last log lines at the bottom - often the answer is already sitting there.

If you see start-limit-hit, the current failure is just systemd refusing to retry; the real error is a few restarts back:

systemctl reset-failed app.service   # clear the rate limit
journalctl -u app.service -n 200     # find the original failure

2. journalctl -xeu - the full story

journalctl -xeu app.service

-u filters to the unit, -e jumps to the end, -x adds systemd's explanatory text. This shows both systemd's view and the service's own stdout/stderr, interleaved in order - which is exactly what you need to tell "systemd couldn't start it" from "it started and died". For deeper time-based digging (what else happened on the box at that moment), the journalctl forensics toolkit applies.

3. What do systemd exit codes like 203/EXEC mean?

They mean systemd itself failed to launch the service, before your app ran a single line. Those exit codes land in the 200+ range. These are the ones you'll actually meet:

status Meaning Usual cause
203/EXEC Couldn't execute the binary Wrong ExecStart path, missing execute bit, missing interpreter/shebang
200/CHDIR Couldn't change directory WorkingDirectory= doesn't exist
217/USER Unknown user User= not present on this machine
216/GROUP Unknown group Same, for Group=
226/NAMESPACE Sandbox setup failed ProtectSystem=, ReadWritePaths= pointing at nonexistent paths
1, 2, … (< 200) The app itself exited Its own config/logs - debug the app, not systemd

203/EXEC deserves its own minute because it's the most common:

grep ExecStart /etc/systemd/system/app.service
ls -l /usr/local/bin/app        # exists? executable?
head -1 /usr/local/bin/app      # script? does the shebang interpreter exist?
file /usr/local/bin/app         # right architecture?

Remember ExecStart= needs an absolute path, and shell features (|, &&, $VAR expansion beyond ${VAR} semantics, globs) don't work - there is no shell unless you invoke one explicitly.

4. Why does it run from my shell but fail under systemd?

Because the service does not get your login environment. No ~/.bashrc, usually no PATH beyond the compiled-in default, no HOME you'd expect. The app that runs fine when you launch it by hand and dies under systemd almost always differs in environment, user, or working directory. Compare directly:

sudo -u appuser /usr/local/bin/app        # closest manual reproduction
systemctl show app.service -p Environment,EnvironmentFiles,User,WorkingDirectory

If the unit uses EnvironmentFile=, check the file exists and is parseable - and note that EnvironmentFile=-/etc/default/app (leading dash) silently ignores a missing file, which is how you get a service that starts but is misconfigured.

5. Dependencies and ordering

After= is ordering, Requires=/Wants= are dependency - a unit with After=network.target but nothing else can still start before the network is genuinely usable. For services that need actual connectivity at start, After=network-online.target + Wants=network-online.target is the documented pattern. Check what systemd thinks the relationships are:

systemctl list-dependencies app.service
systemd-analyze verify app.service        # catches typos, missing units, bad directives

Type= mismatches also live here: a Type=forking unit whose binary doesn't fork will hang until the start timeout and be killed (Result: timeout); a Type=notify unit whose binary never calls sd_notify does the same. When in doubt, Type=simple (or exec) and let the process run in the foreground.

6. The classics: ports, permissions, limits

  • Port already taken: the log says bind: address already in use. Finding the squatter is its own short story - who owns this port.
  • Permissions: the service runs as User=app, and the config/socket/state directory is root-owned. Check every path the app touches as that user. Modern units add sandbox directives (ProtectSystem=strict, PrivateTmp=) that produce read-only-filesystem errors in unexpected places - ReadWritePaths= is the fix, not loosening file modes.
  • Resource limits: LimitNOFILE= too low shows up as too many open files at startup for daemons that open many sockets early.

7. After editing: reload, restart, verify

sudo systemctl daemon-reload      # required after ANY unit file edit
sudo systemctl restart app.service
systemctl is-active app.service && journalctl -u app.service -n 20

Forgetting daemon-reload means you're testing the old unit file - a genuinely cruel way to lose twenty minutes. And verification means more than is-active: confirm the service actually answers (curl, a health endpoint, a client query). "Green in systemctl" and "working" overlap less than you'd hope. This exact scenario - a unit failing for a reason buried two layers down - is a staple of Linux interview scenarios for the same reason it's on this checklist: it separates people who read the system's evidence from people who restart things and hope. There's no Linux lab for it yet, but the practice labs drill the same read-the-evidence-first loop on a broken Kubernetes cluster.

FAQ

What does status=203/EXEC mean? systemd could not execute the ExecStart binary: wrong path, not executable, missing shebang interpreter, or wrong architecture. It never means the app ran and failed - the app never ran.

Why does my service work manually but fail under systemd? Different environment, user, working directory, or sandboxing. Reproduce with sudo -u <serviceuser> <exact ExecStart command> and compare systemctl show -p Environment,User,WorkingDirectory against your shell.

What is start-limit-hit and how do I clear it? The unit crashed too many times too fast, so systemd stopped retrying. systemctl reset-failed <unit> clears the counter - but the original crash reason is still in journalctl -u <unit>, and clearing the limit without reading it just schedules the next failure.

Do I need daemon-reload after editing a unit file? Yes, always. systemd caches unit definitions; without systemctl daemon-reload your edits are invisible, including fixes that would have worked.