By Riya Madaan · Updated
systemd service won't start: a working checklist
A systemd service failed to start - the checklist that finds why: systemctl status, journalctl -xeu, exit code 203, ExecStart paths, deps, and limits.
Job for app.service failed because the control process exited with error code. Thanks,
systemd. When a systemd service failed to start, the actual reason is always recorded - but
it's spread across three places (unit status, the journal, and the unit file), and the error
you see first is frequently the least informative of the three. Here's the checklist, ordered
by how often each step ends the investigation.
1. systemctl status - read all of it
systemctl status app.service
Don't just check the red/green. Read three specific fields:
Active:-failed (Result: exit-code)vsfailed (Result: timeout)vsfailed (Result: start-limit-hit)are three different investigations.Process:- the exit status.status=1is the app failing on its own terms;status=203/EXECand friends are systemd failing to even launch it (see the table below).- The last log lines at the bottom - often the answer is already sitting there.
If you see start-limit-hit, the current failure is just systemd refusing to retry; the
real error is a few restarts back:
systemctl reset-failed app.service # clear the rate limit
journalctl -u app.service -n 200 # find the original failure
2. journalctl -xeu - the full story
journalctl -xeu app.service
-u filters to the unit, -e jumps to the end, -x adds systemd's explanatory text. This
shows both systemd's view and the service's own stdout/stderr, interleaved in order - which
is exactly what you need to tell "systemd couldn't start it" from "it started and died". For
deeper time-based digging (what else happened on the box at that moment), the
journalctl forensics toolkit applies.
3. What do systemd exit codes like 203/EXEC mean?
They mean systemd itself failed to launch the service, before your app ran a single line. Those exit codes land in the 200+ range. These are the ones you'll actually meet:
| status | Meaning | Usual cause |
|---|---|---|
203/EXEC |
Couldn't execute the binary | Wrong ExecStart path, missing execute bit, missing interpreter/shebang |
200/CHDIR |
Couldn't change directory | WorkingDirectory= doesn't exist |
217/USER |
Unknown user | User= not present on this machine |
216/GROUP |
Unknown group | Same, for Group= |
226/NAMESPACE |
Sandbox setup failed | ProtectSystem=, ReadWritePaths= pointing at nonexistent paths |
1, 2, … (< 200) |
The app itself exited | Its own config/logs - debug the app, not systemd |
203/EXEC deserves its own minute because it's the most common:
grep ExecStart /etc/systemd/system/app.service
ls -l /usr/local/bin/app # exists? executable?
head -1 /usr/local/bin/app # script? does the shebang interpreter exist?
file /usr/local/bin/app # right architecture?
Remember ExecStart= needs an absolute path, and shell features (|, &&, $VAR
expansion beyond ${VAR} semantics, globs) don't work - there is no shell unless you invoke
one explicitly.
4. Why does it run from my shell but fail under systemd?
Because the service does not get your login environment. No ~/.bashrc, usually no PATH beyond
the compiled-in default, no HOME you'd expect. The app that runs fine when you launch it by
hand and dies under systemd almost always differs in environment, user, or working directory.
Compare directly:
sudo -u appuser /usr/local/bin/app # closest manual reproduction
systemctl show app.service -p Environment,EnvironmentFiles,User,WorkingDirectory
If the unit uses EnvironmentFile=, check the file exists and is parseable - and note that
EnvironmentFile=-/etc/default/app (leading dash) silently ignores a missing file, which is
how you get a service that starts but is misconfigured.
5. Dependencies and ordering
After= is ordering, Requires=/Wants= are dependency - a unit with After=network.target
but nothing else can still start before the network is genuinely usable. For services that need
actual connectivity at start, After=network-online.target + Wants=network-online.target is
the documented pattern. Check what systemd thinks the relationships are:
systemctl list-dependencies app.service
systemd-analyze verify app.service # catches typos, missing units, bad directives
Type= mismatches also live here: a Type=forking unit whose binary doesn't fork will hang
until the start timeout and be killed (Result: timeout); a Type=notify unit whose binary
never calls sd_notify does the same. When in doubt, Type=simple (or exec) and let the
process run in the foreground.
6. The classics: ports, permissions, limits
- Port already taken: the log says
bind: address already in use. Finding the squatter is its own short story - who owns this port. - Permissions: the service runs as
User=app, and the config/socket/state directory is root-owned. Check every path the app touches as that user. Modern units add sandbox directives (ProtectSystem=strict,PrivateTmp=) that produce read-only-filesystem errors in unexpected places -ReadWritePaths=is the fix, not loosening file modes. - Resource limits:
LimitNOFILE=too low shows up as too many open files at startup for daemons that open many sockets early.
7. After editing: reload, restart, verify
sudo systemctl daemon-reload # required after ANY unit file edit
sudo systemctl restart app.service
systemctl is-active app.service && journalctl -u app.service -n 20
Forgetting daemon-reload means you're testing the old unit file - a genuinely cruel way to
lose twenty minutes. And verification means more than is-active: confirm the service
actually answers (curl, a health endpoint, a client query). "Green in systemctl" and
"working" overlap less than you'd hope. This exact scenario - a unit failing for a reason
buried two layers down - is a staple of
Linux interview scenarios for the same reason
it's on this checklist: it separates people who read the system's evidence from people who
restart things and hope. There's no Linux lab for it yet, but the
practice labs drill the same read-the-evidence-first loop on a broken Kubernetes
cluster.
FAQ
What does status=203/EXEC mean?
systemd could not execute the ExecStart binary: wrong path, not executable, missing shebang
interpreter, or wrong architecture. It never means the app ran and failed - the app never ran.
Why does my service work manually but fail under systemd?
Different environment, user, working directory, or sandboxing. Reproduce with
sudo -u <serviceuser> <exact ExecStart command> and compare
systemctl show -p Environment,User,WorkingDirectory against your shell.
What is start-limit-hit and how do I clear it?
The unit crashed too many times too fast, so systemd stopped retrying. systemctl reset-failed <unit> clears the counter - but the original crash reason is still in
journalctl -u <unit>, and clearing the limit without reading it just schedules the next
failure.
Do I need daemon-reload after editing a unit file?
Yes, always. systemd caches unit definitions; without systemctl daemon-reload your edits are
invisible, including fixes that would have worked.