Services and logs: systemctl and journalctl
The shop’s API is down. On the web server, check the app with systemctl status, read its log with journalctl -u, restart it and confirm it answers again.
Programs that run all the time on a server, like a web server, a database or the shop's own app, are services. On most Linux systems a program called systemd starts them when the machine boots, restarts them if asked, and keeps their logs. You talk to it with systemctl and read the logs with journalctl. The machines here (your laptop, web, db and the router) are pretend ones inside your browser, so nothing you type reaches a real network.
Right now the shop's API answers with an error. See for yourself from your laptop:
curl -s http://web/api/502 Bad Gateway means nginx, the web server in front, is fine, but the app behind it isn't answering. The app runs on web, so log in there with ssh web.
systemctl
| Command | Does |
|---|---|
systemctl status app | is it running, since when, its main process and its last log lines |
sudo systemctl start app / stop app | starts or stops it |
sudo systemctl restart app | stops and starts it (after a change, or a crash) |
sudo systemctl enable app | starts it at every boot (disable undoes that) |
systemctl is-active app | just active or inactive, handy in scripts |
systemctl list-units --type=service | every service and its state |
Changing a service needs sudo; looking doesn't.
journalctl
Everything services print goes to the journal. journalctl reads it, oldest first:
| Command | Shows |
|---|---|
journalctl -u app | only the app's lines |
journalctl -u app -n 20 | its last 20 lines |
journalctl -u app -p err | only errors |
journalctl -u app --since "1 hour ago" | the recent part |
journalctl -f | new lines as they arrive (Ctrl+C stops) |
Many programs also write their own files under /var/log: /var/log/nginx/access.log and error.log for nginx, for example.
Restart, then check
After a restart, look again: systemctl status should say active (running), and the real test is whether the site answers. A restart gets things going again, but the log line that explains the crash is what someone has to fix later.
Your turn
Get the API at http://web/api/ working again. On web: (1) check the app service’s status; (2) find the error message in its log; (3) restart it. Then (4) back on your laptop, check that the API answers.
Your task
Get the API at http://web/api/ working again. On web: (1) check the app service’s status; (2) find the error message in its log; (3) restart it. Then (4) back on your laptop, check that the API answers.
- Check the app’s status on web (not done yet)
- Find the error in its log (not done yet)
- Restart it (not done yet)
- From your laptop, the API answers (not done yet)
The checklist ticks itself off as you work in the terminal: any way that gets the result counts.
Hints come one at a time, then one way to do it. Try each before the next.
What this lesson uses, in one place.
systemctl- Control services (systemd) More on systemctl
journalctl- Read the system log More on journalctl
curl- Fetch a URL, or test a web server More on curl
Every everyday command on one page: the Linux commands cheat sheet.