Linux services and logs (systemd)

The course builds an OpenZiti network from three kinds of program, each running on Ubuntu Linux machines:

• The controller manages the network: it holds the settings and decides who may connect to what.
• A router carries the network’s traffic between machines.
• A tunneler runs on a client machine and sends that machine’s traffic into the network. In Module 7, the fleet machines are the course’s three Ubuntu client machines that run it.

This page explains how Linux starts those programs, keeps track of them and records what they say. You don’t need to know how to code to follow it.

What a service is

Linux is an operating system: the base software that runs a computer and decides which program may do what. Ubuntu is one version of Linux, and it’s the one the course’s machines run.

Most programs you know open a window and stop when you close it. A service is different: it’s a program that runs in the background, with no window, and usually starts by itself when the machine boots (starts up after being switched on or restarted). The controller, the router and the tunneler are all services. You log in to the machine over SSH, the way you log in to a remote Linux machine from a terminal, to check on them. They keep running after you log out.

systemd starts and tracks services

systemd is the part of Linux that starts services and keeps track of them. It’s the first program to run when the machine boots, and it starts the rest of the services from there.

systemd knows about each service from a unit file: a short text file that describes one thing systemd manages. A unit is that one thing. Units come in several types. A unit whose name ends in .service describes a program that systemd runs and keeps track of while it runs. The course’s three are ziti-controller.service, ziti-router.service and ziti-edge-tunnel.service. When you type a name without the ending, such as ziti-router, systemd adds .service for you.

A unit file for a service has a few labelled sections. Here is a made-up minimal one:

[Unit]
Description=Example service

[Service]
ExecStart=/usr/bin/example-program

[Unit] describes the service. [Service] says how to run it: ExecStart= is the command systemd runs to start it. A service section can also tell systemd to restart the program when it exits, with a setting called Restart=. The tunneler’s unit file in Module 7 sets it. Real unit files usually have a third section, [Install], which is used only when you enable the service. To see the real unit file for a service, run systemctl cat <service>.

Start and enable are different

The command systemctl is how you give systemd instructions. Two of its words sound alike but do different jobs:

• start runs the service now. It doesn’t make it come back after a reboot.
• enable sets the service to start by itself at boot. On its own, it doesn’t start it now.

enable --now does both in one step, and the labs use it often. disable undoes enable, and disable --now also stops the service.

Module 7 shows the difference. Right after you install the tunneler, systemctl is-enabled prints disabled and systemctl is-active prints inactive: the program is on the machine, but nothing has started it. After sudo systemctl enable --now ziti-edge-tunnel.service it’s active, and later in that lab, a reboot brings it back because it’s enabled.

Checking on a service

systemctl status <service> is the main check. It prints whether the service is loaded (systemd has read its unit file) and enabled, whether it’s running (active (running), inactive or failed), and then its last 10 log lines. systemctl is-active <service> prints one word, such as active, which is quicker to read.

Some commands show long output inside a pager: a program that shows text one screen at a time. Press q to leave it. Adding --no-pager prints everything straight into the terminal instead, which the labs often add.

To stop a service, use stop. To stop it and start it again in one step, use restart. The labs restart a service after changing its settings, so that it reads them again.

The journal: a service’s log

A log is the record of messages a program writes while it runs: what it started, what it connected to, and what went wrong. On Linux with systemd, a background service called systemd-journald collects these messages, including everything a service prints, and stores them in the journal.

You read the journal with journalctl. On its own it shows every message on the machine, oldest first. The labs always narrow it down:

• -u <service> shows only that service’s messages.
• -n 50 shows only the newest 50.
• -f shows the newest messages and keeps printing new ones as they arrive, until you press Ctrl+C.
• --since -15min means from 15 minutes ago until now.

When a service is failed, its journal usually says why. The labs’ troubleshooting steps often start there.

systemctl tells systemd what to start and when; the service's messages go to the journal, which journalctl reads.Swipe sideways to see the whole diagram →

Packages that add a service

On Ubuntu, you install software with apt, the package tool. A package is a bundle of a program and the files it needs. You install one with sudo apt install <package> (sudo runs one command as the administrator; Why sudo below explains it).

All three OpenZiti packages, openziti-controller, openziti-router and ziti-edge-tunnel, bring their own unit file, so installing them adds a service. The controller and router packages also bring a setup script. It asks you questions, and when it succeeds it enables and starts the service. In Module 3, installing the controller starts the script for you; in Module 4, you run the router’s script yourself after installing. A service set up without the script is started with sudo systemctl enable --now <service>.

Why sudo

Ubuntu turns off direct login for root, the account that may change anything on the system. Instead, a user with permission puts sudo in front of a command to run that one command with root’s rights, after typing their own password. Installing packages and starting, stopping or enabling services all change the system, so they need sudo.

Reading the whole journal is limited too. By default only root and members of three groups (named sets of users who share permissions), systemd-journal, adm and wheel, can read it. Members of the adm group can read it without sudo. The labs usually add sudo anyway, so the command works either way. Checking a status changes nothing, so some status commands in the labs run without it too.

Commands you’ll use in the course

Command	What it does
sudo apt install <package>	Installs a package; the OpenZiti packages also add their service.
systemctl status <service> --no-pager	Shows whether the service is enabled and running, followed by its newest log lines.
systemctl is-active <service>	Prints one word for the service’s state, such as active or inactive.
systemctl cat <service>	Shows the service’s unit file, the text that tells systemd how to run it.
sudo systemctl start <service>	Starts the service now.
sudo systemctl stop <service>	Stops the service now.
sudo systemctl restart <service>	Stops the service and starts it again, so it reads its settings afresh.
sudo systemctl enable --now <service>	Sets the service to start at every boot, and starts it now.
sudo systemctl disable --now <service>	Stops the service and stops it starting at boot.
sudo journalctl -u <service> -n 50 --no-pager	Prints the service’s newest 50 log lines.
sudo journalctl -u <service> -f	Shows the service’s log live, until you press Ctrl+C.
sudo journalctl -u <service> --since -15min --no-pager	Prints the service’s log from the last 15 minutes.