I will confess. I started writing this post about some stupid systemd tricks. However, I wanted to explain a little about systemd first, and that wound up being longer than the tricks. So this Linux Fu will be some very fundamental systemd information. The next one will have some examples, including how to automount a Raspberry Pi Pico. Of course, by the end of this post, you’ll have only scratched the surface of systemd, but I did want to give you some context for reading through the rest of it.

Like many long-time Unix users, I’m not a big fan of systemd. Then again, I’m also waiting for the whole “windows, icon, mouse, pointer” fad to die down. Like it or not, systemd is here and probably here to stay for the foreseeable future. I don’t want to get into a flame war over systemd. Love it or hate it, it is a fact of life. I will say that it does have some interesting features. I will also say that the documentation has gotten better over time. But I will also say that it made many changes that perhaps didn’t need to be made and made some simple things more complicated than they needed to be.

In the old days, we used “init scripts,” and you can still do so if you are really motivated. They weren’t well documented either, but it was pretty easy to puzzle out the shell scripts that would run, and we all know how to write shell scripts. The systemd way is to use services that are not defined by shell scripts. However, systemd tries to do lots of other things, too. It can replace cron and run things periodically. It can replace inetd, syslog, and many other traditional services. This is a benefit or a drawback, depending on your point of view.

(Editor’s note: And this logging functionality was exactly what was abused in last week’s insane liblzma / ssh backdoor.)

Configuring systemd requires you to create files in one of several locations. In systemd lingo, they are “units.” For the purpose of this Linux Fu, we’ll look at only a few kinds of units: services, mounts, and timers. Services let you run programs in response to something like system start-up. You can require that certain other services are already running or are not running and many other options. If the service dies, you can ask systemd to automatically restart it, or not. Timers can trigger a service at a particular time, much like cron does. Another unit you’ll run into are sockets that represent — you guessed it — a network socket.

Basics

If you haven’t used systemd, the main interface to it is systemctl (not sysctl, which is something different). With it, you can enable and disable “units,” which are usually “services.” So, for example, my ssh server is a service unit. By defining a unit, someone could say, “Well, if you are starting in multiuser mode, wait for networking to be available and run the ssh server.” Or, sort of like inetd, run the ssh server when someone opens a connection on port 22. Not all units are services. Some are network sockets, and some are timers. There are other kinds, too. A timer is like a cron job. You can set them up to run every so often.

Systemd maintains two sets of units. One set is for the system, and you need to be root to work with those. But it also manages user-level things. For example, your sound server, probably pulseaudio or pipewire, has a systemd service unit. Not only does it launch the service, it restarts it if it dies. If you change a unit file, you have to tell systemd to reread all the files with systemctl daemon-reload. You can enable, disable, start, and stop services and other units with the systemctl command, too. There’s also a way to mask and unmask units, which is like a super disable.

While the documentation has become better over the years, the format for the units is still pretty hairy because there are so many options available. Most of the time, you only need a small subset of the many available options.

For Example

Here’s a fairly simple service unit:

[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

You can probably puzzle most of that out. Here’s a very complicated service unit:

[Unit]
Description=Entropy Daemon based on the HAVEGE algorithm
Documentation=man:haveged(8) http://www.issihosts.com/haveged/
DefaultDependencies=no
After=apparmor.service systemd-tmpfiles-setup.service systemd-tmpfiles-setup-dev.service
Before=sysinit.target shutdown.target
Conflicts=shutdown.target
# RNDADDENTROPY ioctl requires host-level CAP_SYS_ADMIN, fails in unprivileged container
ConditionVirtualization=!container

[Service]
EnvironmentFile=-/etc/default/haveged
ExecStart=/usr/sbin/haveged --Foreground --verbose=1 $DAEMON_ARGS
Restart=always
SuccessExitStatus=137 143
SecureBits=noroot-locked
CapabilityBoundingSet=CAP_SYS_ADMIN
PrivateTmp=true
PrivateDevices=true
PrivateNetwork=true
ProtectSystem=full
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
RestrictNamespaces=true
RestrictRealtime=true

LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
SystemCallFilter=@basic-io @file-system @io-event @network-io @signal
SystemCallFilter=arch_prctl brk ioctl mprotect sysinfo

[Install]
WantedBy=sysinit.target

This online form takes the tedium out of writing this purely fictional service (link in text).

Whoa! There’s a lot to parse there. You can find all the documentation, of course. But many people copy some starter service and modify it. Another option is to use a wizard to help you create your unit files. For example, you can find some online. Fill out the forms on the left and observe the file on the right. Installation instructions are up top. If you don’t know what a field asks for, there’s a little info button to give you some help text. You can often pick things from a drop-down. Sure, you can write it all yourself, but this does make it a little easier.

If you want to make a timer, you need a service file and a timer file. There’s an online wizard for that, too. It isn’t quite as helpful, and you need to understand how systemd represents time intervals. This is a bit more complicated than cron because you can set times relative to the timer’s activation, the time of system boot, or relative to the timer’s last activation. You can also set specific dates and times, add a random delay, or make timers persist.

That last option is like using anacron. Suppose you have a timer set to run at 1 AM daily on a laptop. The problem is that by 1 AM, the laptop is probably off. A persistent timer will realize that when you turn the laptop on at 8 AM that it should have run the 1 AM trigger, and then do so.

There’s More…

A similar web page can make your timers, too.

There is so much more to know about units. Some of that is related to features. Some of it is related to quirks. For example, comments can’t go at the end of a line — they have to be on their own line. For another example, special characters in the file names get hex escapes (this will be important later). To further complicate things, there are ways to make templates and add in pieces. All of that is beyond the scope of this post, but be aware, I’m painting the simplest possible picture of the systemd universe.

The other problem is exactly where to put your unit files. They are scattered everywhere, and while there is a good reason for where they live, it is still annoying.

The short version is you should probably put system units in /etc/systemd/system unless you have a good reason to put it somewhere else. Units for all users can go in /etc/systemd/user. If you want a unit just for one user, try ~/.config/systemd/user. If you are looking for the location of an existing unit, usually “systemctl status xxx.service” will tell you. Otherwise, try “systemctl show -P FragmentPath xxx.service.”

If you want to read the actual documentation:

System unit directories

The systemd system manager reads unit configuration from various directories. Packages that want to install unit files shall place them in the directory returned by pkg-config systemd –variable=systemdsystemunitdir. Other directories checked are /usr/local/lib/systemd/system and /usr/lib/systemd/system. User configuration always takes precedence. pkg-config systemd –variable=systemdsystemconfdir returns the path of the system configuration directory. Packages should alter the content of these directories only with the enable and disable commands of the systemctl(1) tool. Full list of directories is provided in systemd.unit(5).

User unit directories

Similar rules apply for the user unit directories. However, here the XDG Base Directory specification is followed to find units. Applications should place their unit files in the directory returned by pkg-config systemd –variable=systemduserunitdir. Global configuration is done in the directory reported by pkg-config systemd –variable=systemduserconfdir. The enable and disable commands of the systemctl(1) tool can handle both global (i.e. for all users) and private (for one user) enabling/disabling of units. Full list of directories is provided in systemd.unit(5).

Stupid Trick #0: Run Some Program on Startup

This isn’t really a trick, but I wanted at least one example this time to show you how simple a homebrew service can be.

In an earlier post, I used a cheap “smart display” to show Hackaday’s RSS feed. If you want that to run on startup — or you just want the normal Python code to get the system status display — you need to run it somewhere. Sure, you could execute it by hand or put it in your startup files. But the systemd way to do it is with a unit. Here’s mine:

[Unit] 
Description=[Operate Smart Display] 

[Service] 
Type=simple 
StandardOutput=journal 
ExecStart=/home/alw/apps/turing-smart-screen-python-3.1.0/service 

[Install] 
WantedBy=default.target

That’s easy enough. It defines a simple service that outputs to the journal. It starts by default. Piece of cake. Next time, we’ll see a few more simple examples and maybe that will spur you to write your own unit files. Watch for it!