Running unattended

By the end of this chapter, Substation scans your band around the clock: started with the computer, started again after a fault, and keeping up with what it hears.

The folder so far

The radio folder as Substation with Subsample and Subsequence left it: Substation installed in an environment inside it, switched on, its defaults kept as defaults.yaml, your band in config.yaml with settings for each receiver you own, OSC sent to Subsequence on another computer and to Subsample on this one, and four test signals made. The stand-in from that chapter is not needed here.

$ mkdir radio
$ cd radio
$ python3 -m venv .venv
$ source .venv/bin/activate

$ pip install "substation==0.8.2"
…
Successfully installed … substation-…

$ substation --init
Created /home/keanu/radio/config.yaml
This is the full default configuration, fully commented. Edit it to suit
your hardware and bands, then start a scan, e.g. `substation --band amateur_2m`.
Any setting you delete falls back to the built-in default.

$ mv config.yaml defaults.yaml

bands:
  amateur_2m_simplex:
    type: NFM
    reception_class: general
    freq_start: 145.2e+6
    freq_end: 145.5625e+6
    channel_spacing: 12.5e+3
    sample_rate: 2.4e+6
    snr_threshold_db: 4.5
    activation_variance_db: 3
    device_overrides:
      hackrf:
        sample_rate: 2.0e+6
        sdr_gain_db: 36
        snr_threshold_db: 6
      airspy:
        sample_rate: 2.5e+6
        sdr_gain_elements:
          LNA: 10
          MIX: 5
          VGA: 12
        snr_threshold_db: 6
      airspyhf:
        sample_rate: 0.912e+6
        snr_threshold_db: 6

# Makes test-signal.wav: eight seconds of the 2m band as a receiver hands it over,
# hiss with a transmission in the middle, for Substation to scan in place of a receiver.

import wave

import numpy

# The frequency the transmission is sent on, in hertz, and how many seconds it lasts.
frequency = 145_500_000
length = 3

# The frequency the recording is centred on, and how many IQ samples it holds
# a second, as a receiver set for the 2m band hands them over.
centre = 145_000_000
rate = 2_400_000

# Two seconds of hiss first, while Substation measures it, and three after.
start = 2
seconds = start + length + 3

# The hiss is random, but the same every time the script runs.
noise = numpy.random.default_rng(1)

with wave.open("test-signal.wav", "wb") as file:

	# Each IQ sample as two 16-bit numbers, I and Q, held as two audio channels.
	file.setnchannels(2)
	file.setsampwidth(2)
	file.setframerate(rate)

	# A second at a time, so the script needs little memory.
	for second in range(seconds):

		# The moment of each IQ sample, in seconds.
		t = second + numpy.arange(rate) / rate

		# The receiver's own hiss.
		hiss = (noise.standard_normal(rate) + 1j * noise.standard_normal(rate)) * 0.01

		# A buzz like a voice: overtones of a low note, each softer than the one below.
		voice = sum(5.3 / n ** 2 * numpy.sin(2 * numpy.pi * 150 * n * t) for n in range(2, 7))

		# Louder and softer six times a second, like syllables.
		loudness = numpy.abs(numpy.sin(2 * numpy.pi * 3 * t))

		# FM: the sound swings the transmission's frequency to either side of its own.
		phase = 2 * numpy.pi * (frequency - centre) * t + loudness * voice

		# The transmission, only while it is on, added to the hiss.
		on = (t >= start) & (t < start + length)
		iq = hiss + numpy.where(on, 0.3 * numpy.exp(1j * phase), 0)

		# Written as pairs of 16-bit numbers, I then Q.
		pairs = numpy.stack([iq.real, iq.imag], axis=1)
		file.writeframes((pairs * 32767).astype("<i2").tobytes())

$ python make_signal.py

# Makes test-noise.wav: six seconds of the 2m band as a receiver hands it over,
# with three things in it strong enough to cross a threshold: a steady noise,
# a burst too short to be a transmission, and a transmission.

import wave

import numpy

# The frequency of each, in hertz.
steady = 145_300_000
burst = 145_400_000
transmission = 145_500_000

# The frequency the recording is centred on, and how many IQ samples it holds
# a second, as a receiver set for the 2m band hands them over.
centre = 145_000_000
rate = 2_400_000

# Six seconds in all.
seconds = 6

# The hiss and the noise are random, but the same every time the script runs.
noise = numpy.random.default_rng(2)

# How far the steady noise has wandered, carried from one second to the next.
drift = 0.0

with wave.open("test-noise.wav", "wb") as file:

	# Each IQ sample as two 16-bit numbers, I and Q, held as two audio channels.
	file.setnchannels(2)
	file.setsampwidth(2)
	file.setframerate(rate)

	# A second at a time, so the script needs little memory.
	for second in range(seconds):

		# The moment of each IQ sample, in seconds.
		t = second + numpy.arange(rate) / rate

		# The receiver's own hiss.
		hiss = (noise.standard_normal(rate) + 1j * noise.standard_normal(rate)) * 0.01

		# The steady noise: a signal whose frequency wanders at random, carrying nothing.
		wander = drift + numpy.cumsum(noise.standard_normal(rate)) * 0.02
		drift = wander[-1]
		steady_noise = 0.03 * numpy.exp(1j * (2 * numpy.pi * (steady - centre) * t + wander))

		# The test signal's buzz, rising and falling, for the burst and the transmission.
		voice = sum(5.3 / n ** 2 * numpy.sin(2 * numpy.pi * 150 * n * t) for n in range(2, 7))
		loudness = numpy.abs(numpy.sin(2 * numpy.pi * 3 * t))

		# The burst, a twentieth of a second long, three seconds in.
		bursting = (t >= 3) & (t < 3.05)
		brief = numpy.where(bursting, 0.3 * numpy.exp(1j * (2 * numpy.pi * (burst - centre) * t + loudness * voice)), 0)

		# The transmission, from two seconds in to four.
		talking = (t >= 2) & (t < 4)
		sent = numpy.where(talking, 0.3 * numpy.exp(1j * (2 * numpy.pi * (transmission - centre) * t + loudness * voice)), 0)

		# All of it together, written as pairs of 16-bit numbers, I then Q.
		iq = hiss + steady_noise + brief + sent
		pairs = numpy.stack([iq.real, iq.imag], axis=1)
		file.writeframes((pairs * 32767).astype("<i2").tobytes())

$ python make_noise.py

# Makes test-loud.wav: eight seconds of the 2m band as a receiver hands it over,
# with a transmission so strong it overloads the receiver, as one sent close by
# would with the gain turned up too far.

import wave

import numpy

# The frequency the transmission is sent on, in hertz, and how many seconds it lasts.
frequency = 145_500_000
length = 3

# The frequency the recording is centred on, and how many IQ samples it holds
# a second, as a receiver set for the 2m band hands them over.
centre = 145_000_000
rate = 2_400_000

# Three seconds of hiss first, while Substation measures it, and two after.
start = 3
seconds = start + length + 2

# The hiss is random, but the same every time the script runs.
noise = numpy.random.default_rng(1)

with wave.open("test-loud.wav", "wb") as file:

	# Each IQ sample as two 16-bit numbers, I and Q, held as two audio channels.
	file.setnchannels(2)
	file.setsampwidth(2)
	file.setframerate(rate)

	# A second at a time, so the script needs little memory.
	for second in range(seconds):

		# The moment of each IQ sample, in seconds.
		t = second + numpy.arange(rate) / rate

		# The receiver's own hiss.
		hiss = (noise.standard_normal(rate) + 1j * noise.standard_normal(rate)) * 0.01

		# A buzz like a voice, louder and softer like syllables, as before.
		voice = sum(5.3 / n ** 2 * numpy.sin(2 * numpy.pi * 150 * n * t) for n in range(2, 7))
		loudness = numpy.abs(numpy.sin(2 * numpy.pi * 3 * t))
		phase = 2 * numpy.pi * (frequency - centre) * t + loudness * voice

		# The transmission, ten times as strong as the one in test-signal.wav.
		on = (t >= start) & (t < start + length)
		iq = hiss + numpy.where(on, 3 * numpy.exp(1j * phase), 0)

		# A receiver can hand over nothing stronger than its largest number,
		# so everything stronger is cut off there, as an overloaded one does.
		pairs = numpy.clip(numpy.stack([iq.real, iq.imag], axis=1), -1, 1)
		file.writeframes((pairs * 32767).astype("<i2").tobytes())

$ python make_loud.py

# Makes test-tone.wav: nine seconds of the 2m band as a receiver hands it over,
# with a transmission like the one in test-signal.wav, carrying a low tone under
# its voice, as many radios send.

import wave

import numpy

# The frequency the transmission is sent on, in hertz, and how many seconds it lasts.
frequency = 145_500_000
length = 3

# The tone under the voice, in hertz.
tone = 88.5

# The frequency the recording is centred on, and how many IQ samples it holds
# a second, as a receiver set for the 2m band hands them over.
centre = 145_000_000
rate = 2_400_000

# Three and a half seconds of hiss first, so that Substation has measured it
# before the transmission starts, and nine seconds in all.
start = 3.5
seconds = 9

# The hiss is random, but the same every time the script runs.
noise = numpy.random.default_rng(3)

with wave.open("test-tone.wav", "wb") as file:

	# Each IQ sample as two 16-bit numbers, I and Q, held as two audio channels.
	file.setnchannels(2)
	file.setsampwidth(2)
	file.setframerate(rate)

	# A second at a time, so the script needs little memory.
	for second in range(seconds):

		# The moment of each IQ sample, in seconds.
		t = second + numpy.arange(rate) / rate

		# The receiver's own hiss.
		hiss = (noise.standard_normal(rate) + 1j * noise.standard_normal(rate)) * 0.01

		# A buzz like a voice: overtones of a low note, each softer than the one below.
		voice = sum(5.3 / n ** 2 * numpy.sin(2 * numpy.pi * 150 * n * t) for n in range(2, 7))

		# Louder and softer six times a second, like syllables.
		loudness = numpy.abs(numpy.sin(2 * numpy.pi * 3 * t))

		# The tone, as steady as the voice is not.
		under = 4 * numpy.sin(2 * numpy.pi * tone * t)

		# FM: the sound swings the transmission's frequency to either side of its own.
		phase = 2 * numpy.pi * (frequency - centre) * t + loudness * voice + under

		# The transmission, only while it is on, added to the hiss.
		on = (t >= start) & (t < start + length)
		iq = hiss + numpy.where(on, 0.3 * numpy.exp(1j * phase), 0)

		# Written as pairs of 16-bit numbers, I then Q.
		pairs = numpy.stack([iq.real, iq.imag], axis=1)
		file.writeframes((pairs * 32767).astype("<i2").tobytes())

$ python make_tone.py

osc:
  sequencer_enabled: true
  sequencer_host: 192.0.2.20
  sampler_enabled: true

How a scan ends

Every command tells whatever started it how it ended, with a number called its exit status: 0 when it succeeded, and another number when it failed. The terminal keeps the last one as $?, which echo prints.

Substation's exit status says whether a fault stopped it. It is 0 when you stop a scan with Ctrl+C, or when an IQ file has played to its end, and 1 when a fault stopped it: a receiver that fails, is unplugged or cannot be found, or a band it does not know, as here, where the band's name is mistyped:

$ substation --band amateur_2m_simplx
… - ERROR - Band 'amateur_2m_simplx' not found. Available bands: …
$ echo $?
1

That number is what lets the computer start Substation again after a fault, and leave it stopped when you stopped it.

Starting with the computer

A program that runs with nobody at the keyboard is called a service, and on Debian, Ubuntu, Raspberry Pi OS and Fedora, systemd is what starts them as the computer starts, and can start one again when it stops. A file called a unit tells systemd how to run a service. This writes one for Substation, /etc/systemd/system/substation.service, filling in your own account's name where it says $USER, and your home folder where it says $HOME, as the file is written:

$ sudo tee /etc/systemd/system/substation.service > /dev/null <<UNIT
[Unit]
Description=Substation, scanning the 2m band
Wants=network-online.target
After=network-online.target

[Service]
User=$USER
WorkingDirectory=$HOME/radio
ExecStart=$HOME/radio/.venv/bin/substation --band amateur_2m_simplex
KillSignal=SIGINT
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
UNIT

User and WorkingDirectory run it as you, in the radio folder, as you have run it in every chapter: it uses your receiver as you can, reads the config.yaml there, and saves its recordings into audio there.
ExecStart is the command it runs. It names the environment's own substation, which works without the environment switched on, as .venv/bin/python did in Substation with Subsample and Subsequence.
Wants and After wait until the computer is on the network before starting it, since it sends OSC messages to another computer.
KillSignal=SIGINT has systemd stop it as Ctrl+C does, so that it closes its recordings and the receiver before it ends. Otherwise systemd stops a service with a signal called SIGTERM, which ends Substation at once and leaves a recording it was making unfinished: without the fields What a recording holds read from it, and without its length.
Restart=on-failure starts it again whenever it ends with a fault, and RestartSec=10s waits ten seconds first. systemd gives up on a service started more than five times in ten seconds, so with the wait between, it keeps trying for as long as the fault lasts, such as a receiver unplugged until you plug it back in.
WantedBy=multi-user.target has it start as the computer starts, once it is enabled, below.

Before systemd runs it, ask systemd to read it:

$ systemd-analyze verify /etc/systemd/system/substation.service

Nothing printed means nothing is wrong. systemd-analyze reads the unit as systemd would, and says what it finds, such as a setting's name misspelt, or an ExecStart program that is not where the unit says.

Then start it, and have it start with the computer from now on. Nothing below is checked when this site is built, since it needs a computer running systemd, and your receiver:

Example not checked: it is not run when this site is built.

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now substation
$ sudo systemctl status substation
$ sudo journalctl -u substation -f

daemon-reload has systemd read the new unit, and enable --now starts it now and whenever the computer starts.
status says whether it is running, with the last lines of its log.
journalctl -u substation -f shows its log as each line arrives, until you press Ctrl+C. Substation's log goes there now, rather than to a terminal.

Substation reads config.yaml only as it starts, so after changing it, start Substation again with sudo systemctl restart substation. To scan by hand with the same receiver, stop the service first with sudo systemctl stop substation. sudo systemctl disable substation stops it starting with the computer.

Room for the recordings

Substation keeps every recording it does not discard, in a folder for each day and band, and never clears old ones away. At its 16,000 audio samples a second, a WAV recording takes nearly 2 MB for every minute of transmissions, and a FLAC recording, as What a recording holds showed, takes less. du -sh audio says how much room they take, and df -h . how much is left.

More than one receiver

One Substation scans one band with one receiver. To scan two bands at once, plug in a second receiver, with an antenna for its band, as An antenna for your band worked out, and give it a Substation of its own, in a unit of its own. --device-index chooses which receiver of a kind to use, counting from 0, so a second RTL-SDR is 1. This one scans the UK's CB radio band, cb_uk, which records out of the box:

$ sudo tee /etc/systemd/system/substation-cb.service > /dev/null <<UNIT
[Unit]
Description=Substation, scanning UK CB radio
Wants=network-online.target
After=network-online.target

[Service]
User=$USER
WorkingDirectory=$HOME/radio
ExecStart=$HOME/radio/.venv/bin/substation --band cb_uk --device-index 1
CPUAffinity=3
KillSignal=SIGINT
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
UNIT

$ systemd-analyze verify /etc/systemd/system/substation-cb.service

A receiver of another kind is the first of its kind: a HackRF One beside an RTL-SDR is --device-type hackrf, with no --device-index.
Both scans read the same config.yaml, save their recordings into a folder for each band, and send their OSC messages where its osc settings say, those to Subsequence naming the band they came from.
CPUAffinity=3 keeps this scan to one of the computer's cores, the fourth, since they are numbered from 0, and nproc says how many there are. Most of a scan's work runs on one core, so where two scans must keep up, keep each to a core of its own. Add CPUAffinity=2 under [Service] in substation.service too, with sudo nano /etc/systemd/system/substation.service, so that the two never share one.

Then have systemd read both, start the first again, and start the second:

Example not checked: it is not run when this site is built.

$ sudo systemctl daemon-reload
$ sudo systemctl restart substation
$ sudo systemctl enable --now substation-cb

Keeping up

Substation has to finish with each slice before the next arrives, about a fifth of a second later, as How Substation hears a transmission described. A slice that takes longer is logged as a warning starting Processing overrun, with how long it took against how long it lasted. While Substation is behind, the slices still to come wait their turn in a queue, and when the queue is full it drops IQ samples as they arrive, logging Sample queue full; dropping samples: whatever was sent in them is cut short or lost.

To find either in a service's log:

Example not checked: it is not run when this site is built.

$ sudo journalctl -u substation | grep -E "overrun|queue full"

The band's sample_rate decides most of the work, since Substation processes every IQ sample, mostly on one core. If either warning keeps coming:

Scan a narrower band, at a lower sample_rate.
Leave out radio channels you do not need with exclude_channel_indices, which takes their numbers as the log and the recordings' names give them.
Hold more slices in the queue with sample_queue_maxsize, 200 unless you set it, which rides out a burst, such as several radio channels turning on at once. A computer that falls behind all the time fills the queue whatever its size.

Substation's widest bands have not yet been shown to keep up: air_civil_1, air_civil_2 and dmr, each with a sample_rate of 12.5 MHz. On the one desktop computer they have been tested on, processing fell behind and IQ samples were dropped. On a Raspberry Pi, or wherever Processing overrun keeps coming, scan a narrower band, such as one of dmr_1 to dmr_5, each a part of dmr.

An RTL-SDR handing over a long stretch at a time stops with Failed to submit transfer unless Linux sets aside more memory for USB, as Installing Substation did.

Where to go from here

This is the guide's last chapter. Substation now scans your band around the clock, records what it hears, and sends each transmission where Substation with Subsample and Subsequence set it to. The configuration reference gives every setting, the band library every band Substation ships, and the command-line reference every option.