Part 6 of 6 Running one for other people

TLS, a proxy and systemd

By the end of this chapter, the team's instance answers at https://tasks.example.com, listens on loopback with something in front of it terminating TLS, and is kept running by systemd across a restart.

This is the chapter the earlier parts of this guide have been borrowing from. Everything Keanu reached in Part 2 was reaching a server set up like this.

$ sudo useradd --system --no-create-home --shell /usr/sbin/nologin subroutine
$ sudo python3 -m venv /opt/subroutine
$ sudo /opt/subroutine/bin/pip install "subroutine[postgres]"
…
Successfully installed …
$ sudo install -d -o subroutine -g subroutine -m 0755 /var/lib/subroutine
$ sudo -u postgres createuser subroutine
$ sudo -u postgres createdb --owner=subroutine subroutine

$ sudo -u subroutine env \
    XDG_CONFIG_HOME=/var/lib/subroutine/config \
    XDG_DATA_HOME=/var/lib/subroutine/data \
    XDG_STATE_HOME=/var/lib/subroutine/state \
    SUBROUTINE_DATABASE_URL=postgresql+psycopg:///subroutine \
    /opt/subroutine/bin/subroutine init --workspace metacortex --instance-name MetaCortex
Ready. Try: subroutine add "something to do"
…
$ sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
database_url = "postgresql+psycopg:///subroutine"
SETTING

Why it refuses a public address

Ask for a bind beyond this machine with nothing in front of it, and you get this:

$ sudo -u subroutine env \
    XDG_CONFIG_HOME=/var/lib/subroutine/config \
    XDG_DATA_HOME=/var/lib/subroutine/data \
    XDG_STATE_HOME=/var/lib/subroutine/state \
    /opt/subroutine/bin/subroutine serve --host 0.0.0.0
Refusing to listen on 0.0.0.0 without TLS: bearer tokens sent over plain HTTP are compromised tokens.
Either put a TLS-terminating proxy in front and set public_url to its https:// address, or pass --insecure if this network is genuinely trusted.

It is a refusal rather than a warning on purpose. A warning on a server that runs for months scrolls out of sight in its first minute, and the risk lasts as long as the process does.

The reason is the credentials. Subroutine authenticates with bearer tokens, and a bearer token sent over plain HTTP is a compromised token: anything on the path has it, and reading it does not make it expire.

There are three honest ways past this, and one of them is the one to use.

The way to use: a proxy, and Subroutine on loopback

Leave the bind at its default, 127.0.0.1, put a proxy in front, and tell Subroutine the address that proxy serves:

$ sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
public_url = "https://tasks.example.com"
SETTING

The refusal never fires, because nothing beyond this machine is reaching Subroutine directly. public_url is what clients and agents are told to come back to, and it is published at GET /v1/meta, so an agent handed a token can find out where it is talking to without being told separately.

The proxy is another product's, and its configuration is yours rather than ours. With nginx:

Not checked: the build does not run this.

server {
    listen 443 ssl;
    server_name tasks.example.com;

    ssl_certificate     /etc/letsencrypt/live/tasks.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/tasks.example.com/privkey.pem;
    add_header Strict-Transport-Security "max-age=31536000" always;

    location / {
        proxy_pass http://127.0.0.1:8471;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Or Caddy, which gets a certificate by itself:

Not checked: the build does not run this.

tasks.example.com {
    reverse_proxy 127.0.0.1:8471
}

Neither is checked by anything here, and both are a starting point for however your machines are already set up. What is checked is the Subroutine end: the refusals above, the bind, the setting, and the two addresses below answering through whatever you put in front.

The other two ways, and when each is honest

A public bind with public_url already https://. For TLS terminated somewhere this check cannot see, such as a load balancer. The https:// scheme is what satisfies it, which makes this you taking responsibility rather than the program verifying anything. A wrong scheme is caught:

$ sudo -u subroutine env \
    XDG_CONFIG_HOME=/var/lib/subroutine/config \
    XDG_DATA_HOME=/var/lib/subroutine/data \
    XDG_STATE_HOME=/var/lib/subroutine/state \
    SUBROUTINE_PUBLIC_URL=http://tasks.example.com \
    /opt/subroutine/bin/subroutine serve --host 0.0.0.0
Refusing to listen on 0.0.0.0 without TLS: public_url is set to 'http://tasks.example.com', which is not an https:// address.
Point public_url at the https:// address your proxy serves this on, or pass --insecure if this network is genuinely trusted.

--insecure. The honest way to say this is my own network and I know what I am doing. It goes on the command rather than into a file, because it is a decision about this invocation rather than a property of the installation, and a flag you type is one you cannot forget you typed.

Whichever you choose, be clear what a bind beyond loopback without TLS means: bearer tokens cross that network in clear, and anything that can see the traffic can replay them. On a home network that can be a reasonable trade. It should be one you have made rather than one you inherited from a flag you copied.

Serving it

$ sudo -u subroutine env \
    XDG_CONFIG_HOME=/var/lib/subroutine/config \
    XDG_DATA_HOME=/var/lib/subroutine/data \
    XDG_STATE_HOME=/var/lib/subroutine/state \
    /opt/subroutine/bin/subroutine serve
Serving on http://127.0.0.1:8471
Reached at https://tasks.example.com
…

The first line is where it is listening, and the second is where people will find it: that is public_url read back, which is the quickest confirmation that the setting took.

Two addresses for whatever is watching

Neither needs a credential, and both are worth pointing a monitor at:

$ curl -s https://tasks.example.com/healthz
{"status":"ok","api_version":"1.0"}
$ curl -s https://tasks.example.com/readyz
{"status":"ready","api_version":"1.0","schema_revision":"…"}

/healthz says the process is up. /readyz says it can reach its database, that the database is at the schema this build expects, and that it is still serving the instance it started on.

That last one is the answer to am I serving the data I think I am. A process whose database is replaced underneath it keeps its handle on the old one and goes on reading data nobody else can see; comparing the instance is what turns that from silence into a refusal. It is also why putting a backup back as a clone makes a running service report itself not ready: a clone is deliberately a new instance, and the process needs restarting.

Everything else does need a credential, which is the other thing worth checking from outside:

$ curl -so /dev/null -w '%{http_code}\n' https://tasks.example.com/v1/tasks
401

The systemd unit

/etc/systemd/system/subroutine.service:

$ sudo tee /etc/systemd/system/subroutine.service > /dev/null <<'UNIT'
[Unit]
Description=Subroutine
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=subroutine
Group=subroutine

StateDirectory=subroutine
Environment=XDG_CONFIG_HOME=/var/lib/subroutine/config
Environment=XDG_DATA_HOME=/var/lib/subroutine/data
Environment=XDG_STATE_HOME=/var/lib/subroutine/state

ExecStart=/opt/subroutine/bin/subroutine serve
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

[Install]
WantedBy=multi-user.target
UNIT

StateDirectory=subroutine makes /var/lib/subroutine and hands it to the service account, which is the directory An account, an install and a first run made by hand for init. The three Environment= lines are why the service never needs the account to have a home.

ExecStart takes no --host or --port because both are settings, and their defaults are 127.0.0.1 and 8471.

TimeoutStopSec is the outer half of a pair, and the order matters. Subroutine stops accepting, gives requests already in flight fifteen seconds to finish, and exits. systemd's timeout has to be the longer of the two, or it would kill a shutdown that was about to complete. Left at systemd's default, stopping a server with a request stuck on something takes a minute and a half, and you find that out during an incident, because that is the only time anything is stuck long enough to notice.

ProtectSystem=strict makes the whole filesystem read-only apart from what StateDirectory grants. That is why a backup directory anywhere else has to be named, with ReadWritePaths=-/srv/backups/subroutine: the leading - means ignore this path if it is not there, so an unmounted volume costs you a backup rather than a service that will not start.

Before asking systemd to run it, ask systemd to read it:

$ systemd-analyze verify /etc/systemd/system/subroutine.service

Silence is the answer you want. It parses the unit, resolves ExecStart and checks every directive, so a program that is not where you said, a setting that does not exist and a typo in a directive name all turn up here rather than in journalctl at the wrong hour.

The rest is systemd's own, and nothing here checks it:

Not checked: the build does not run this.

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now subroutine
$ sudo systemctl status subroutine
$ sudo journalctl -u subroutine -f

Counting the right address

Failed authentications are counted per address, so that guessing a token gets slower. Through a proxy every request arrives from the proxy, so without help they share one allowance, and one client hammering with a stale credential makes everybody else's mistakes answer 429 instead of 401.

Name the proxy, and the real caller is counted instead:

Not checked: the build does not run this.

trusted_proxies = ["127.0.0.1"]

That is the address this instance sees the proxy connecting from, which is not always the one you think of as the proxy's: beside it on the same machine that is 127.0.0.1, and on another machine it is that machine's address on your network.

Name only proxies you control. The header this reads is written by whoever sends the request, so the setting is you vouching for a particular peer. Point it at something you do not control and any caller can choose which bucket it is counted in, which is worse than leaving it empty. Left empty, the header is ignored entirely, which is right when nothing is in front.

Leave cors_origins empty

That includes anybody who has just found the web interface. It is served by this instance from this instance's own address, so nothing about it is cross-origin and nothing about it needs this setting. Fill it in and you are telling browsers that pages somewhere else may make authenticated requests here.

There is one case for it, which is a front end of your own on another address, and it is not one you reach by accident.

With that, the instance is served, kept running and reachable at the address the rest of this guide has been using. The next chapter gives the people their way in.