Part 6 of 6 Running one for other people
Upgrading
By the end of this chapter, the instance has been checked before and after, its database has been migrated in the right order, and you know what an instance says when new code meets a database that has not caught up.
The commands are four lines. What is worth the reading is the order they go in, one line of output that tells you whether the first of them happened at all, and what the machine does on the day the order goes wrong.
The work so far
Laurence has the instance of An account, an install and a first run, served at the address TLS, a proxy and systemd gave it, backed up where Backups, and putting one back put it:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin subroutine
sudo python3 -m venv /opt/subroutine
sudo /opt/subroutine/bin/pip install "subroutine[postgres]"
sudo install -d -o subroutine -g subroutine -m 0755 /var/lib/subroutine
sudo install -d -o subroutine -g subroutine -m 0700 /srv/backups/subroutinesudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine init --workspace metacortex --instance-name MetaCortex
sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
public_url = "https://tasks.example.com"
backup_directory = "/srv/backups/subroutine"
SETTING
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db backup
Before: is this machine coherent
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine doctor
subroutine doctor is one command for the question is this machine what I think it is: what is running and where it came from, which configuration it is reading, what each connection answers, and when a backup was last taken. It exits non-zero when something needs attention, so it can be the last line of an update script.
Run it as the service account, with the same three variables as everything else in this part. That is what the config, data and state lines are for. If they are not the three the unit sets, you are looking at a different installation from the one that serves requests, and every line under them is true about the wrong machine.
An instance nobody can reach prints something different here, and not the difference you would guess: it shows one exposure line saying nothing is reachable from outside, instead of cors_origins and rate_limit, because neither setting is in force when nothing can reach it. So a machine you have not published yet is not missing a line; it has a substitution.
The line that must move is program. That is the whole before and after: it names the version and the path of the copy this procedure upgrades, so a number that has not changed means the install did not take. local moves as well when the release carried a migration, and stays where it is when it did not.
Does this one need a migration
Whether a release moves the database is on the release itself. Each entry in the changelog that moves the schema carries a notice saying so, with the revisions it moves between, and the project's own checks refuse a release that moves the schema without one. So will this need downtime is answered before anything is downloaded, which is the point of it being written there rather than discovered here.
Subroutine does not check for updates unless you ask. With nothing set, an instance can run for years without making an outbound request. Asking is a command:
Not checked: the build does not run this.
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db upgrade --checkIt answers in a few lines: what is running, what has been released, and whether taking it changes the schema. It changes nothing, so it is safe on a machine you have not decided about, and it reports what is running rather than what a package manager believes is installed. Nothing here checks that command, because it asks the package index and these examples run with no network.
Or set check = true under [releases] and the instance asks once a day, only while somebody is using it, and says what it found in its own log, in the browser to whoever may administer it, and in subroutine whoami.
The order, and why it is that order
The package manager moves the code. Subroutine moves the database. In that order, and it will not do the first for you: a tool that installs software over itself fights whatever installed it, cannot do it safely while running, and is worse at it than your package manager.
Not checked: the build does not run this.
sudo systemctl stop subroutine
sudo /opt/subroutine/bin/pip install --upgrade subroutine
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db upgrade
sudo systemctl start subroutineStop first and start last, so there is never a moment where new code is serving an old database.
Those three variables are not decoration, and this is the one step where leaving them off fails quietly. db upgrade acts on a database, and it finds that database through configuration, so without them it reads your config.toml rather than the service's, finds whatever database that names, and reports on the wrong one. It will look like it worked.
What the migration step says
It is safe to run when there is nothing to do, which also makes it the cheapest way to ask the question:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db upgrade
Read the version on the first line, because it is the only part of this that can tell you the install worked. A release that carries no migration and an install that never happened print the same two schema numbers and the same Nothing to do., so if that version is not the one you just installed, the database is fine and the software did not move. It happens more easily than it sounds: a copy installed from a checkout carries a development version, which compares as newer than anything published, so an upgrade declines it without failing.
When there is something to do, the same command reports what is installed and what the database is at, backs up and verifies the copy where it landed, migrates, and then reads the schema back rather than assuming. Its own copies are the second of the three kinds Backups, and putting one back describes, so they bound themselves and never touch your routine backups.
If a migration fails, the message says where it stopped and where the backup is, with the restore command spelled out. It does not claim the database is unchanged: each migration runs in its own transaction, so an upgrade spanning three releases can leave the first two applied, and that is exactly the case where somebody needs the truth rather than reassurance.
Add --yes where the instance is marked protected and there is no terminal to answer for it, as in a deploy script.
When it goes in the wrong order anyway
A mistimed deploy is the ordinary way to get new code onto an old database, and the instance does not pretend otherwise. /readyz answers 503 naming both revisions, and every write is refused with a 409 saying the same thing, while reads go on being served.
That is a choice rather than a half measure. Refusing to start would take the /readyz sentence away, so you would get a connection refused and have to go to the journal for the reason, and it would stop somebody looking something up during an upgrade they are not part of.
What it does not protect you from, said plainly. A migration that backfills an existing column leaves that column present and empty until it runs, so a read can be complete, plausible and wrong. Refusing writes stops you building on top of that. It cannot stop you being shown it.
The command line makes the same comparison and refuses in the same way, naming the remedy in the direction of the mismatch: a database behind the software is upgraded, and a database ahead of it means the software is what moved wrongly, because there is no downgrade. The administrative commands are deliberately outside that check, so db current, db backup, db backups, db restore and db upgrade itself keep working while it is firing, which is when you need them.
After: the cheap question
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db current
Then doctor again, and program should be the version you installed.
Why doctor in your own terminal says something else
Run it from your own shell and program will not move, correctly. It is shorter to type and it works, so it is what you will reach for, and it reports your install rather than the server's, which this procedure never touches. The server then appears as a connection line, and that connection is what moves. An unchanged program there is not a failed upgrade; it is a different question being answered.
And your own copy is now behind the server, which is expected and says so. Nothing upgraded it, so it is an older program talking to a newer instance, and subroutine whoami prints both versions and ends by saying they disagree, so a call may be refused for a field one of them does not have. Meeting that sentence for the first time straight after an upgrade reads like damage, and it is the check working. Upgrade your own copy the way you installed it, whenever suits. Nothing on the server is waiting for it.
Tracking a git ref rather than a release
Everything above assumes a release, which is the ordinary case. Following the repository is a legitimate choice for something you host yourself, and two things work differently.
The upgrade is the same command with the URL in it, because pip will not upgrade a direct reference by name:
Not checked: the build does not run this.
sudo /opt/subroutine/bin/pip install --upgrade "subroutine[postgres] @ git+https://github.com/simonholliday/subroutine"And you have to check that it took, because on a direct URL pip clones, resolves the commit, builds the metadata and prints neither Successfully installed nor already up to date, so its output cannot tell already there from declined to replace. The version answers:
/opt/subroutine/bin/subroutine --version
The part after +g in a development version is the commit. If it has not moved, neither has the software, whatever the install printed.
With that, Part 3 is done: the instance is installed under an account of its own, on a database that suits the team, served at an address behind a proxy, with the people and their agents in it, backed up, its credentials accounted for, and a way to move it forward that says at each step whether the step happened.