Part 6 of 6 Running one for other people
Backups, and putting one back
By the end of this chapter, the team's instance is backed up to a volume of its own and pruned on a schedule, and a backup has been put back twice: once as this instance recovering its own data, and once as a separate copy to look at.
The commands are short. What takes the space is the two questions nobody asks until the bad afternoon, which are what a backup protects you against and which of the two kinds of restore you meant.
The work so far
Laurence has the instance of An account, an install and a first run, moved onto PostgreSQL by PostgreSQL, and when to switch, with a task in it:
sudo useradd --system --no-create-home --shell /usr/sbin/nologin subroutine
sudo python3 -m venv /opt/subroutine
sudo /opt/subroutine/bin/pip install "subroutine[postgres]"
sudo install -d -o subroutine -g subroutine -m 0755 /var/lib/subroutine
sudo -u postgres createuser subroutine
sudo -u postgres createdb --owner=subroutine subroutinesudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
SUBROUTINE_DATABASE_URL=postgresql+psycopg:///subroutine \
/opt/subroutine/bin/subroutine init --workspace metacortex --instance-name MetaCortex
sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
database_url = "postgresql+psycopg:///subroutine"
SETTING
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine add "Rewrite the home page copy"
Where they go, and what that protects
Left alone, backups go into the instance's own data directory. For a single machine that is a real backup: it protects you from a bad migration, from a restore you did not mean, from a delete nobody meant, and from a database that corrupts itself. It fails for exactly one thing, which is losing the device.
So if the device matters, put them on another volume:
sudo install -d -o subroutine -g subroutine -m 0700 /srv/backups/subroutine
sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
backup_directory = "/srv/backups/subroutine"
SETTINGThat is the path TLS, a proxy and systemd had to name in the unit, because ProtectSystem=strict makes everything outside the state directory read-only.
It is a judgement about what you are protecting against, not a requirement. Nothing here refuses a path, warns about one, or nags about how old the newest copy is: how old is too old depends on whether this is a laptop or a server, and only you know which.
A network mount is the intended destination and works. The file is built locally and then copied, because the tools that write it cannot write to a share any more than a live SQLite database can live on one. Delivery is then verified where the file landed, by size and by reading the schema version back out of it, and a copy that fails verification is deleted rather than left looking like a backup.
Permissions may not survive the trip. A backup is written owner-only, because it holds every task, every comment and every token hash, and is exactly as sensitive as the database it copies. Many network mounts fix their permissions when they are mounted, so the change succeeds and does nothing, leaving the file as readable as everything else on the share. That is a question for the share rather than for this.
Taking one
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db backup
The last line is the one to read. A backup that ran is not the same as a backup with your work in it, and a count you recognise is the cheapest check there is.
The name is subroutine-<instance>-<when>-<schema>, and the suffix says how to read it back. .dump is a PostgreSQL archive, which pg_restore loads; .db is a SQLite copy, which is simply a database. They are not interchangeable in either direction, and a restore refuses the wrong one before anything is dropped rather than discovering it partway through.
So match on subroutine-* in anything that sweeps this directory, never on a suffix. A glob written against *.sql matches nothing on either engine as they are today, and only the backups taken by much older versions, which is the worst of the three outcomes because it looks like it is working.
Every backup carries the schema it was taken on, inside the file. The name echoes it, and the value inside is the authority, because anybody can rename a file. Restoring an older one works and offers you the upgrade; restoring a newer one is refused outright, because there is no downgrade and a partial read is worse than a clear failure.
What is there
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db backups
One kind so far, so nothing says which kind. That changes below, and it is worth knowing why before it does.
Three kinds of copy share this directory, and each has its own lifetime. A routine backup is one you asked for, and nothing removes it unless you say so. The other two are copies the program takes on its own initiative at a moment it knows is risky, and they bound themselves: an upgrade keeps its newest few rollback points, and a restore keeps its safety copy for a week. Neither ever counts or deletes a routine backup, and pruning never counts or deletes either of them.
That separation is the point rather than a detail. One shared counter meant an hourly prune reached back a day and deleted the rollback point for the upgrade that had gone wrong the day before, which is the copy you want precisely then. It also meant nothing ever removed a rollback point at all, so one accumulated per upgrade for ever.
Pruning, from a timer
--keep prunes to the newest few routine backups after taking one:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db backup --keep 1
It names every file it deletes, which is why this belongs in a timer rather than a cron line nobody reads: the timer's log is then the record of what went, and the first thing you want on the day a file you expected is not there.
The timer itself is systemd's, and nothing here checks it. It is the same shape as the unit in TLS, a proxy and systemd, with OnCalendar= and a service that runs the command above.
Putting one back
One backup is left, and this holds its name for the rest of the chapter. The directory is the service account's, so reading it needs sudo:
backup=/srv/backups/subroutine/$(sudo ls /srv/backups/subroutine | grep '\.dump$')Each backup has a small file beside it. subroutine-….dump.counts.json holds the counts printed above, which is where the listing reads them from rather than by opening the archive; pruning takes the pair away together. It is why the line above asks for the archive by name, and it is worth knowing before a script that sweeps this directory finds twice as many files as you have backups.
A restore will not run until you have said which kind it is:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db restore "$backup"
Neither is a safe default. A recovery keeps the instance's identity, which is what agents key their caches on. A clone mints a new one, so that two live instances never claim to be the same. Getting it wrong is invisible in both directions, so you are asked.
Stop the service first. A running one keeps its file handles on the database that was just replaced: it goes on writing to something that no longer has a name, its reads are stale, and its next write can land on top of the restored data, while the API answers normally throughout.
Not checked: the build does not run this.
sudo systemctl stop subroutine/readyz is not the check here. It notices a replacement that changed the identity, which is the clone case, and a recovery of the same instance underneath a running process is silent. Stopping the service is the answer, not watching an address.
Something has happened since the backup, so there is something to lose:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine add "Book the dojo for Thursday"
Now put the backup back, as this instance's own data returning:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db restore "$backup" --recover
The middle line is the safety copy, taken of what you were about to replace, and it is the answer to restoring the wrong file. It is one of the three kinds, so nothing you do to routine backups touches it.
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine list
The later task is gone and the earlier one is back, which is what a restore is.
The same file, as a separate copy
The other kind is for looking at rather than recovering: a copy of the production instance on a machine where you can do anything you like to it.
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db restore "$backup" --as-clone
That last line is the whole difference. A new identity is what stops two live copies claiming to be the same instance, and it is why a running service reports itself not ready after a clone rather than carrying on: it started on one instance and is now looking at another.
The listing says what each copy is for now:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db backups
One routine copy and one safety copy per restore, labelled now that there is more than one kind. The middle one is the interesting one: it holds two tasks, because it is what was there just before the recovery, including the task the recovery threw away. That is the file you want if you restore the wrong thing, and it is the reason the safety copy exists at all.
Two things this will not do to you. A backup from the other engine is refused before anything is dropped, so a SQLite copy cannot be poured into a PostgreSQL instance by accident; moving an instance between engines is subroutine db copy, which PostgreSQL, and when to switch uses. And the safety copy is never allowed to block the restore: where the database being replaced is too damaged to copy, which is the usual reason to be restoring at all, you are told so plainly and asked whether to go on, rather than refused.
Marking it as one worth protecting
It is worth typing this setting wrong on purpose once, because what happens next is the reason you can trust the rest of your configuration:
sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
protectd = true
SETTING
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine list
A setting it does not recognise is named on every command, with the nearest real one suggested, rather than ignored. protectd = true is not a protected instance and never was, and the difference between the two is a line of output rather than an afternoon.
So fix it:
sudo -u subroutine sed -i '/^protectd/d' /var/lib/subroutine/config/subroutine/config.toml
sudo -u subroutine tee -a /var/lib/subroutine/config/subroutine/config.toml > /dev/null <<'SETTING'
protected = true
SETTINGA destructive command now asks before it does anything:
sudo -u subroutine env \
XDG_CONFIG_HOME=/var/lib/subroutine/config \
XDG_DATA_HOME=/var/lib/subroutine/data \
XDG_STATE_HOME=/var/lib/subroutine/state \
/opt/subroutine/bin/subroutine db restore "$backup" --recover --yes
--yes is what that command needed, and it is the point rather than a way round it. At a terminal, without it, the same command asks This instance is marked protected. Go on? and waits. Where there is nothing to ask, which is a timer or a script, it refuses and says to pass --yes if you are certain. So nothing automated replaces this database by accident, and you can still do it deliberately in one word.
With that, the instance has copies somewhere else, the copies are pruned by something that writes down what it removed, and both ways of putting one back have been done once while nothing was at stake, which is the only good time to find out which is which.